2024-04-05T23:03:13.898 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-05T23:03:13.899 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731 --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} -- /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/orig.config.yaml 2024-04-05T23:03:13.960 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-05T23:03:14.052 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731 branch: wip-yuri4-testing-2024-04-04-1128-reef description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: yweinste@redhat.com first_in_suite: false job_id: '7641731' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri4-testing-2024-04-04-1128-reef ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-04-04-1128-reef sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5779 sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2024-04-04-1128-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhKN200afSOEQt8jS2PAtA+auKHXkuwd4t84/Xc01o8FAlxnbFSwngxY3y4AKZ54ltc0IxzGtPK4IeurQbL0+8= smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGE/ewOEzbuRlNTBnHCBUfCrNXyio0sdGQ70x+KuVD0SfW2MGXB1sB9LT4bN/Mkifm6DTomOKFyXZBdCSPc09JA= smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqL/Yp1fzQGCtcEKcn6RtQwjTyV7jysI6HKbhXnc2d0NScxPd6BhzW/NLwIN4Jt96vofWPfzMezXV74dE0ZUzA= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-05_18:13:21 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-05T23:03:14.053 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa; will attempt to use it 2024-04-05T23:03:14.053 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks 2024-04-05T23:03:14.053 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-05T23:03:14.055 INFO:teuthology.task.internal:Checking packages... 2024-04-05T23:03:14.079 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '8919605b0a5dc1f1650c8aec3051c77c0e3e240e' 2024-04-05T23:03:14.079 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-05T23:03:14.079 INFO:teuthology.packaging:ref: None 2024-04-05T23:03:14.079 INFO:teuthology.packaging:tag: None 2024-04-05T23:03:14.079 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-04-04-1128-reef 2024-04-05T23:03:14.080 INFO:teuthology.packaging:sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:03:14.080 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-04-04-1128-reef 2024-04-05T23:03:14.220 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1173.g8919605b 2024-04-05T23:03:14.222 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-05T23:03:14.228 INFO:teuthology.task.internal:no buildpackages task found 2024-04-05T23:03:14.228 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-05T23:03:14.264 INFO:teuthology.task.internal:Saving configuration 2024-04-05T23:03:14.287 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-05T23:03:14.292 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T23:03:14.317 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 22:56:58.800689', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhKN200afSOEQt8jS2PAtA+auKHXkuwd4t84/Xc01o8FAlxnbFSwngxY3y4AKZ54ltc0IxzGtPK4IeurQbL0+8='} 2024-04-05T23:03:14.340 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 22:56:58.801532', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGE/ewOEzbuRlNTBnHCBUfCrNXyio0sdGQ70x+KuVD0SfW2MGXB1sB9LT4bN/Mkifm6DTomOKFyXZBdCSPc09JA='} 2024-04-05T23:03:14.361 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 22:56:58.799143', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqL/Yp1fzQGCtcEKcn6RtQwjTyV7jysI6HKbhXnc2d0NScxPd6BhzW/NLwIN4Jt96vofWPfzMezXV74dE0ZUzA='} 2024-04-05T23:03:14.361 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-05T23:03:14.368 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-05T23:03:14.368 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-05T23:03:14.368 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-05T23:03:14.368 INFO:teuthology.run_tasks:Running task console_log... 2024-04-05T23:03:14.494 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feb94af3160>, signals=[15]) 2024-04-05T23:03:14.494 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-05T23:03:14.501 INFO:teuthology.task.internal:Opening connections... 2024-04-05T23:03:14.501 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:03:14.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T23:03:14.580 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-04-05T23:03:14.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T23:03:14.654 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-04-05T23:03:14.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T23:03:14.728 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-05T23:03:14.735 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-04-05T23:03:14.757 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-04-05T23:03:14.757 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-04-05T23:03:14.818 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Red Hat Enterprise Linux" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="8.6 (Ootpa)" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:ID="rhel" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="fedora" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="8.6" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el8" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.redhat.com/" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-05T23:03:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:03:14.820 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-05T23:03:14.820 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-05T23:03:14.820 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-05T23:03:14.820 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-05T23:03:14.820 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-04-05T23:03:14.845 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-04-05T23:03:14.866 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-04-05T23:03:14.866 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-04-05T23:03:14.929 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="Red Hat Enterprise Linux" 2024-04-05T23:03:14.929 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8.6 (Ootpa)" 2024-04-05T23:03:14.929 INFO:teuthology.orchestra.run.smithi027.stdout:ID="rhel" 2024-04-05T23:03:14.929 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="fedora" 2024-04-05T23:03:14.929 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8.6" 2024-04-05T23:03:14.929 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8" 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://www.redhat.com/" 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-05T23:03:14.930 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-05T23:03:14.931 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-04-05T23:03:14.953 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-04-05T23:03:14.974 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-04-05T23:03:14.974 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-04-05T23:03:15.034 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="Red Hat Enterprise Linux" 2024-04-05T23:03:15.034 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="8.6 (Ootpa)" 2024-04-05T23:03:15.034 INFO:teuthology.orchestra.run.smithi130.stdout:ID="rhel" 2024-04-05T23:03:15.034 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="fedora" 2024-04-05T23:03:15.034 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="8.6" 2024-04-05T23:03:15.034 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el8" 2024-04-05T23:03:15.034 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-05T23:03:15.034 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2024-04-05T23:03:15.035 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-05T23:03:15.035 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://www.redhat.com/" 2024-04-05T23:03:15.035 INFO:teuthology.orchestra.run.smithi130.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-05T23:03:15.035 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-05T23:03:15.035 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:03:15.035 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-05T23:03:15.035 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-05T23:03:15.035 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-05T23:03:15.035 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-05T23:03:15.035 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-04-05T23:03:15.061 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-05T23:03:15.072 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-05T23:03:15.109 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-05T23:03:15.110 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-04-05T23:03:15.112 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-04-05T23:03:15.114 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-04-05T23:03:15.135 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-05T23:03:15.141 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-05T23:03:15.141 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-04-05T23:03:15.173 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-04-05T23:03:15.175 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-04-05T23:03:15.201 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-05T23:03:15.276 INFO:teuthology.run_tasks:Running task kernel... 2024-04-05T23:03:15.292 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-05T23:03:15.292 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-05T23:03:15.292 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-05T23:03:15.293 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-05T23:03:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:03:15.314 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-04-05T23:03:15.564 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-05T23:03:15.564 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-04-05T23:03:15.565 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-04-05T23:03:15.586 INFO:teuthology.orchestra.run.smithi016.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-05T23:03:15.587 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-513.9.1.el8_9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-05T23:03:15.587 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-05T23:03:15.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum upgrade -y linux-firmware 2024-04-05T23:03:17.241 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2024-04-05T23:03:17.241 INFO:teuthology.orchestra.run.smithi016.stdout:Unable to read consumer identity 2024-04-05T23:03:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:03:17.554 INFO:teuthology.orchestra.run.smithi016.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-05T23:03:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:03:19.337 INFO:teuthology.orchestra.run.smithi016.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-04-05T23:03:20.599 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-04-05T23:03:24.621 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 553 kB/s | 24 kB 00:00 2024-04-05T23:03:25.492 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-05T23:03:25.494 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-05T23:03:25.494 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-05T23:03:25.664 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-04-05T23:03:25.664 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-05T23:03:25.809 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ 2024-04-05T23:03:25.809 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ --input-file=- 2024-04-05T23:03:25.812 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi016.front.sepia.ceph.com to complete... 2024-04-05T23:03:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05 23:03:51 URL:https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [401601944/401601944] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-05T23:03:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:FINISHED --2024-04-05 23:03:51-- 2024-04-05T23:03:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:Total wall clock time: 26s 2024-04-05T23:03:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:Downloaded: 1 files, 383M in 26s (14.9 MB/s) 2024-04-05T23:03:51.813 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-04-05T23:03:51.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-05T23:03:53.788 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying... ######################################## 2024-04-05T23:03:53.886 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing... ######################################## 2024-04-05T23:03:54.063 INFO:teuthology.orchestra.run.smithi016.stdout:Updating / installing... 2024-04-05T23:04:38.611 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-04-05T23:04:42.964 INFO:teuthology.orchestra.run.smithi016.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-05T23:05:05.781 INFO:teuthology.orchestra.run.smithi016.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-05T23:07:25.635 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-05T23:07:25.636 DEBUG:teuthology.orchestra.run.smithi016:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-05T23:07:25.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-04-05T23:07:25.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-04-05T23:07:25.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-04-05T23:07:25.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-05T23:07:25.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-04-05T23:07:25.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-04-05T23:07:25.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-04-05T23:07:25.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-04-05T23:07:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-05T23:07:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-04-05T23:07:25.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-04-05T23:07:25.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-04-05T23:07:25.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-04-05T23:07:25.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-04-05T23:07:25.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-04-05T23:07:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-05T23:07:25.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-05T23:07:25.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-04-05T23:07:25.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-04-05T23:07:25.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-04-05T23:07:25.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-04-05T23:07:25.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-04-05T23:07:25.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-04-05T23:07:25.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-04-05T23:07:25.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-04-05T23:07:25.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-04-05T23:07:25.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-05T23:07:25.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-04-05T23:07:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-04-05T23:07:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-04-05T23:07:25.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-04-05T23:07:25.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-05T23:07:25.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-04-05T23:07:25.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-04-05T23:07:25.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-05T23:07:25.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-05T23:07:25.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-04-05T23:07:25.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-04-05T23:07:25.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-05T23:07:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-04-05T23:07:25.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-05T23:07:25.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-04-05T23:07:25.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-04-05T23:07:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-04-05T23:07:25.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-05T23:07:25.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-05T23:07:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-05T23:07:25.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-05T23:07:25.745 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-04-05T23:07:25.746 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-05T23:07:25.747 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-05T23:07:25.748 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-05T23:07:25.749 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-04-05T23:07:25.750 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-05T23:07:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-05T23:07:25.752 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-05T23:07:25.752 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-04-05T23:07:25.752 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-05T23:07:25.752 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-05T23:07:25.753 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-05T23:07:25.753 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-05T23:07:25.753 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-05T23:07:25.753 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-05T23:07:25.753 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-05T23:07:25.754 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-05T23:07:25.755 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-05T23:07:25.756 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-05T23:07:25.757 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-05T23:07:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-05T23:07:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-04-05T23:07:25.760 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-04-05T23:07:25.761 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-05T23:07:25.762 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-04-05T23:07:25.763 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-04-05T23:07:25.764 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-05T23:07:25.765 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-04-05T23:07:25.766 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-05T23:07:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-05T23:07:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-04-05T23:07:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-04-05T23:07:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-04-05T23:07:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-04-05T23:07:25.772 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-05T23:07:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-04-05T23:07:25.774 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-04-05T23:07:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-04-05T23:07:25.776 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-05T23:07:25.777 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-04-05T23:07:25.778 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-04-05T23:07:25.779 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-04-05T23:07:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-04-05T23:07:25.781 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-05T23:07:25.782 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-04-05T23:07:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-05T23:07:25.784 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-05T23:07:25.785 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-05T23:07:25.786 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-04-05T23:07:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-05T23:07:25.788 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-04-05T23:07:25.789 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-05T23:07:25.790 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-04-05T23:07:25.791 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-04-05T23:07:25.792 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-04-05T23:07:25.793 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-04-05T23:07:25.794 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-04-05T23:07:25.795 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-04-05T23:07:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-04-05T23:07:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-04-05T23:07:25.798 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-04-05T23:07:25.799 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-04-05T23:07:25.800 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-04-05T23:07:25.801 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-04-05T23:07:25.802 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-04-05T23:07:25.803 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-04-05T23:07:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-05T23:07:25.805 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-05T23:07:25.806 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-04-05T23:07:25.807 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-04-05T23:07:25.808 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-05T23:07:25.809 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-05T23:07:25.810 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-04-05T23:07:25.811 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-05T23:07:25.812 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-05T23:07:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-04-05T23:07:25.814 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-05T23:07:25.815 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-05T23:07:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-05T23:07:25.817 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-04-05T23:07:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-04-05T23:07:25.819 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-04-05T23:07:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-04-05T23:07:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-04-05T23:07:25.822 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-04-05T23:07:25.823 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-04-05T23:07:25.824 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-04-05T23:07:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-04-05T23:07:25.826 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-04-05T23:07:25.827 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-04-05T23:07:25.828 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-04-05T23:07:25.829 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-05T23:07:25.830 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-04-05T23:07:25.831 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-04-05T23:07:25.832 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-04-05T23:07:25.832 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-04-05T23:07:25.832 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-04-05T23:07:25.832 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-04-05T23:07:25.832 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-04-05T23:07:25.832 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-04-05T23:07:25.832 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-04-05T23:07:25.832 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-04-05T23:07:25.832 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-04-05T23:07:25.836 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-04-05T23:07:25.837 DEBUG:teuthology.orchestra.run.smithi016:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout:postinstall scriptlet (using /bin/sh): 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout:fi 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout:for file in vmlinuz System.map config; do 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: fi 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout:done 2024-04-05T23:07:25.867 INFO:teuthology.orchestra.run.smithi016.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-05T23:07:25.868 INFO:teuthology.orchestra.run.smithi016.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-04-05T23:07:25.868 INFO:teuthology.orchestra.run.smithi016.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-04-05T23:07:25.868 INFO:teuthology.orchestra.run.smithi016.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-04-05T23:07:25.868 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-04-05T23:07:25.868 INFO:teuthology.orchestra.run.smithi016.stdout:fi 2024-04-05T23:07:25.868 INFO:teuthology.orchestra.run.smithi016.stdout:postuninstall scriptlet (using /bin/sh): 2024-04-05T23:07:25.868 INFO:teuthology.orchestra.run.smithi016.stdout:if [ -x /sbin/update-bootloader ]; then 2024-04-05T23:07:25.868 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-04-05T23:07:25.868 INFO:teuthology.orchestra.run.smithi016.stdout:fi 2024-04-05T23:07:25.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-04-05T23:07:25.966 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-04-05T23:07:25.966 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.02 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 123.el8 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Thu 14 Dec 2023 12:39:42 AM UTC 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Group : System Environment/Base 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 9265450 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Relocations : (not relocatable) 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : Red Hat, Inc. 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : Red Hat, Inc. 2024-04-05T23:07:25.967 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-05T23:07:25.968 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-04-05T23:07:25.968 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-04-05T23:07:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:07:25.968 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-05T23:07:25.968 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-05T23:07:25.968 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-05T23:07:25.968 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-04-05T23:07:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:07:25.968 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-04-05T23:07:25.969 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-05T23:07:25.969 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 6.8.0-g1df0c7e8a235 2024-04-05T23:07:25.970 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-05T23:07:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-04-05T23:07:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-04-05T23:07:29.179 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-04-05T23:07:29.203 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.LEvb3s4xFY 2024-04-05T23:07:29.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LEvb3s4xFY 2024-04-05T23:07:29.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.LEvb3s4xFY 2024-04-05T23:07:29.471 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.LEvb3s4xFY is 6KB 2024-04-05T23:07:29.497 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.LEvb3s4xFY 2024-04-05T23:07:29.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-04-05T23:07:29.589 INFO:teuthology.orchestra.run.smithi016.stdout:2db68f91c8f90721a6596efe657a5324-0-rescue.conf 2024-04-05T23:07:29.590 INFO:teuthology.orchestra.run.smithi016.stdout:2db68f91c8f90721a6596efe657a5324-6.8.0-g1df0c7e8a235.conf 2024-04-05T23:07:29.590 INFO:teuthology.orchestra.run.smithi016.stdout:963f8b6c99184de696879b3f7da9ecce-0-rescue.conf 2024-04-05T23:07:29.590 INFO:teuthology.orchestra.run.smithi016.stdout:963f8b6c99184de696879b3f7da9ecce-4.18.0-372.9.1.el8.x86_64.conf 2024-04-05T23:07:29.590 INFO:teuthology.orchestra.run.smithi016.stdout:963f8b6c99184de696879b3f7da9ecce-4.18.0-513.9.1.el8_9.x86_64.conf 2024-04-05T23:07:29.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 2db68f91c8f90721a6596efe657a5324-6.8.0-g1df0c7e8a235 2024-04-05T23:07:29.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-04-05T23:07:59.789 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-05T23:07:59.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-05T23:07:59.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T23:08:18.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-04-05T23:08:27.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-05T23:08:27.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T23:09:27.339 DEBUG:teuthology.orchestra.remote:timed out 2024-04-05T23:09:39.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-05T23:09:39.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T23:09:39.798 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-05T23:09:40.209 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-05T23:09:40.210 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-04-05T23:09:40.210 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-04-05T23:09:40.250 INFO:teuthology.orchestra.run.smithi016.stdout:6.8.0-g1df0c7e8a235 2024-04-05T23:09:40.250 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-04-05T23:09:40.250 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-05T23:09:40.251 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2024-04-05T23:09:41.252 DEBUG:teuthology.parallel:result is None 2024-04-05T23:09:41.252 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-05T23:09:41.263 INFO:teuthology.task.internal:Creating test directory... 2024-04-05T23:09:41.263 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-05T23:09:41.268 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-05T23:09:41.270 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-05T23:09:41.309 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-05T23:09:41.322 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-05T23:09:41.344 INFO:teuthology.task.internal:Creating archive directory... 2024-04-05T23:09:41.344 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-05T23:09:41.353 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-05T23:09:41.355 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-05T23:09:41.412 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-05T23:09:41.423 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-05T23:09:41.423 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-05T23:09:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:09:41.447 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-05T23:09:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:09:41.466 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-05T23:09:41.471 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-05T23:09:41.489 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-05T23:09:41.550 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T23:09:41.562 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T23:09:41.568 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T23:09:41.599 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T23:09:41.751 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T23:09:41.792 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T23:09:41.796 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-05T23:09:41.808 INFO:teuthology.task.internal:Configuring sudo... 2024-04-05T23:09:41.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-05T23:09:41.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-05T23:09:41.814 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-05T23:09:41.876 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-05T23:09:41.889 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-05T23:09:41.890 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-05T23:09:41.894 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-05T23:09:41.897 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-05T23:09:41.931 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T23:09:42.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T23:09:42.140 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T23:09:42.209 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T23:09:42.291 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:09:42.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-05T23:09:42.366 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T23:09:42.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T23:09:42.471 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T23:09:42.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T23:09:42.607 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:09:42.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-05T23:09:42.679 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T23:09:42.725 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T23:09:42.799 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T23:09:42.843 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T23:09:42.888 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:09:42.889 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-05T23:09:42.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-04-05T23:09:42.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-04-05T23:09:42.963 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-04-05T23:09:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T23:09:43.054 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T23:09:43.070 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T23:09:43.485 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-05T23:09:43.495 INFO:teuthology.task.internal:Starting timer... 2024-04-05T23:09:43.495 INFO:teuthology.run_tasks:Running task pcp... 2024-04-05T23:09:43.551 INFO:teuthology.run_tasks:Running task selinux... 2024-04-05T23:09:43.581 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-05T23:09:43.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-04-05T23:09:43.720 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-04-05T23:09:43.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-04-05T23:09:43.804 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-04-05T23:09:43.806 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2024-04-05T23:09:43.872 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2024-04-05T23:09:43.873 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-05T23:09:43.874 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-04-05T23:09:43.925 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-04-05T23:09:43.925 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-04-05T23:09:43.955 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-04-05T23:09:43.955 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2024-04-05T23:09:43.978 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2024-04-05T23:09:43.979 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi130.front.sepia.ceph.com': 'permissive'} 2024-04-05T23:09:43.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T23:09:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:09:44.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T23:09:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:09:44.082 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T23:09:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:09:44.110 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-05T23:09:44.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-04-05T23:09:44.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-04-05T23:09:44.204 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2024-04-05T23:09:44.237 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-05T23:09:44.247 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-05T23:09:44.265 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-05T23:09:44.267 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi027.front.sepia.ceph.com,smithi130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-05T23:20:45.824 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi130.front.sepia.ceph.com')] 2024-04-05T23:20:45.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-05T23:20:45.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T23:20:45.959 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-05T23:20:46.067 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-05T23:20:46.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-05T23:20:46.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T23:20:46.139 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-04-05T23:20:46.208 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-05T23:20:46.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-04-05T23:20:46.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T23:20:46.282 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-04-05T23:20:46.353 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-04-05T23:20:46.353 INFO:teuthology.run_tasks:Running task clock... 2024-04-05T23:20:46.362 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-05T23:20:46.362 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-05T23:20:46.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T23:20:46.367 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-05T23:20:46.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T23:20:46.370 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-05T23:20:46.370 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T23:20:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-05T23:20:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-05T23:20:46.427 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-05T23:20:46.437 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-05T23:20:46.442 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-05T23:20:46.472 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-05T23:20:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-04-05T23:20:46.502 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-04-05T23:20:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-05T23:20:46.522 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2024-04-05T23:20:46.533 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-05T23:20:46.535 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2024-04-05T23:20:46.549 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-05T23:20:46.562 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-04-05T23:20:46.564 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-05T23:20:46.591 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-04-05T23:20:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2024-04-05T23:20:46.621 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T23:20:46.621 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-05T23:20:46.621 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.621 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.621 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.621 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.624 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-05T23:20:46.646 INFO:teuthology.orchestra.run.smithi130.stderr:bash: ntpq: command not found 2024-04-05T23:20:46.650 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T23:20:46.650 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-04-05T23:20:46.650 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.650 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.650 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.650 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-05T23:20:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:bash: ntpq: command not found 2024-04-05T23:20:46.764 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T23:20:46.764 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-05T23:20:46.764 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.764 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.764 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.764 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:20:46.765 INFO:teuthology.run_tasks:Running task pexec... 2024-04-05T23:20:46.774 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-05T23:20:46.775 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:20:46.775 DEBUG:teuthology.orchestra.run.smithi016:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-05T23:20:46.775 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi027.front.sepia.ceph.com 2024-04-05T23:20:46.775 DEBUG:teuthology.orchestra.run.smithi027:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-05T23:20:46.776 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi130.front.sepia.ceph.com 2024-04-05T23:20:46.776 DEBUG:teuthology.orchestra.run.smithi130:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-05T23:20:47.152 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2024-04-05T23:20:47.175 INFO:teuthology.orchestra.run.smithi027.stdout:Updating Subscription Management repositories. 2024-04-05T23:20:47.293 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2024-04-05T23:20:49.034 INFO:teuthology.orchestra.run.smithi027.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-05T23:20:49.119 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 51 kB/s | 4.5 kB 00:00 2024-04-05T23:20:49.468 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-05T23:20:49.692 INFO:teuthology.orchestra.run.smithi027.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-05T23:20:49.725 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-05T23:20:49.902 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-05T23:20:52.566 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-04-05T23:20:52.567 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-04-05T23:20:52.567 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-04-05T23:20:52.567 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-04-05T23:20:52.568 INFO:teuthology.orchestra.run.smithi130.stdout:Resetting modules: 2024-04-05T23:20:52.568 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools 2024-04-05T23:20:52.568 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:20:52.568 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-04-05T23:20:52.568 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-04-05T23:20:52.568 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:20:52.871 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-05T23:20:52.872 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-05T23:20:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-05T23:20:52.872 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-05T23:20:52.872 INFO:teuthology.orchestra.run.smithi027.stdout:Resetting modules: 2024-04-05T23:20:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: container-tools 2024-04-05T23:20:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:20:52.873 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-05T23:20:52.873 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-05T23:20:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:20:53.394 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-05T23:20:53.395 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-05T23:20:53.395 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-05T23:20:53.395 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-05T23:20:53.395 INFO:teuthology.orchestra.run.smithi016.stdout:Resetting modules: 2024-04-05T23:20:53.396 INFO:teuthology.orchestra.run.smithi016.stdout: container-tools 2024-04-05T23:20:53.396 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:20:53.396 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-05T23:20:53.396 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-05T23:20:53.396 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:21:03.634 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-04-05T23:21:04.122 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2024-04-05T23:21:06.176 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-05T23:21:06.740 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 kB/s | 4.1 kB 00:00 2024-04-05T23:21:07.442 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-05T23:21:07.934 INFO:teuthology.orchestra.run.smithi027.stdout:Updating Subscription Management repositories. 2024-04-05T23:21:08.317 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-05T23:21:08.966 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2024-04-05T23:21:09.564 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-04-05T23:21:09.725 INFO:teuthology.orchestra.run.smithi130.stdout:========================================================================================================== 2024-04-05T23:21:09.725 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-04-05T23:21:09.725 INFO:teuthology.orchestra.run.smithi130.stdout:========================================================================================================== 2024-04-05T23:21:09.725 INFO:teuthology.orchestra.run.smithi130.stdout:Installing group/module packages: 2024-04-05T23:21:09.725 INFO:teuthology.orchestra.run.smithi130.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-04-05T23:21:09.725 INFO:teuthology.orchestra.run.smithi130.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout:Installing weak dependencies: 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-05T23:21:09.726 INFO:teuthology.orchestra.run.smithi130.stdout:Installing module profiles: 2024-04-05T23:21:09.727 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools/common 2024-04-05T23:21:09.727 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:21:09.727 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling module streams: 2024-04-05T23:21:09.727 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools rhel8 2024-04-05T23:21:09.727 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:21:09.727 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-04-05T23:21:09.727 INFO:teuthology.orchestra.run.smithi130.stdout:========================================================================================================== 2024-04-05T23:21:09.727 INFO:teuthology.orchestra.run.smithi130.stdout:Install 13 Packages 2024-04-05T23:21:09.727 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:21:09.728 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 26 M 2024-04-05T23:21:09.728 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 87 M 2024-04-05T23:21:09.728 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-04-05T23:21:09.826 INFO:teuthology.orchestra.run.smithi027.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-05T23:21:09.964 INFO:teuthology.orchestra.run.smithi130.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 105 kB/s | 25 kB 00:00 2024-04-05T23:21:09.989 INFO:teuthology.orchestra.run.smithi130.stdout:(2/13): python3-pyxdg-0.25-16.el8.noarch.rpm 359 kB/s | 94 kB 00:00 2024-04-05T23:21:10.023 INFO:teuthology.orchestra.run.smithi130.stdout:(3/13): criu-libs-3.18-4.module+el8.9.0+20326+3 133 kB/s | 39 kB 00:00 2024-04-05T23:21:10.182 INFO:teuthology.orchestra.run.smithi130.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 188 kB/s | 41 kB 00:00 2024-04-05T23:21:10.223 INFO:teuthology.orchestra.run.smithi130.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 4.3 MB/s | 1.0 MB 00:00 2024-04-05T23:21:10.424 INFO:teuthology.orchestra.run.smithi130.stdout:(6/13): buildah-1.31.3-3.module+el8.9.0+21243+a 22 MB/s | 8.8 MB 00:00 2024-04-05T23:21:10.457 INFO:teuthology.orchestra.run.smithi130.stdout:(7/13): cockpit-podman-75-1.module+el8.9.0+2124 2.6 MB/s | 739 kB 00:00 2024-04-05T23:21:10.482 INFO:teuthology.orchestra.run.smithi130.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 924 kB/s | 239 kB 00:00 2024-04-05T23:21:10.497 INFO:teuthology.orchestra.run.smithi027.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-05T23:21:10.658 INFO:teuthology.orchestra.run.smithi130.stdout:(9/13): netavark-1.7.0-2.module+el8.9.0+21243+a 16 MB/s | 3.7 MB 00:00 2024-04-05T23:21:10.799 INFO:teuthology.orchestra.run.smithi130.stdout:(10/13): python3-podman-4.6.0-2.module+el8.9.0+ 449 kB/s | 154 kB 00:00 2024-04-05T23:21:10.958 INFO:teuthology.orchestra.run.smithi130.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 8.4 MB/s | 2.5 MB 00:00 2024-04-05T23:21:11.045 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-05T23:21:11.066 INFO:teuthology.orchestra.run.smithi130.stdout:(12/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 14 MB/s | 8.1 MB 00:00 2024-04-05T23:21:11.092 INFO:teuthology.orchestra.run.smithi130.stdout:(13/13): udica-0.2.6-20.module+el8.9.0+21243+a5 169 kB/s | 49 kB 00:00 2024-04-05T23:21:11.092 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-04-05T23:21:11.092 INFO:teuthology.orchestra.run.smithi130.stdout:Total 19 MB/s | 26 MB 00:01 2024-04-05T23:21:11.381 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-04-05T23:21:11.402 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-04-05T23:21:11.402 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-04-05T23:21:11.507 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-05T23:21:11.631 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-04-05T23:21:11.640 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-04-05T23:21:12.315 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-04-05T23:21:12.808 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-04-05T23:21:13.003 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-04-05T23:21:13.174 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-04-05T23:21:13.333 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-04-05T23:21:13.545 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-04-05T23:21:13.677 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-05T23:21:13.773 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-04-05T23:21:13.855 INFO:teuthology.orchestra.run.smithi027.stdout:========================================================================================================== 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout:========================================================================================================== 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout:Installing group/module packages: 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-04-05T23:21:13.856 INFO:teuthology.orchestra.run.smithi027.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout:Installing module profiles: 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout: container-tools/common 2024-04-05T23:21:13.857 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:21:13.858 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling module streams: 2024-04-05T23:21:13.858 INFO:teuthology.orchestra.run.smithi027.stdout: container-tools rhel8 2024-04-05T23:21:13.858 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:21:13.858 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-05T23:21:13.858 INFO:teuthology.orchestra.run.smithi027.stdout:========================================================================================================== 2024-04-05T23:21:13.858 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2024-04-05T23:21:13.858 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:21:13.859 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 26 M 2024-04-05T23:21:13.859 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 87 M 2024-04-05T23:21:13.859 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-05T23:21:14.007 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-04-05T23:21:14.035 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 141 kB/s | 25 kB 00:00 2024-04-05T23:21:14.060 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): criu-libs-3.18-4.module+el8.9.0+20326+3 195 kB/s | 39 kB 00:00 2024-04-05T23:21:14.077 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 429 kB/s | 94 kB 00:00 2024-04-05T23:21:14.169 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 305 kB/s | 41 kB 00:00 2024-04-05T23:21:14.261 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 5.0 MB/s | 1.0 MB 00:00 2024-04-05T23:21:14.328 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 4.5 MB/s | 739 kB 00:00 2024-04-05T23:21:14.404 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.6 MB/s | 239 kB 00:00 2024-04-05T23:21:14.570 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): netavark-1.7.0-2.module+el8.9.0+21243+a 15 MB/s | 3.7 MB 00:00 2024-04-05T23:21:14.596 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 801 kB/s | 154 kB 00:00 2024-04-05T23:21:14.696 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): buildah-1.31.3-3.module+el8.9.0+21243+ 14 MB/s | 8.8 MB 00:00 2024-04-05T23:21:14.805 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 12 MB/s | 2.5 MB 00:00 2024-04-05T23:21:14.871 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 282 kB/s | 49 kB 00:00 2024-04-05T23:21:14.893 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-05T23:21:14.922 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-04-05T23:21:14.988 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 20 MB/s | 8.1 MB 00:00 2024-04-05T23:21:14.989 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-05T23:21:14.989 INFO:teuthology.orchestra.run.smithi027.stdout:Total 23 MB/s | 26 MB 00:01 2024-04-05T23:21:15.079 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================================== 2024-04-05T23:21:15.079 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================================== 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout:Installing group/module packages: 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-04-05T23:21:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout:Installing module profiles: 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: container-tools/common 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling module streams: 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: container-tools rhel8 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================================== 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout:Install 13 Packages 2024-04-05T23:21:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:21:15.082 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 26 M 2024-04-05T23:21:15.082 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 87 M 2024-04-05T23:21:15.082 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-05T23:21:15.167 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-04-05T23:21:15.211 INFO:teuthology.orchestra.run.smithi016.stdout:(1/13): python3-pyxdg-0.25-16.el8.noarch.rpm 735 kB/s | 94 kB 00:00 2024-04-05T23:21:15.278 INFO:teuthology.orchestra.run.smithi016.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 127 kB/s | 25 kB 00:00 2024-04-05T23:21:15.282 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-05T23:21:15.305 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-05T23:21:15.306 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-05T23:21:15.329 INFO:teuthology.orchestra.run.smithi016.stdout:(3/13): criu-libs-3.18-4.module+el8.9.0+20326+3 160 kB/s | 39 kB 00:00 2024-04-05T23:21:15.412 INFO:teuthology.orchestra.run.smithi016.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 204 kB/s | 41 kB 00:00 2024-04-05T23:21:15.504 INFO:teuthology.orchestra.run.smithi016.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 4.5 MB/s | 1.0 MB 00:00 2024-04-05T23:21:15.540 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-04-05T23:21:15.550 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-05T23:21:15.559 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-05T23:21:15.630 INFO:teuthology.orchestra.run.smithi016.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 3.3 MB/s | 739 kB 00:00 2024-04-05T23:21:15.680 INFO:teuthology.orchestra.run.smithi016.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.3 MB/s | 239 kB 00:00 2024-04-05T23:21:15.897 INFO:teuthology.orchestra.run.smithi016.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 16 MB/s | 8.8 MB 00:00 2024-04-05T23:21:15.922 INFO:teuthology.orchestra.run.smithi016.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 635 kB/s | 154 kB 00:00 2024-04-05T23:21:16.023 INFO:teuthology.orchestra.run.smithi016.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 9.5 MB/s | 3.7 MB 00:00 2024-04-05T23:21:16.123 INFO:teuthology.orchestra.run.smithi016.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 13 MB/s | 2.5 MB 00:00 2024-04-05T23:21:16.173 INFO:teuthology.orchestra.run.smithi016.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 329 kB/s | 49 kB 00:00 2024-04-05T23:21:16.288 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-05T23:21:16.298 INFO:teuthology.orchestra.run.smithi016.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 20 MB/s | 8.1 MB 00:00 2024-04-05T23:21:16.299 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-05T23:21:16.299 INFO:teuthology.orchestra.run.smithi016.stdout:Total 21 MB/s | 26 MB 00:01 2024-04-05T23:21:16.407 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-04-05T23:21:16.610 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-04-05T23:21:16.774 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-05T23:21:16.801 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-05T23:21:16.801 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-05T23:21:16.802 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-04-05T23:21:16.864 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-04-05T23:21:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-04-05T23:21:17.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-04-05T23:21:17.207 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-04-05T23:21:17.249 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-04-05T23:21:17.325 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-05T23:21:17.335 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-05T23:21:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-04-05T23:21:17.622 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-04-05T23:21:17.870 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-04-05T23:21:18.113 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-05T23:21:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-04-05T23:21:18.652 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-04-05T23:21:18.842 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-04-05T23:21:19.052 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-04-05T23:21:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-04-05T23:21:19.223 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-04-05T23:21:19.355 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-04-05T23:21:19.406 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-04-05T23:21:19.692 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-04-05T23:21:19.747 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-04-05T23:21:19.932 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-04-05T23:21:20.846 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-04-05T23:21:20.903 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-04-05T23:21:21.105 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-04-05T23:21:21.135 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-04-05T23:21:21.303 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-04-05T23:21:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-04-05T23:21:21.745 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-04-05T23:21:21.745 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-04-05T23:21:21.746 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-04-05T23:21:22.543 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-04-05T23:21:22.818 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-04-05T23:21:22.969 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-04-05T23:21:23.645 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-04-05T23:21:23.646 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-04-05T23:21:28.045 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-04-05T23:21:28.046 INFO:teuthology.orchestra.run.smithi130.stdout:Installed products updated. 2024-04-05T23:21:28.596 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:21:28.596 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-04-05T23:21:28.596 INFO:teuthology.orchestra.run.smithi130.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:28.596 INFO:teuthology.orchestra.run.smithi130.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:21:28.597 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-04-05T23:21:28.754 DEBUG:teuthology.parallel:result is None 2024-04-05T23:21:31.351 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-04-05T23:21:31.351 INFO:teuthology.orchestra.run.smithi027.stdout:Installed products updated. 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-05T23:21:31.952 INFO:teuthology.orchestra.run.smithi027.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:31.953 INFO:teuthology.orchestra.run.smithi027.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:31.953 INFO:teuthology.orchestra.run.smithi027.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-04-05T23:21:31.953 INFO:teuthology.orchestra.run.smithi027.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-05T23:21:31.953 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:21:31.953 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-05T23:21:32.113 DEBUG:teuthology.parallel:result is None 2024-04-05T23:21:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-04-05T23:21:34.348 INFO:teuthology.orchestra.run.smithi016.stdout:Installed products updated. 2024-04-05T23:21:34.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:21:34.986 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-05T23:21:34.986 INFO:teuthology.orchestra.run.smithi016.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:34.986 INFO:teuthology.orchestra.run.smithi016.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:34.986 INFO:teuthology.orchestra.run.smithi016.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-04-05T23:21:34.986 INFO:teuthology.orchestra.run.smithi016.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:21:34.987 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-05T23:21:35.217 DEBUG:teuthology.parallel:result is None 2024-04-05T23:21:35.217 INFO:teuthology.run_tasks:Running task install... 2024-04-05T23:21:35.226 DEBUG:teuthology.task.install:project ceph 2024-04-05T23:21:35.226 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e'}} 2024-04-05T23:21:35.226 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e'} 2024-04-05T23:21:35.226 INFO:teuthology.task.install:Using flavor: default 2024-04-05T23:21:35.232 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-05T23:21:35.232 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-04-05T23:21:35.232 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e', 'tag': None, 'wait_for_package': False} 2024-04-05T23:21:35.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:21:35.235 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e', 'tag': None, 'wait_for_package': False} 2024-04-05T23:21:35.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:21:35.236 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e', 'tag': None, 'wait_for_package': False} 2024-04-05T23:21:35.236 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:21:35.400 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/ 2024-04-05T23:21:35.400 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1173.g8919605b 2024-04-05T23:21:35.454 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/ 2024-04-05T23:21:35.454 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1173.g8919605b 2024-04-05T23:21:35.458 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/ 2024-04-05T23:21:35.458 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1173.g8919605b 2024-04-05T23:21:35.572 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-05T23:21:35.572 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:21:35.572 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-05T23:21:35.606 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-05T23:21:35.607 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-05T23:21:35.683 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-05T23:21:35.729 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-05T23:21:35.730 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:21:35.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-05T23:21:35.732 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-05T23:21:35.732 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:21:35.732 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-05T23:21:35.764 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-05T23:21:35.767 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-05T23:21:35.767 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-05T23:21:35.800 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-05T23:21:35.801 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-05T23:21:35.833 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2024-04-05T23:21:35.834 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2024-04-05T23:21:35.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-05T23:21:35.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-05T23:21:35.924 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-05T23:21:35.993 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-04-05T23:21:35.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-04-05T23:21:35.999 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-05T23:21:36.063 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-04-05T23:21:36.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-04-05T23:21:36.214 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2024-04-05T23:21:36.394 INFO:teuthology.orchestra.run.smithi027.stdout:Updating Subscription Management repositories. 2024-04-05T23:21:36.543 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2024-04-05T23:21:37.674 INFO:teuthology.orchestra.run.smithi130.stdout:61 files removed 2024-04-05T23:21:37.695 INFO:teuthology.orchestra.run.smithi027.stdout:61 files removed 2024-04-05T23:21:37.726 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-05T23:21:37.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-05T23:21:38.050 INFO:teuthology.orchestra.run.smithi016.stdout:61 files removed 2024-04-05T23:21:38.064 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2024-04-05T23:21:38.108 INFO:teuthology.orchestra.run.smithi027.stdout:Updating Subscription Management repositories. 2024-04-05T23:21:38.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-05T23:21:38.588 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2024-04-05T23:21:39.936 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 208 kB/s | 80 kB 00:00 2024-04-05T23:21:39.964 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 159 kB/s | 80 kB 00:00 2024-04-05T23:21:40.498 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 23 kB/s | 11 kB 00:00 2024-04-05T23:21:40.537 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 20 kB/s | 11 kB 00:00 2024-04-05T23:21:40.589 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 302 kB/s | 80 kB 00:00 2024-04-05T23:21:40.933 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 3.6 kB/s | 1.5 kB 00:00 2024-04-05T23:21:40.970 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 3.7 kB/s | 1.5 kB 00:00 2024-04-05T23:21:41.023 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-04-05T23:21:41.298 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 6.1 kB/s | 1.5 kB 00:00 2024-04-05T23:21:41.591 INFO:teuthology.orchestra.run.smithi016.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2024-04-05T23:21:41.671 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 5.1 kB/s | 3.5 kB 00:00 2024-04-05T23:21:41.691 INFO:teuthology.orchestra.run.smithi130.stdout:Copr repo for python3-asyncssh owned by ceph 4.7 kB/s | 3.5 kB 00:00 2024-04-05T23:21:42.690 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-04-05T23:21:42.850 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-05T23:21:43.482 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 16 MB 00:01 2024-04-05T23:21:46.409 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 517 kB/s | 24 kB 00:00 2024-04-05T23:21:47.075 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 444 kB/s | 24 kB 00:00 2024-04-05T23:21:47.289 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 381 kB/s | 24 kB 00:00 2024-04-05T23:21:47.460 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 9.0 MB/s | 8.8 MB 00:00 2024-04-05T23:21:47.844 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-04-05T23:21:48.682 INFO:teuthology.orchestra.run.smithi027.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 MB/s | 60 MB 00:01 2024-04-05T23:21:50.253 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 MB/s | 60 MB 00:01 2024-04-05T23:21:51.171 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 MB/s | 66 MB 00:01 2024-04-05T23:22:01.142 INFO:teuthology.orchestra.run.smithi027.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 46 MB/s | 66 MB 00:01 2024-04-05T23:22:01.570 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 66 MB 00:01 2024-04-05T23:22:03.639 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 MB/s | 60 MB 00:01 2024-04-05T23:22:12.459 INFO:teuthology.orchestra.run.smithi027.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 15 MB/s | 8.8 MB 00:00 2024-04-05T23:22:16.550 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-05T23:22:16.551 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-05T23:22:16.570 INFO:teuthology.orchestra.run.smithi130.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-04-05T23:22:16.571 INFO:teuthology.orchestra.run.smithi130.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-04-05T23:22:16.572 INFO:teuthology.orchestra.run.smithi130.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-04-05T23:22:16.573 INFO:teuthology.orchestra.run.smithi130.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-04-05T23:22:16.576 INFO:teuthology.orchestra.run.smithi130.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-04-05T23:22:16.580 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-04-05T23:22:16.581 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-04-05T23:22:16.799 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-04-05T23:22:16.813 INFO:teuthology.orchestra.run.smithi130.stdout:=================================================================================================================================== 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout:=================================================================================================================================== 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:18.2.2-1173.g8919605b.el8 ceph 6.4 k 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:18.2.2-1173.g8919605b.el8 ceph 5.5 M 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:18.2.2-1173.g8919605b.el8 ceph 859 k 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1173.g8919605b.el8 ceph 166 k 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:18.2.2-1173.g8919605b.el8 ceph 1.6 M 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 138 k 2024-04-05T23:22:16.814 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 3.5 M 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 7.4 M 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 49 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:18.2.2-1173.g8919605b.el8 ceph 13 M 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:18.2.2-1173.g8919605b.el8 ceph 47 M 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 263 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 173 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror x86_64 2:18.2.2-1173.g8919605b.el8 ceph 261 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 18 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:18.2.2-1173.g8919605b.el8 ceph 31 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 781 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:18.2.2-1173.g8919605b.el8 ceph 139 k 2024-04-05T23:22:16.815 INFO:teuthology.orchestra.run.smithi130.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:18.2.2-1173.g8919605b.el8 ceph 196 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:18.2.2-1173.g8919605b.el8 ceph 372 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:18.2.2-1173.g8919605b.el8 ceph 364 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:18.2.2-1173.g8919605b.el8 ceph 107 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:18.2.2-1173.g8919605b.el8 ceph 95 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:18.2.2-1173.g8919605b.el8 ceph 3.5 M 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:18.2.2-1173.g8919605b.el8 ceph 181 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-04-05T23:22:16.816 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 3.7 M 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 3.6 M 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-04-05T23:22:16.817 INFO:teuthology.orchestra.run.smithi130.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:18.2.2-1173.g8919605b.el8 ceph 22 M 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 24 k 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:18.2.2-1173.g8919605b.el8 ceph 2.3 M 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 245 k 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:18.2.2-1173.g8919605b.el8 ceph 4.8 M 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:18.2.2-1173.g8919605b.el8 ceph 18 M 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 15 k 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:18.2.2-1173.g8919605b.el8 ceph 24 k 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-05T23:22:16.818 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:18.2.2-1173.g8919605b.el8 ceph 142 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 475 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 6.5 M 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-04-05T23:22:16.819 INFO:teuthology.orchestra.run.smithi130.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-05T23:22:16.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:18.2.2-1173.g8919605b.el8 ceph 45 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:18.2.2-1173.g8919605b.el8 ceph 129 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-05T23:22:16.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-05T23:22:16.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-05T23:22:16.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling module streams: 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql 10 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout:=================================================================================================================================== 2024-04-05T23:22:16.824 INFO:teuthology.orchestra.run.smithi130.stdout:Install 130 Packages 2024-04-05T23:22:16.825 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 16 Packages 2024-04-05T23:22:16.825 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:22:16.825 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 215 M 2024-04-05T23:22:16.825 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-04-05T23:22:17.495 INFO:teuthology.orchestra.run.smithi130.stdout:(1/146): ceph-18.2.2-1173.g8919605b.el8.x86_64. 20 kB/s | 6.4 kB 00:00 2024-04-05T23:22:18.562 INFO:teuthology.orchestra.run.smithi130.stdout:(2/146): ceph-fuse-18.2.2-1173.g8919605b.el8.x8 805 kB/s | 859 kB 00:01 2024-04-05T23:22:18.770 INFO:teuthology.orchestra.run.smithi130.stdout:(3/146): ceph-immutable-object-cache-18.2.2-117 796 kB/s | 166 kB 00:00 2024-04-05T23:22:19.495 INFO:teuthology.orchestra.run.smithi130.stdout:(4/146): ceph-base-18.2.2-1173.g8919605b.el8.x8 2.4 MB/s | 5.5 MB 00:02 2024-04-05T23:22:19.820 INFO:teuthology.orchestra.run.smithi130.stdout:(5/146): ceph-mgr-18.2.2-1173.g8919605b.el8.x86 4.8 MB/s | 1.6 MB 00:00 2024-04-05T23:22:19.902 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-05T23:22:19.903 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-05T23:22:19.923 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-04-05T23:22:19.924 INFO:teuthology.orchestra.run.smithi027.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-04-05T23:22:19.926 INFO:teuthology.orchestra.run.smithi027.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-04-05T23:22:19.928 INFO:teuthology.orchestra.run.smithi027.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-04-05T23:22:19.931 INFO:teuthology.orchestra.run.smithi027.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-04-05T23:22:19.935 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-04-05T23:22:19.937 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-04-05T23:22:20.176 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout:=================================================================================================================================== 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout:=================================================================================================================================== 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:18.2.2-1173.g8919605b.el8 ceph 6.4 k 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:18.2.2-1173.g8919605b.el8 ceph 5.5 M 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:18.2.2-1173.g8919605b.el8 ceph 859 k 2024-04-05T23:22:20.192 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1173.g8919605b.el8 ceph 166 k 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:18.2.2-1173.g8919605b.el8 ceph 1.6 M 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 138 k 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 3.5 M 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 7.4 M 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 49 k 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:18.2.2-1173.g8919605b.el8 ceph 13 M 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:18.2.2-1173.g8919605b.el8 ceph 47 M 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 263 k 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 173 k 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror x86_64 2:18.2.2-1173.g8919605b.el8 ceph 261 k 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 18 k 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-04-05T23:22:20.193 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:18.2.2-1173.g8919605b.el8 ceph 31 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 781 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:18.2.2-1173.g8919605b.el8 ceph 139 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:18.2.2-1173.g8919605b.el8 ceph 196 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:18.2.2-1173.g8919605b.el8 ceph 372 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:18.2.2-1173.g8919605b.el8 ceph 364 k 2024-04-05T23:22:20.194 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:18.2.2-1173.g8919605b.el8 ceph 107 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:18.2.2-1173.g8919605b.el8 ceph 95 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:18.2.2-1173.g8919605b.el8 ceph 3.5 M 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:18.2.2-1173.g8919605b.el8 ceph 181 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 3.7 M 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 3.6 M 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-04-05T23:22:20.195 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:18.2.2-1173.g8919605b.el8 ceph 22 M 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 24 k 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:18.2.2-1173.g8919605b.el8 ceph 2.3 M 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 245 k 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:18.2.2-1173.g8919605b.el8 ceph 4.8 M 2024-04-05T23:22:20.196 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:18.2.2-1173.g8919605b.el8 ceph 18 M 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 15 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:18.2.2-1173.g8919605b.el8 ceph 24 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:18.2.2-1173.g8919605b.el8 ceph 142 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-05T23:22:20.197 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 475 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 6.5 M 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-04-05T23:22:20.198 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:18.2.2-1173.g8919605b.el8 ceph 45 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:18.2.2-1173.g8919605b.el8 ceph 129 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-05T23:22:20.199 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-05T23:22:20.200 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-05T23:22:20.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-04-05T23:22:20.202 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling module streams: 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql 10 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout:=================================================================================================================================== 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout:Install 130 Packages 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 16 Packages 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:22:20.203 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 215 M 2024-04-05T23:22:20.204 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-05T23:22:20.585 INFO:teuthology.orchestra.run.smithi027.stdout:(1/146): ceph-18.2.2-1173.g8919605b.el8.x86_64. 19 kB/s | 6.4 kB 00:00 2024-04-05T23:22:20.631 INFO:teuthology.orchestra.run.smithi130.stdout:(6/146): ceph-mon-18.2.2-1173.g8919605b.el8.x86 6.0 MB/s | 4.8 MB 00:00 2024-04-05T23:22:20.879 INFO:teuthology.orchestra.run.smithi130.stdout:(7/146): ceph-mds-18.2.2-1173.g8919605b.el8.x86 1.1 MB/s | 2.3 MB 00:02 2024-04-05T23:22:21.033 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-05T23:22:21.034 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-05T23:22:21.055 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-04-05T23:22:21.057 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-04-05T23:22:21.058 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-04-05T23:22:21.060 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-04-05T23:22:21.063 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-04-05T23:22:21.067 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-04-05T23:22:21.069 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-04-05T23:22:21.316 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-05T23:22:21.332 INFO:teuthology.orchestra.run.smithi016.stdout:=================================================================================================================================== 2024-04-05T23:22:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-05T23:22:21.332 INFO:teuthology.orchestra.run.smithi016.stdout:=================================================================================================================================== 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:18.2.2-1173.g8919605b.el8 ceph 6.4 k 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:18.2.2-1173.g8919605b.el8 ceph 5.5 M 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:18.2.2-1173.g8919605b.el8 ceph 859 k 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1173.g8919605b.el8 ceph 166 k 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:18.2.2-1173.g8919605b.el8 ceph 1.6 M 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 138 k 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 3.5 M 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 7.4 M 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 49 k 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:18.2.2-1173.g8919605b.el8 ceph 13 M 2024-04-05T23:22:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:18.2.2-1173.g8919605b.el8 ceph 47 M 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 263 k 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 173 k 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:18.2.2-1173.g8919605b.el8 ceph 261 k 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 18 k 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:18.2.2-1173.g8919605b.el8 ceph 31 k 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 781 k 2024-04-05T23:22:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:18.2.2-1173.g8919605b.el8 ceph 139 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:18.2.2-1173.g8919605b.el8 ceph 196 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:18.2.2-1173.g8919605b.el8 ceph 372 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:18.2.2-1173.g8919605b.el8 ceph 364 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:18.2.2-1173.g8919605b.el8 ceph 107 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:18.2.2-1173.g8919605b.el8 ceph 95 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:18.2.2-1173.g8919605b.el8 ceph 3.5 M 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:18.2.2-1173.g8919605b.el8 ceph 181 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-04-05T23:22:21.335 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 3.7 M 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 3.6 M 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-04-05T23:22:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:18.2.2-1173.g8919605b.el8 ceph 22 M 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 24 k 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:18.2.2-1173.g8919605b.el8 ceph 2.3 M 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 245 k 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:18.2.2-1173.g8919605b.el8 ceph 4.8 M 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:18.2.2-1173.g8919605b.el8 ceph 18 M 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1173.g8919605b.el8 ceph-noarch 15 k 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:18.2.2-1173.g8919605b.el8 ceph 24 k 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-05T23:22:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:18.2.2-1173.g8919605b.el8 ceph 142 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 475 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:18.2.2-1173.g8919605b.el8 ceph 6.5 M 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-04-05T23:22:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-05T23:22:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:18.2.2-1173.g8919605b.el8 ceph 45 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:18.2.2-1173.g8919605b.el8 ceph 129 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-05T23:22:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-05T23:22:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-05T23:22:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-05T23:22:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling module streams: 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql 10 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout:=================================================================================================================================== 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout:Install 130 Packages 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 16 Packages 2024-04-05T23:22:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:22:21.345 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 215 M 2024-04-05T23:22:21.345 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-05T23:22:21.500 INFO:teuthology.orchestra.run.smithi027.stdout:(2/146): ceph-fuse-18.2.2-1173.g8919605b.el8.x8 938 kB/s | 859 kB 00:00 2024-04-05T23:22:21.619 INFO:teuthology.orchestra.run.smithi027.stdout:(3/146): ceph-immutable-object-cache-18.2.2-117 1.4 MB/s | 166 kB 00:00 2024-04-05T23:22:21.631 INFO:teuthology.orchestra.run.smithi016.stdout:(1/146): ceph-18.2.2-1173.g8919605b.el8.x86_64. 27 kB/s | 6.4 kB 00:00 2024-04-05T23:22:22.353 INFO:teuthology.orchestra.run.smithi027.stdout:(4/146): ceph-base-18.2.2-1173.g8919605b.el8.x8 2.6 MB/s | 5.5 MB 00:02 2024-04-05T23:22:22.544 INFO:teuthology.orchestra.run.smithi027.stdout:(5/146): ceph-mds-18.2.2-1173.g8919605b.el8.x86 2.5 MB/s | 2.3 MB 00:00 2024-04-05T23:22:22.664 INFO:teuthology.orchestra.run.smithi016.stdout:(2/146): ceph-fuse-18.2.2-1173.g8919605b.el8.x8 831 kB/s | 859 kB 00:01 2024-04-05T23:22:22.898 INFO:teuthology.orchestra.run.smithi016.stdout:(3/146): ceph-immutable-object-cache-18.2.2-117 712 kB/s | 166 kB 00:00 2024-04-05T23:22:22.911 INFO:teuthology.orchestra.run.smithi027.stdout:(6/146): ceph-mgr-18.2.2-1173.g8919605b.el8.x86 2.8 MB/s | 1.6 MB 00:00 2024-04-05T23:22:23.887 INFO:teuthology.orchestra.run.smithi027.stdout:(7/146): ceph-mon-18.2.2-1173.g8919605b.el8.x86 3.6 MB/s | 4.8 MB 00:01 2024-04-05T23:22:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:(4/146): ceph-base-18.2.2-1173.g8919605b.el8.x8 2.1 MB/s | 5.5 MB 00:02 2024-04-05T23:22:24.071 INFO:teuthology.orchestra.run.smithi130.stdout:(8/146): ceph-osd-18.2.2-1173.g8919605b.el8.x86 5.3 MB/s | 18 MB 00:03 2024-04-05T23:22:24.163 INFO:teuthology.orchestra.run.smithi130.stdout:(9/146): ceph-selinux-18.2.2-1173.g8919605b.el8 266 kB/s | 24 kB 00:00 2024-04-05T23:22:24.487 INFO:teuthology.orchestra.run.smithi016.stdout:(5/146): ceph-mgr-18.2.2-1173.g8919605b.el8.x86 3.7 MB/s | 1.6 MB 00:00 2024-04-05T23:22:24.615 INFO:teuthology.orchestra.run.smithi016.stdout:(6/146): ceph-mds-18.2.2-1173.g8919605b.el8.x86 1.3 MB/s | 2.3 MB 00:01 2024-04-05T23:22:25.357 INFO:teuthology.orchestra.run.smithi016.stdout:(7/146): ceph-mon-18.2.2-1173.g8919605b.el8.x86 5.6 MB/s | 4.8 MB 00:00 2024-04-05T23:22:27.087 INFO:teuthology.orchestra.run.smithi027.stdout:(8/146): ceph-radosgw-18.2.2-1173.g8919605b.el8 4.1 MB/s | 13 MB 00:03 2024-04-05T23:22:27.187 INFO:teuthology.orchestra.run.smithi027.stdout:(9/146): ceph-selinux-18.2.2-1173.g8919605b.el8 244 kB/s | 24 kB 00:00 2024-04-05T23:22:27.507 INFO:teuthology.orchestra.run.smithi016.stdout:(8/146): ceph-radosgw-18.2.2-1173.g8919605b.el8 6.1 MB/s | 13 MB 00:02 2024-04-05T23:22:27.591 INFO:teuthology.orchestra.run.smithi016.stdout:(9/146): ceph-selinux-18.2.2-1173.g8919605b.el8 293 kB/s | 24 kB 00:00 2024-04-05T23:22:28.870 INFO:teuthology.orchestra.run.smithi027.stdout:(10/146): ceph-common-18.2.2-1173.g8919605b.el8 2.5 MB/s | 22 MB 00:08 2024-04-05T23:22:29.079 INFO:teuthology.orchestra.run.smithi027.stdout:(11/146): cephfs-mirror-18.2.2-1173.g8919605b.e 1.2 MB/s | 261 kB 00:00 2024-04-05T23:22:29.204 INFO:teuthology.orchestra.run.smithi027.stdout:(12/146): libcephfs-devel-18.2.2-1173.g8919605b 248 kB/s | 31 kB 00:00 2024-04-05T23:22:29.465 INFO:teuthology.orchestra.run.smithi130.stdout:(10/146): ceph-common-18.2.2-1173.g8919605b.el8 1.8 MB/s | 22 MB 00:12 2024-04-05T23:22:29.719 INFO:teuthology.orchestra.run.smithi130.stdout:(11/146): cephfs-mirror-18.2.2-1173.g8919605b.e 1.0 MB/s | 261 kB 00:00 2024-04-05T23:22:29.729 INFO:teuthology.orchestra.run.smithi027.stdout:(13/146): ceph-osd-18.2.2-1173.g8919605b.el8.x8 2.7 MB/s | 18 MB 00:06 2024-04-05T23:22:29.763 INFO:teuthology.orchestra.run.smithi027.stdout:(14/146): libcephfs2-18.2.2-1173.g8919605b.el8. 1.4 MB/s | 781 kB 00:00 2024-04-05T23:22:29.804 INFO:teuthology.orchestra.run.smithi130.stdout:(12/146): libcephfs-devel-18.2.2-1173.g8919605b 365 kB/s | 31 kB 00:00 2024-04-05T23:22:29.880 INFO:teuthology.orchestra.run.smithi027.stdout:(15/146): libcephsqlite-18.2.2-1173.g8919605b.e 946 kB/s | 142 kB 00:00 2024-04-05T23:22:30.011 INFO:teuthology.orchestra.run.smithi027.stdout:(16/146): librados-devel-18.2.2-1173.g8919605b. 562 kB/s | 139 kB 00:00 2024-04-05T23:22:30.122 INFO:teuthology.orchestra.run.smithi027.stdout:(17/146): libradosstriper1-18.2.2-1173.g8919605 1.9 MB/s | 475 kB 00:00 2024-04-05T23:22:30.222 INFO:teuthology.orchestra.run.smithi027.stdout:(18/146): python3-ceph-argparse-18.2.2-1173.g89 446 kB/s | 45 kB 00:00 2024-04-05T23:22:30.238 INFO:teuthology.orchestra.run.smithi130.stdout:(13/146): libcephfs2-18.2.2-1173.g8919605b.el8. 1.8 MB/s | 781 kB 00:00 2024-04-05T23:22:30.332 INFO:teuthology.orchestra.run.smithi027.stdout:(19/146): python3-ceph-common-18.2.2-1173.g8919 1.2 MB/s | 129 kB 00:00 2024-04-05T23:22:30.371 INFO:teuthology.orchestra.run.smithi130.stdout:(14/146): libcephsqlite-18.2.2-1173.g8919605b.e 1.0 MB/s | 142 kB 00:00 2024-04-05T23:22:30.440 INFO:teuthology.orchestra.run.smithi027.stdout:(20/146): python3-cephfs-18.2.2-1173.g8919605b. 1.8 MB/s | 196 kB 00:00 2024-04-05T23:22:30.471 INFO:teuthology.orchestra.run.smithi130.stdout:(15/146): librados-devel-18.2.2-1173.g8919605b. 1.4 MB/s | 139 kB 00:00 2024-04-05T23:22:30.661 INFO:teuthology.orchestra.run.smithi027.stdout:(21/146): python3-rados-18.2.2-1173.g8919605b.e 1.6 MB/s | 372 kB 00:00 2024-04-05T23:22:30.780 INFO:teuthology.orchestra.run.smithi130.stdout:(16/146): libradosstriper1-18.2.2-1173.g8919605 1.5 MB/s | 475 kB 00:00 2024-04-05T23:22:30.815 INFO:teuthology.orchestra.run.smithi027.stdout:(22/146): python3-rbd-18.2.2-1173.g8919605b.el8 2.3 MB/s | 364 kB 00:00 2024-04-05T23:22:30.907 INFO:teuthology.orchestra.run.smithi027.stdout:(23/146): python3-rgw-18.2.2-1173.g8919605b.el8 1.1 MB/s | 107 kB 00:00 2024-04-05T23:22:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:(24/146): rbd-fuse-18.2.2-1173.g8919605b.el8.x8 1.0 MB/s | 95 kB 00:00 2024-04-05T23:22:31.682 INFO:teuthology.orchestra.run.smithi016.stdout:(10/146): ceph-common-18.2.2-1173.g8919605b.el8 2.1 MB/s | 22 MB 00:10 2024-04-05T23:22:31.722 INFO:teuthology.orchestra.run.smithi130.stdout:(17/146): ceph-radosgw-18.2.2-1173.g8919605b.el 1.2 MB/s | 13 MB 00:10 2024-04-05T23:22:31.839 INFO:teuthology.orchestra.run.smithi130.stdout:(18/146): python3-ceph-argparse-18.2.2-1173.g89 383 kB/s | 45 kB 00:00 2024-04-05T23:22:31.941 INFO:teuthology.orchestra.run.smithi016.stdout:(11/146): cephfs-mirror-18.2.2-1173.g8919605b.e 1.0 MB/s | 261 kB 00:00 2024-04-05T23:22:32.033 INFO:teuthology.orchestra.run.smithi016.stdout:(12/146): libcephfs-devel-18.2.2-1173.g8919605b 338 kB/s | 31 kB 00:00 2024-04-05T23:22:32.039 INFO:teuthology.orchestra.run.smithi130.stdout:(19/146): python3-ceph-common-18.2.2-1173.g8919 644 kB/s | 129 kB 00:00 2024-04-05T23:22:32.331 INFO:teuthology.orchestra.run.smithi130.stdout:(20/146): python3-cephfs-18.2.2-1173.g8919605b. 674 kB/s | 196 kB 00:00 2024-04-05T23:22:32.391 INFO:teuthology.orchestra.run.smithi016.stdout:(13/146): libcephfs2-18.2.2-1173.g8919605b.el8. 2.1 MB/s | 781 kB 00:00 2024-04-05T23:22:32.468 INFO:teuthology.orchestra.run.smithi027.stdout:(25/146): rbd-mirror-18.2.2-1173.g8919605b.el8. 2.4 MB/s | 3.5 MB 00:01 2024-04-05T23:22:32.475 INFO:teuthology.orchestra.run.smithi016.stdout:(14/146): libcephsqlite-18.2.2-1173.g8919605b.e 1.7 MB/s | 142 kB 00:00 2024-04-05T23:22:32.550 INFO:teuthology.orchestra.run.smithi027.stdout:(26/146): librgw2-18.2.2-1173.g8919605b.el8.x86 2.6 MB/s | 6.5 MB 00:02 2024-04-05T23:22:32.567 INFO:teuthology.orchestra.run.smithi016.stdout:(15/146): librados-devel-18.2.2-1173.g8919605b. 1.5 MB/s | 139 kB 00:00 2024-04-05T23:22:32.642 INFO:teuthology.orchestra.run.smithi027.stdout:(27/146): rbd-nbd-18.2.2-1173.g8919605b.el8.x86 1.0 MB/s | 181 kB 00:00 2024-04-05T23:22:32.667 INFO:teuthology.orchestra.run.smithi027.stdout:(28/146): ceph-grafana-dashboards-18.2.2-1173.g 202 kB/s | 24 kB 00:00 2024-04-05T23:22:32.734 INFO:teuthology.orchestra.run.smithi027.stdout:(29/146): ceph-mgr-cephadm-18.2.2-1173.g8919605 1.5 MB/s | 138 kB 00:00 2024-04-05T23:22:32.798 INFO:teuthology.orchestra.run.smithi130.stdout:(21/146): python3-rados-18.2.2-1173.g8919605b.e 798 kB/s | 372 kB 00:00 2024-04-05T23:22:32.800 INFO:teuthology.orchestra.run.smithi016.stdout:(16/146): libradosstriper1-18.2.2-1173.g8919605 2.0 MB/s | 475 kB 00:00 2024-04-05T23:22:33.265 INFO:teuthology.orchestra.run.smithi130.stdout:(22/146): python3-rbd-18.2.2-1173.g8919605b.el8 781 kB/s | 364 kB 00:00 2024-04-05T23:22:33.448 INFO:teuthology.orchestra.run.smithi130.stdout:(23/146): python3-rgw-18.2.2-1173.g8919605b.el8 582 kB/s | 107 kB 00:00 2024-04-05T23:22:33.682 INFO:teuthology.orchestra.run.smithi130.stdout:(24/146): librgw2-18.2.2-1173.g8919605b.el8.x86 2.3 MB/s | 6.5 MB 00:02 2024-04-05T23:22:33.707 INFO:teuthology.orchestra.run.smithi130.stdout:(25/146): rbd-fuse-18.2.2-1173.g8919605b.el8.x8 367 kB/s | 95 kB 00:00 2024-04-05T23:22:34.057 INFO:teuthology.orchestra.run.smithi130.stdout:(26/146): rbd-nbd-18.2.2-1173.g8919605b.el8.x86 517 kB/s | 181 kB 00:00 2024-04-05T23:22:34.191 INFO:teuthology.orchestra.run.smithi130.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1173.g 177 kB/s | 24 kB 00:00 2024-04-05T23:22:34.226 INFO:teuthology.orchestra.run.smithi027.stdout:(30/146): ceph-mgr-dashboard-18.2.2-1173.g89196 2.3 MB/s | 3.5 MB 00:01 2024-04-05T23:22:34.383 INFO:teuthology.orchestra.run.smithi130.stdout:(28/146): ceph-mgr-cephadm-18.2.2-1173.g8919605 721 kB/s | 138 kB 00:00 2024-04-05T23:22:34.418 INFO:teuthology.orchestra.run.smithi027.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1173.g89 1.3 MB/s | 245 kB 00:00 2024-04-05T23:22:34.543 INFO:teuthology.orchestra.run.smithi027.stdout:(32/146): ceph-mgr-rook-18.2.2-1173.g8919605b.e 392 kB/s | 49 kB 00:00 2024-04-05T23:22:34.660 INFO:teuthology.orchestra.run.smithi027.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1173.g8 126 kB/s | 15 kB 00:00 2024-04-05T23:22:35.010 INFO:teuthology.orchestra.run.smithi027.stdout:(34/146): ceph-volume-18.2.2-1173.g8919605b.el8 752 kB/s | 263 kB 00:00 2024-04-05T23:22:35.150 INFO:teuthology.orchestra.run.smithi130.stdout:(29/146): rbd-mirror-18.2.2-1173.g8919605b.el8. 2.4 MB/s | 3.5 MB 00:01 2024-04-05T23:22:35.236 INFO:teuthology.orchestra.run.smithi027.stdout:(35/146): cephadm-18.2.2-1173.g8919605b.el8.noa 768 kB/s | 173 kB 00:00 2024-04-05T23:22:35.415 INFO:teuthology.orchestra.run.smithi027.stdout:(36/146): cephfs-top-18.2.2-1173.g8919605b.el8. 101 kB/s | 18 kB 00:00 2024-04-05T23:22:35.576 INFO:teuthology.orchestra.run.smithi016.stdout:(17/146): librgw2-18.2.2-1173.g8919605b.el8.x86 2.4 MB/s | 6.5 MB 00:02 2024-04-05T23:22:35.768 INFO:teuthology.orchestra.run.smithi016.stdout:(18/146): python3-ceph-argparse-18.2.2-1173.g89 232 kB/s | 45 kB 00:00 2024-04-05T23:22:35.851 INFO:teuthology.orchestra.run.smithi016.stdout:(19/146): python3-ceph-common-18.2.2-1173.g8919 1.5 MB/s | 129 kB 00:00 2024-04-05T23:22:35.903 INFO:teuthology.orchestra.run.smithi027.stdout:(37/146): ceph-mgr-diskprediction-local-18.2.2- 2.3 MB/s | 7.4 MB 00:03 2024-04-05T23:22:35.928 INFO:teuthology.orchestra.run.smithi027.stdout:(38/146): gperftools-libs-2.7-9.el8.x86_64.rpm 597 kB/s | 306 kB 00:00 2024-04-05T23:22:35.962 INFO:teuthology.orchestra.run.smithi027.stdout:(39/146): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-04-05T23:22:36.010 INFO:teuthology.orchestra.run.smithi016.stdout:(20/146): python3-cephfs-18.2.2-1173.g8919605b. 1.2 MB/s | 196 kB 00:00 2024-04-05T23:22:36.162 INFO:teuthology.orchestra.run.smithi027.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 321 kB/s | 75 kB 00:00 2024-04-05T23:22:36.227 INFO:teuthology.orchestra.run.smithi016.stdout:(21/146): python3-rados-18.2.2-1173.g8919605b.e 1.7 MB/s | 372 kB 00:00 2024-04-05T23:22:36.237 INFO:teuthology.orchestra.run.smithi027.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 583 kB/s | 44 kB 00:00 2024-04-05T23:22:36.471 INFO:teuthology.orchestra.run.smithi027.stdout:(42/146): python3-asyncssh-2.7.0-2.el8.noarch.r 935 kB/s | 476 kB 00:00 2024-04-05T23:22:36.496 INFO:teuthology.orchestra.run.smithi027.stdout:(43/146): python3-beautifulsoup4-4.6.3-2.el8.1. 716 kB/s | 185 kB 00:00 2024-04-05T23:22:36.555 INFO:teuthology.orchestra.run.smithi027.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 394 kB/s | 33 kB 00:00 2024-04-05T23:22:36.588 INFO:teuthology.orchestra.run.smithi027.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 178 kB/s | 16 kB 00:00 2024-04-05T23:22:36.702 INFO:teuthology.orchestra.run.smithi016.stdout:(22/146): ceph-test-18.2.2-1173.g8919605b.el8.x 5.2 MB/s | 47 MB 00:09 2024-04-05T23:22:36.705 INFO:teuthology.orchestra.run.smithi027.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 1.1 MB/s | 173 kB 00:00 2024-04-05T23:22:36.814 INFO:teuthology.orchestra.run.smithi027.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 750 kB/s | 81 kB 00:00 2024-04-05T23:22:36.873 INFO:teuthology.orchestra.run.smithi027.stdout:(48/146): python3-cherrypy-18.4.0-1.el8.noarch. 1.3 MB/s | 384 kB 00:00 2024-04-05T23:22:36.894 INFO:teuthology.orchestra.run.smithi016.stdout:(23/146): ceph-osd-18.2.2-1173.g8919605b.el8.x8 1.5 MB/s | 18 MB 00:12 2024-04-05T23:22:36.898 INFO:teuthology.orchestra.run.smithi027.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2024-04-05T23:22:36.919 INFO:teuthology.orchestra.run.smithi016.stdout:(24/146): python3-rgw-18.2.2-1173.g8919605b.el8 493 kB/s | 107 kB 00:00 2024-04-05T23:22:36.932 INFO:teuthology.orchestra.run.smithi027.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 301 kB/s | 18 kB 00:00 2024-04-05T23:22:36.945 INFO:teuthology.orchestra.run.smithi016.stdout:(25/146): python3-rbd-18.2.2-1173.g8919605b.el8 508 kB/s | 364 kB 00:00 2024-04-05T23:22:36.995 INFO:teuthology.orchestra.run.smithi016.stdout:(26/146): rbd-fuse-18.2.2-1173.g8919605b.el8.x8 946 kB/s | 95 kB 00:00 2024-04-05T23:22:36.999 INFO:teuthology.orchestra.run.smithi027.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 732 kB/s | 49 kB 00:00 2024-04-05T23:22:37.050 INFO:teuthology.orchestra.run.smithi130.stdout:(30/146): ceph-test-18.2.2-1173.g8919605b.el8.x 3.7 MB/s | 47 MB 00:12 2024-04-05T23:22:37.054 INFO:teuthology.orchestra.run.smithi016.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1173.g 408 kB/s | 24 kB 00:00 2024-04-05T23:22:37.121 INFO:teuthology.orchestra.run.smithi016.stdout:(28/146): rbd-nbd-18.2.2-1173.g8919605b.el8.x86 1.0 MB/s | 181 kB 00:00 2024-04-05T23:22:37.124 INFO:teuthology.orchestra.run.smithi027.stdout:(52/146): python3-more-itertools-7.2.0-3.el8.no 470 kB/s | 59 kB 00:00 2024-04-05T23:22:37.184 INFO:teuthology.orchestra.run.smithi130.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1173.g89 1.8 MB/s | 245 kB 00:00 2024-04-05T23:22:37.205 INFO:teuthology.orchestra.run.smithi016.stdout:(29/146): ceph-mgr-cephadm-18.2.2-1173.g8919605 919 kB/s | 138 kB 00:00 2024-04-05T23:22:37.232 INFO:teuthology.orchestra.run.smithi027.stdout:(53/146): python3-natsort-7.1.1-2.el8.noarch.rp 557 kB/s | 60 kB 00:00 2024-04-05T23:22:37.259 INFO:teuthology.orchestra.run.smithi130.stdout:(32/146): ceph-mgr-rook-18.2.2-1173.g8919605b.e 657 kB/s | 49 kB 00:00 2024-04-05T23:22:37.334 INFO:teuthology.orchestra.run.smithi130.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1173.g8 197 kB/s | 15 kB 00:00 2024-04-05T23:22:37.449 INFO:teuthology.orchestra.run.smithi027.stdout:(54/146): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-04-05T23:22:37.500 INFO:teuthology.orchestra.run.smithi027.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2024-04-05T23:22:37.501 INFO:teuthology.orchestra.run.smithi130.stdout:(34/146): ceph-volume-18.2.2-1173.g8919605b.el8 1.5 MB/s | 263 kB 00:00 2024-04-05T23:22:37.550 INFO:teuthology.orchestra.run.smithi027.stdout:(56/146): python3-repoze-lru-0.7-6.el8.noarch.r 675 kB/s | 34 kB 00:00 2024-04-05T23:22:37.585 INFO:teuthology.orchestra.run.smithi130.stdout:(35/146): ceph-mgr-diskprediction-local-18.2.2- 3.0 MB/s | 7.4 MB 00:02 2024-04-05T23:22:37.588 INFO:teuthology.orchestra.run.smithi016.stdout:(30/146): rbd-mirror-18.2.2-1173.g8919605b.el8. 5.2 MB/s | 3.5 MB 00:00 2024-04-05T23:22:37.610 INFO:teuthology.orchestra.run.smithi130.stdout:(36/146): cephadm-18.2.2-1173.g8919605b.el8.noa 1.6 MB/s | 173 kB 00:00 2024-04-05T23:22:37.634 INFO:teuthology.orchestra.run.smithi027.stdout:(57/146): python3-kubernetes-11.0.0-6.el8.noarc 1.6 MB/s | 1.2 MB 00:00 2024-04-05T23:22:37.635 INFO:teuthology.orchestra.run.smithi130.stdout:(37/146): cephfs-top-18.2.2-1173.g8919605b.el8. 362 kB/s | 18 kB 00:00 2024-04-05T23:22:37.680 INFO:teuthology.orchestra.run.smithi016.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1173.g89 2.6 MB/s | 245 kB 00:00 2024-04-05T23:22:37.743 INFO:teuthology.orchestra.run.smithi027.stdout:(58/146): python3-rsa-4.9-2.el8.noarch.rpm 591 kB/s | 64 kB 00:00 2024-04-05T23:22:37.747 INFO:teuthology.orchestra.run.smithi016.stdout:(32/146): ceph-mgr-rook-18.2.2-1173.g8919605b.e 739 kB/s | 49 kB 00:00 2024-04-05T23:22:37.768 INFO:teuthology.orchestra.run.smithi027.stdout:(59/146): python3-routes-2.4.1-12.el8.noarch.rp 902 kB/s | 196 kB 00:00 2024-04-05T23:22:37.793 INFO:teuthology.orchestra.run.smithi027.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 371 kB/s | 19 kB 00:00 2024-04-05T23:22:37.794 INFO:teuthology.orchestra.run.smithi130.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 372 kB/s | 59 kB 00:00 2024-04-05T23:22:37.806 INFO:teuthology.orchestra.run.smithi016.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1173.g8 254 kB/s | 15 kB 00:00 2024-04-05T23:22:37.819 INFO:teuthology.orchestra.run.smithi027.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2024-04-05T23:22:37.844 INFO:teuthology.orchestra.run.smithi027.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 582 kB/s | 29 kB 00:00 2024-04-05T23:22:37.852 INFO:teuthology.orchestra.run.smithi130.stdout:(39/146): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-04-05T23:22:37.869 INFO:teuthology.orchestra.run.smithi027.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 550 kB/s | 27 kB 00:00 2024-04-05T23:22:37.878 INFO:teuthology.orchestra.run.smithi130.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 900 kB/s | 75 kB 00:00 2024-04-05T23:22:37.881 INFO:teuthology.orchestra.run.smithi016.stdout:(34/146): ceph-volume-18.2.2-1173.g8919605b.el8 3.4 MB/s | 263 kB 00:00 2024-04-05T23:22:37.920 INFO:teuthology.orchestra.run.smithi130.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2024-04-05T23:22:37.956 INFO:teuthology.orchestra.run.smithi016.stdout:(35/146): cephadm-18.2.2-1173.g8919605b.el8.noa 2.3 MB/s | 173 kB 00:00 2024-04-05T23:22:37.978 INFO:teuthology.orchestra.run.smithi027.stdout:(64/146): python3-waitress-1.4.3-1.el8.noarch.r 1.9 MB/s | 255 kB 00:00 2024-04-05T23:22:37.986 INFO:teuthology.orchestra.run.smithi130.stdout:(42/146): python3-asyncssh-2.7.0-2.el8.noarch.r 3.5 MB/s | 476 kB 00:00 2024-04-05T23:22:38.015 INFO:teuthology.orchestra.run.smithi016.stdout:(36/146): cephfs-top-18.2.2-1173.g8919605b.el8. 311 kB/s | 18 kB 00:00 2024-04-05T23:22:38.028 INFO:teuthology.orchestra.run.smithi027.stdout:(65/146): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2024-04-05T23:22:38.045 INFO:teuthology.orchestra.run.smithi130.stdout:(43/146): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2024-04-05T23:22:38.062 INFO:teuthology.orchestra.run.smithi027.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 1.3 MB/s | 251 kB 00:00 2024-04-05T23:22:38.070 INFO:teuthology.orchestra.run.smithi130.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 395 kB/s | 33 kB 00:00 2024-04-05T23:22:38.095 INFO:teuthology.orchestra.run.smithi027.stdout:(67/146): python3-webtest-2.0.33-1.el8.noarch.r 1.2 MB/s | 85 kB 00:00 2024-04-05T23:22:38.096 INFO:teuthology.orchestra.run.smithi130.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 325 kB/s | 16 kB 00:00 2024-04-05T23:22:38.121 INFO:teuthology.orchestra.run.smithi027.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 392 kB/s | 23 kB 00:00 2024-04-05T23:22:38.146 INFO:teuthology.orchestra.run.smithi027.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2024-04-05T23:22:38.215 INFO:teuthology.orchestra.run.smithi016.stdout:(37/146): gperftools-libs-2.7-9.el8.x86_64.rpm 1.5 MB/s | 306 kB 00:00 2024-04-05T23:22:38.221 INFO:teuthology.orchestra.run.smithi130.stdout:(46/146): python3-cherrypy-18.4.0-1.el8.noarch. 3.0 MB/s | 384 kB 00:00 2024-04-05T23:22:38.263 INFO:teuthology.orchestra.run.smithi027.stdout:(70/146): python3-jsonpointer-1.10-11.el8.noarc 172 kB/s | 20 kB 00:00 2024-04-05T23:22:38.265 INFO:teuthology.orchestra.run.smithi016.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2024-04-05T23:22:38.288 INFO:teuthology.orchestra.run.smithi130.stdout:(47/146): ceph-mgr-dashboard-18.2.2-1173.g89196 925 kB/s | 3.5 MB 00:03 2024-04-05T23:22:38.313 INFO:teuthology.orchestra.run.smithi130.stdout:(48/146): python3-cheroot-8.5.2-1.el8.noarch.rp 712 kB/s | 173 kB 00:00 2024-04-05T23:22:38.338 INFO:teuthology.orchestra.run.smithi130.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2024-04-05T23:22:38.349 INFO:teuthology.orchestra.run.smithi016.stdout:(39/146): ceph-mgr-dashboard-18.2.2-1173.g89196 2.9 MB/s | 3.5 MB 00:01 2024-04-05T23:22:38.364 INFO:teuthology.orchestra.run.smithi130.stdout:(50/146): python3-google-auth-1.1.1-10.el8.noar 571 kB/s | 81 kB 00:00 2024-04-05T23:22:38.374 INFO:teuthology.orchestra.run.smithi016.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 692 kB/s | 75 kB 00:00 2024-04-05T23:22:38.389 INFO:teuthology.orchestra.run.smithi130.stdout:(51/146): python3-jaraco-functools-2.0-4.el8.no 234 kB/s | 18 kB 00:00 2024-04-05T23:22:38.414 INFO:teuthology.orchestra.run.smithi130.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 1.0 MB/s | 49 kB 00:00 2024-04-05T23:22:38.430 INFO:teuthology.orchestra.run.smithi027.stdout:(71/146): python3-prettytable-0.7.2-14.el8.noar 264 kB/s | 44 kB 00:00 2024-04-05T23:22:38.433 INFO:teuthology.orchestra.run.smithi016.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 815 kB/s | 44 kB 00:00 2024-04-05T23:22:38.492 INFO:teuthology.orchestra.run.smithi016.stdout:(42/146): python3-beautifulsoup4-4.6.3-2.el8.1. 3.1 MB/s | 185 kB 00:00 2024-04-05T23:22:38.514 INFO:teuthology.orchestra.run.smithi130.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 471 kB/s | 59 kB 00:00 2024-04-05T23:22:38.542 INFO:teuthology.orchestra.run.smithi016.stdout:(43/146): python3-cachetools-3.1.1-4.el8.noarch 664 kB/s | 33 kB 00:00 2024-04-05T23:22:38.556 INFO:teuthology.orchestra.run.smithi130.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 426 kB/s | 60 kB 00:00 2024-04-05T23:22:38.563 INFO:teuthology.orchestra.run.smithi027.stdout:(72/146): python3-pyasn1-modules-0.3.7-6.el8.no 827 kB/s | 110 kB 00:00 2024-04-05T23:22:38.615 INFO:teuthology.orchestra.run.smithi130.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 268 kB/s | 16 kB 00:00 2024-04-05T23:22:38.617 INFO:teuthology.orchestra.run.smithi016.stdout:(44/146): python3-asyncssh-2.7.0-2.el8.noarch.r 1.7 MB/s | 476 kB 00:00 2024-04-05T23:22:38.643 INFO:teuthology.orchestra.run.smithi016.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 162 kB/s | 16 kB 00:00 2024-04-05T23:22:38.680 INFO:teuthology.orchestra.run.smithi016.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2024-04-05T23:22:38.735 INFO:teuthology.orchestra.run.smithi016.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2024-04-05T23:22:38.748 INFO:teuthology.orchestra.run.smithi130.stdout:(56/146): python3-repoze-lru-0.7-6.el8.noarch.r 252 kB/s | 34 kB 00:00 2024-04-05T23:22:38.760 INFO:teuthology.orchestra.run.smithi016.stdout:(48/146): python3-cherrypy-18.4.0-1.el8.noarch. 3.2 MB/s | 384 kB 00:00 2024-04-05T23:22:38.782 INFO:teuthology.orchestra.run.smithi130.stdout:(57/146): python3-kubernetes-11.0.0-6.el8.noarc 2.6 MB/s | 1.2 MB 00:00 2024-04-05T23:22:38.785 INFO:teuthology.orchestra.run.smithi016.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 223 kB/s | 11 kB 00:00 2024-04-05T23:22:38.811 INFO:teuthology.orchestra.run.smithi016.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 350 kB/s | 18 kB 00:00 2024-04-05T23:22:38.832 INFO:teuthology.orchestra.run.smithi130.stdout:(58/146): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2024-04-05T23:22:38.869 INFO:teuthology.orchestra.run.smithi016.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 840 kB/s | 49 kB 00:00 2024-04-05T23:22:38.911 INFO:teuthology.orchestra.run.smithi016.stdout:(52/146): python3-kubernetes-11.0.0-6.el8.noarc 9.3 MB/s | 1.2 MB 00:00 2024-04-05T23:22:38.924 INFO:teuthology.orchestra.run.smithi130.stdout:(59/146): python3-simplegeneric-0.8.1-17.el8.no 204 kB/s | 19 kB 00:00 2024-04-05T23:22:38.937 INFO:teuthology.orchestra.run.smithi016.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 877 kB/s | 59 kB 00:00 2024-04-05T23:22:38.971 INFO:teuthology.orchestra.run.smithi016.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2024-04-05T23:22:38.983 INFO:teuthology.orchestra.run.smithi130.stdout:(60/146): python3-singledispatch-3.4.0.3-18.el8 411 kB/s | 24 kB 00:00 2024-04-05T23:22:39.004 INFO:teuthology.orchestra.run.smithi016.stdout:(55/146): python3-pecan-1.3.2-9.el8.noarch.rpm 4.1 MB/s | 283 kB 00:00 2024-04-05T23:22:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 268 kB/s | 16 kB 00:00 2024-04-05T23:22:39.042 INFO:teuthology.orchestra.run.smithi130.stdout:(61/146): python3-pecan-1.3.2-9.el8.noarch.rpm 537 kB/s | 283 kB 00:00 2024-04-05T23:22:39.067 INFO:teuthology.orchestra.run.smithi130.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 349 kB/s | 29 kB 00:00 2024-04-05T23:22:39.097 INFO:teuthology.orchestra.run.smithi027.stdout:(73/146): ceph-test-18.2.2-1173.g8919605b.el8.x 4.0 MB/s | 47 MB 00:11 2024-04-05T23:22:39.117 INFO:teuthology.orchestra.run.smithi130.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 366 kB/s | 27 kB 00:00 2024-04-05T23:22:39.131 INFO:teuthology.orchestra.run.smithi027.stdout:(74/146): thrift-0.13.0-2.el8.x86_64.rpm 1.7 MB/s | 1.7 MB 00:01 2024-04-05T23:22:39.138 INFO:teuthology.orchestra.run.smithi016.stdout:(57/146): ceph-mgr-diskprediction-local-18.2.2- 3.8 MB/s | 7.4 MB 00:01 2024-04-05T23:22:39.163 INFO:teuthology.orchestra.run.smithi016.stdout:(58/146): python3-repoze-lru-0.7-6.el8.noarch.r 212 kB/s | 34 kB 00:00 2024-04-05T23:22:39.164 INFO:teuthology.orchestra.run.smithi027.stdout:(75/146): python3-jsonpatch-1.21-2.el8.noarch.r 44 kB/s | 27 kB 00:00 2024-04-05T23:22:39.189 INFO:teuthology.orchestra.run.smithi016.stdout:(59/146): python3-routes-2.4.1-12.el8.noarch.rp 1.2 MB/s | 196 kB 00:00 2024-04-05T23:22:39.217 INFO:teuthology.orchestra.run.smithi130.stdout:(64/146): python3-routes-2.4.1-12.el8.noarch.rp 419 kB/s | 196 kB 00:00 2024-04-05T23:22:39.222 INFO:teuthology.orchestra.run.smithi016.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 319 kB/s | 19 kB 00:00 2024-04-05T23:22:39.247 INFO:teuthology.orchestra.run.smithi016.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 448 kB/s | 24 kB 00:00 2024-04-05T23:22:39.273 INFO:teuthology.orchestra.run.smithi016.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 585 kB/s | 29 kB 00:00 2024-04-05T23:22:39.298 INFO:teuthology.orchestra.run.smithi027.stdout:(76/146): python3-pyasn1-0.3.7-6.el8.noarch.rpm 632 kB/s | 126 kB 00:00 2024-04-05T23:22:39.298 INFO:teuthology.orchestra.run.smithi016.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 556 kB/s | 27 kB 00:00 2024-04-05T23:22:39.332 INFO:teuthology.orchestra.run.smithi016.stdout:(64/146): python3-waitress-1.4.3-1.el8.noarch.r 4.3 MB/s | 255 kB 00:00 2024-04-05T23:22:39.334 INFO:teuthology.orchestra.run.smithi130.stdout:(65/146): python3-websocket-client-0.56.0-5.el8 525 kB/s | 61 kB 00:00 2024-04-05T23:22:39.365 INFO:teuthology.orchestra.run.smithi027.stdout:(77/146): bison-3.0.4-10.el8.x86_64.rpm 3.5 MB/s | 688 kB 00:00 2024-04-05T23:22:39.365 INFO:teuthology.orchestra.run.smithi016.stdout:(65/146): python3-webob-1.8.5-1.el8.1.noarch.rp 3.7 MB/s | 251 kB 00:00 2024-04-05T23:22:39.391 INFO:teuthology.orchestra.run.smithi016.stdout:(66/146): python3-rsa-4.9-2.el8.noarch.rpm 255 kB/s | 64 kB 00:00 2024-04-05T23:22:39.399 INFO:teuthology.orchestra.run.smithi027.stdout:(78/146): python3-werkzeug-0.12.2-4.el8.noarch. 1.7 MB/s | 457 kB 00:00 2024-04-05T23:22:39.416 INFO:teuthology.orchestra.run.smithi016.stdout:(67/146): python3-websocket-client-0.56.0-5.el8 733 kB/s | 61 kB 00:00 2024-04-05T23:22:39.441 INFO:teuthology.orchestra.run.smithi016.stdout:(68/146): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2024-04-05T23:22:39.449 INFO:teuthology.orchestra.run.smithi027.stdout:(79/146): flex-2.6.1-9.el8.x86_64.rpm 2.1 MB/s | 320 kB 00:00 2024-04-05T23:22:39.467 INFO:teuthology.orchestra.run.smithi016.stdout:(69/146): python3-zc-lockfile-2.0-2.el8.noarch. 304 kB/s | 23 kB 00:00 2024-04-05T23:22:39.492 INFO:teuthology.orchestra.run.smithi016.stdout:(70/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2024-04-05T23:22:39.499 INFO:teuthology.orchestra.run.smithi027.stdout:(80/146): libtool-2.4.6-25.el8.x86_64.rpm 5.2 MB/s | 709 kB 00:00 2024-04-05T23:22:39.509 INFO:teuthology.orchestra.run.smithi130.stdout:(66/146): python3-waitress-1.4.3-1.el8.noarch.r 577 kB/s | 255 kB 00:00 2024-04-05T23:22:39.524 INFO:teuthology.orchestra.run.smithi027.stdout:(81/146): python3-markupsafe-0.23-19.el8.x86_64 309 kB/s | 39 kB 00:00 2024-04-05T23:22:39.535 INFO:teuthology.orchestra.run.smithi130.stdout:(67/146): python3-webtest-2.0.33-1.el8.noarch.r 426 kB/s | 85 kB 00:00 2024-04-05T23:22:39.576 INFO:teuthology.orchestra.run.smithi130.stdout:(68/146): python3-webob-1.8.5-1.el8.1.noarch.rp 547 kB/s | 251 kB 00:00 2024-04-05T23:22:39.601 INFO:teuthology.orchestra.run.smithi016.stdout:(71/146): thrift-0.13.0-2.el8.x86_64.rpm 9.4 MB/s | 1.7 MB 00:00 2024-04-05T23:22:39.608 INFO:teuthology.orchestra.run.smithi027.stdout:(82/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.6 MB/s | 259 kB 00:00 2024-04-05T23:22:39.610 INFO:teuthology.orchestra.run.smithi130.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 70 kB 00:00 2024-04-05T23:22:39.635 INFO:teuthology.orchestra.run.smithi130.stdout:(70/146): python3-zc-lockfile-2.0-2.el8.noarch. 183 kB/s | 23 kB 00:00 2024-04-05T23:22:39.659 INFO:teuthology.orchestra.run.smithi016.stdout:(72/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 113 kB/s | 18 kB 00:00 2024-04-05T23:22:39.691 INFO:teuthology.orchestra.run.smithi027.stdout:(83/146): uuid-1.6.2-43.el8.x86_64.rpm 333 kB/s | 64 kB 00:00 2024-04-05T23:22:39.717 INFO:teuthology.orchestra.run.smithi027.stdout:(84/146): python3-pyOpenSSL-19.0.0-1.el8.noarch 539 kB/s | 103 kB 00:00 2024-04-05T23:22:39.734 INFO:teuthology.orchestra.run.smithi016.stdout:(73/146): python3-jwt-1.6.1-2.el8.noarch.rpm 320 kB/s | 43 kB 00:00 2024-04-05T23:22:39.776 INFO:teuthology.orchestra.run.smithi016.stdout:(74/146): python3-packaging-16.8-9.el8.noarch.r 156 kB/s | 48 kB 00:00 2024-04-05T23:22:39.785 INFO:teuthology.orchestra.run.smithi130.stdout:(71/146): python3-jsonpointer-1.10-11.el8.noarc 139 kB/s | 20 kB 00:00 2024-04-05T23:22:39.811 INFO:teuthology.orchestra.run.smithi130.stdout:(72/146): python3-packaging-16.8-9.el8.noarch.r 240 kB/s | 48 kB 00:00 2024-04-05T23:22:39.818 INFO:teuthology.orchestra.run.smithi016.stdout:(75/146): python3-oauthlib-2.1.0-1.el8.noarch.r 975 kB/s | 155 kB 00:00 2024-04-05T23:22:39.842 INFO:teuthology.orchestra.run.smithi027.stdout:(85/146): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2024-04-05T23:22:39.877 INFO:teuthology.orchestra.run.smithi016.stdout:(76/146): python3-pycparser-2.14-14.el8.noarch. 766 kB/s | 109 kB 00:00 2024-04-05T23:22:39.902 INFO:teuthology.orchestra.run.smithi016.stdout:(77/146): python3-requests-oauthlib-1.0.0-1.el8 340 kB/s | 43 kB 00:00 2024-04-05T23:22:39.948 INFO:teuthology.orchestra.run.smithi130.stdout:(73/146): python3-pyasn1-modules-0.3.7-6.el8.no 804 kB/s | 110 kB 00:00 2024-04-05T23:22:39.961 INFO:teuthology.orchestra.run.smithi016.stdout:(78/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 526 kB/s | 44 kB 00:00 2024-04-05T23:22:39.970 INFO:teuthology.orchestra.run.smithi027.stdout:(86/146): python3-numpy-1.14.3-10.el8.x86_64.rp 15 MB/s | 3.7 MB 00:00 2024-04-05T23:22:39.978 INFO:teuthology.orchestra.run.smithi130.stdout:(74/146): python3-prettytable-0.7.2-14.el8.noar 229 kB/s | 44 kB 00:00 2024-04-05T23:22:40.009 INFO:teuthology.orchestra.run.smithi027.stdout:(87/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2024-04-05T23:22:40.011 INFO:teuthology.orchestra.run.smithi130.stdout:(75/146): python3-jsonpatch-1.21-2.el8.noarch.r 422 kB/s | 27 kB 00:00 2024-04-05T23:22:40.013 INFO:teuthology.orchestra.run.smithi016.stdout:(79/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 96 kB/s | 19 kB 00:00 2024-04-05T23:22:40.035 INFO:teuthology.orchestra.run.smithi027.stdout:(88/146): python3-jinja2-2.10.1-3.el8.noarch.rp 1.5 MB/s | 538 kB 00:00 2024-04-05T23:22:40.053 INFO:teuthology.orchestra.run.smithi016.stdout:(80/146): pcre2-devel-10.32-2.el8.x86_64.rpm 3.9 MB/s | 605 kB 00:00 2024-04-05T23:22:40.068 INFO:teuthology.orchestra.run.smithi027.stdout:(89/146): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.2 MB/s | 225 kB 00:00 2024-04-05T23:22:40.089 INFO:teuthology.orchestra.run.smithi016.stdout:(81/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2024-04-05T23:22:40.129 INFO:teuthology.orchestra.run.smithi130.stdout:(76/146): python3-werkzeug-0.12.2-4.el8.noarch. 3.8 MB/s | 457 kB 00:00 2024-04-05T23:22:40.156 INFO:teuthology.orchestra.run.smithi130.stdout:(77/146): python3-pyasn1-0.3.7-6.el8.noarch.rpm 720 kB/s | 126 kB 00:00 2024-04-05T23:22:40.156 INFO:teuthology.orchestra.run.smithi016.stdout:(82/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.5 MB/s | 220 kB 00:00 2024-04-05T23:22:40.198 INFO:teuthology.orchestra.run.smithi016.stdout:(83/146): libselinux-devel-2.9-5.el8.x86_64.rpm 1.4 MB/s | 200 kB 00:00 2024-04-05T23:22:40.202 INFO:teuthology.orchestra.run.smithi027.stdout:(90/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2024-04-05T23:22:40.223 INFO:teuthology.orchestra.run.smithi016.stdout:(84/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 359 kB/s | 48 kB 00:00 2024-04-05T23:22:40.235 INFO:teuthology.orchestra.run.smithi027.stdout:(91/146): librdkafka-0.11.4-3.el8.x86_64.rpm 1.5 MB/s | 354 kB 00:00 2024-04-05T23:22:40.321 INFO:teuthology.orchestra.run.smithi130.stdout:(78/146): flex-2.6.1-9.el8.x86_64.rpm 1.9 MB/s | 320 kB 00:00 2024-04-05T23:22:40.336 INFO:teuthology.orchestra.run.smithi027.stdout:(92/146): openblas-threads-0.3.15-6.el8.x86_64. 17 MB/s | 5.2 MB 00:00 2024-04-05T23:22:40.354 INFO:teuthology.orchestra.run.smithi130.stdout:(79/146): bison-3.0.4-10.el8.x86_64.rpm 3.0 MB/s | 688 kB 00:00 2024-04-05T23:22:40.365 INFO:teuthology.orchestra.run.smithi016.stdout:(85/146): libsepol-devel-2.9-3.el8.x86_64.rpm 417 kB/s | 87 kB 00:00 2024-04-05T23:22:40.390 INFO:teuthology.orchestra.run.smithi016.stdout:(86/146): libcom_err-devel-1.45.6-4.el8.x86_64. 231 kB/s | 39 kB 00:00 2024-04-05T23:22:40.415 INFO:teuthology.orchestra.run.smithi016.stdout:(87/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 265 kB/s | 58 kB 00:00 2024-04-05T23:22:40.453 INFO:teuthology.orchestra.run.smithi027.stdout:(93/146): openblas-0.3.15-6.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-04-05T23:22:40.457 INFO:teuthology.orchestra.run.smithi016.stdout:(88/146): libgfortran-8.5.0-20.el8.x86_64.rpm 9.4 MB/s | 645 kB 00:00 2024-04-05T23:22:40.495 INFO:teuthology.orchestra.run.smithi027.stdout:(94/146): automake-1.16.1-8.el8.noarch.rpm 2.7 MB/s | 714 kB 00:00 2024-04-05T23:22:40.504 INFO:teuthology.orchestra.run.smithi130.stdout:(80/146): python3-markupsafe-0.23-19.el8.x86_64 259 kB/s | 39 kB 00:00 2024-04-05T23:22:40.508 INFO:teuthology.orchestra.run.smithi016.stdout:(89/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.8 MB/s | 172 kB 00:00 2024-04-05T23:22:40.525 INFO:teuthology.orchestra.run.smithi016.stdout:(90/146): fuse-2.9.7-17.el8.x86_64.rpm 521 kB/s | 83 kB 00:00 2024-04-05T23:22:40.545 INFO:teuthology.orchestra.run.smithi027.stdout:(95/146): libpq-13.11-1.el8.x86_64.rpm 948 kB/s | 198 kB 00:00 2024-04-05T23:22:40.546 INFO:teuthology.orchestra.run.smithi130.stdout:(81/146): libtool-2.4.6-25.el8.x86_64.rpm 3.1 MB/s | 709 kB 00:00 2024-04-05T23:22:40.558 INFO:teuthology.orchestra.run.smithi016.stdout:(91/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 993 kB/s | 100 kB 00:00 2024-04-05T23:22:40.600 INFO:teuthology.orchestra.run.smithi016.stdout:(92/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2024-04-05T23:22:40.620 INFO:teuthology.orchestra.run.smithi027.stdout:(96/146): jq-1.6-7.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2024-04-05T23:22:40.646 INFO:teuthology.orchestra.run.smithi027.stdout:(97/146): python3-pytz-2017.2-11.el8.noarch.rpm 360 kB/s | 54 kB 00:00 2024-04-05T23:22:40.680 INFO:teuthology.orchestra.run.smithi130.stdout:(82/146): uuid-1.6.2-43.el8.x86_64.rpm 480 kB/s | 64 kB 00:00 2024-04-05T23:22:40.722 INFO:teuthology.orchestra.run.smithi130.stdout:(83/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-04-05T23:22:40.734 INFO:teuthology.orchestra.run.smithi016.stdout:(93/146): NetworkManager-initscripts-updown-1.4 822 kB/s | 144 kB 00:00 2024-04-05T23:22:40.754 INFO:teuthology.orchestra.run.smithi027.stdout:(98/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-04-05T23:22:40.759 INFO:teuthology.orchestra.run.smithi016.stdout:(94/146): elfutils-libelf-devel-0.189-3.el8.x86 389 kB/s | 62 kB 00:00 2024-04-05T23:22:40.814 INFO:teuthology.orchestra.run.smithi130.stdout:(84/146): python3-pyOpenSSL-19.0.0-1.el8.noarch 775 kB/s | 103 kB 00:00 2024-04-05T23:22:40.846 INFO:teuthology.orchestra.run.smithi027.stdout:(99/146): postgresql-10.23-4.module+el8.9.0+213 7.5 MB/s | 1.5 MB 00:00 2024-04-05T23:22:40.868 INFO:teuthology.orchestra.run.smithi016.stdout:(95/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 212 kB/s | 73 kB 00:00 2024-04-05T23:22:40.910 INFO:teuthology.orchestra.run.smithi016.stdout:(96/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 3.1 MB/s | 562 kB 00:00 2024-04-05T23:22:40.948 INFO:teuthology.orchestra.run.smithi130.stdout:(85/146): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2024-04-05T23:22:40.971 INFO:teuthology.orchestra.run.smithi027.stdout:(100/146): postgresql-contrib-10.23-4.module+el 3.7 MB/s | 812 kB 00:00 2024-04-05T23:22:40.981 INFO:teuthology.orchestra.run.smithi130.stdout:(86/146): python3-jinja2-2.10.1-3.el8.noarch.rp 3.1 MB/s | 538 kB 00:00 2024-04-05T23:22:41.035 INFO:teuthology.orchestra.run.smithi016.stdout:(97/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 8.4 MB/s | 2.3 MB 00:00 2024-04-05T23:22:41.077 INFO:teuthology.orchestra.run.smithi016.stdout:(98/146): python3-cryptography-3.2.1-7.el8_9.x8 3.3 MB/s | 559 kB 00:00 2024-04-05T23:22:41.103 INFO:teuthology.orchestra.run.smithi016.stdout:(99/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 201 kB/s | 47 kB 00:00 2024-04-05T23:22:41.123 INFO:teuthology.orchestra.run.smithi130.stdout:(87/146): python3-numpy-1.14.3-10.el8.x86_64.rp 21 MB/s | 3.7 MB 00:00 2024-04-05T23:22:41.148 INFO:teuthology.orchestra.run.smithi130.stdout:(88/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2024-04-05T23:22:41.178 INFO:teuthology.orchestra.run.smithi016.stdout:(100/146): python3-jsonpointer-1.10-11.el8.noar 140 kB/s | 20 kB 00:00 2024-04-05T23:22:41.194 INFO:teuthology.orchestra.run.smithi016.stdout:(101/146): python3-prettytable-0.7.2-14.el8.noa 375 kB/s | 44 kB 00:00 2024-04-05T23:22:41.223 INFO:teuthology.orchestra.run.smithi027.stdout:(101/146): python3-scipy-1.0.0-21.module+el8.9. 20 MB/s | 14 MB 00:00 2024-04-05T23:22:41.228 INFO:teuthology.orchestra.run.smithi016.stdout:(102/146): python3-pyasn1-modules-0.3.7-6.el8.n 878 kB/s | 110 kB 00:00 2024-04-05T23:22:41.255 INFO:teuthology.orchestra.run.smithi027.stdout:(102/146): libverto-devel-0.3.0-5.el8.x86_64.rp 63 kB/s | 18 kB 00:00 2024-04-05T23:22:41.290 INFO:teuthology.orchestra.run.smithi130.stdout:(89/146): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2024-04-05T23:22:41.315 INFO:teuthology.orchestra.run.smithi130.stdout:(90/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.1 MB/s | 354 kB 00:00 2024-04-05T23:22:41.339 INFO:teuthology.orchestra.run.smithi027.stdout:(103/146): postgresql-server-10.23-4.module+el8 10 MB/s | 5.1 MB 00:00 2024-04-05T23:22:41.345 INFO:teuthology.orchestra.run.smithi016.stdout:(103/146): python3-werkzeug-0.12.2-4.el8.noarch 3.8 MB/s | 457 kB 00:00 2024-04-05T23:22:41.364 INFO:teuthology.orchestra.run.smithi027.stdout:(104/146): python3-jwt-1.6.1-2.el8.noarch.rpm 302 kB/s | 43 kB 00:00 2024-04-05T23:22:41.395 INFO:teuthology.orchestra.run.smithi016.stdout:(104/146): python3-jsonpatch-1.21-2.el8.noarch. 122 kB/s | 27 kB 00:00 2024-04-05T23:22:41.398 INFO:teuthology.orchestra.run.smithi027.stdout:(105/146): python3-oauthlib-2.1.0-1.el8.noarch. 1.1 MB/s | 155 kB 00:00 2024-04-05T23:22:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:(105/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 472 kB/s | 126 kB 00:00 2024-04-05T23:22:41.498 INFO:teuthology.orchestra.run.smithi027.stdout:(106/146): python3-requests-oauthlib-1.0.0-1.el 320 kB/s | 43 kB 00:00 2024-04-05T23:22:41.507 INFO:teuthology.orchestra.run.smithi130.stdout:(91/146): openblas-threads-0.3.15-6.el8.x86_64. 24 MB/s | 5.2 MB 00:00 2024-04-05T23:22:41.512 INFO:teuthology.orchestra.run.smithi016.stdout:(106/146): flex-2.6.1-9.el8.x86_64.rpm 2.7 MB/s | 320 kB 00:00 2024-04-05T23:22:41.533 INFO:teuthology.orchestra.run.smithi130.stdout:(92/146): python3-mako-1.0.6-14.el8.noarch.rpm 723 kB/s | 157 kB 00:00 2024-04-05T23:22:41.588 INFO:teuthology.orchestra.run.smithi016.stdout:(107/146): libtool-2.4.6-25.el8.x86_64.rpm 5.5 MB/s | 709 kB 00:00 2024-04-05T23:22:41.598 INFO:teuthology.orchestra.run.smithi027.stdout:(107/146): python3-pycparser-2.14-14.el8.noarch 421 kB/s | 109 kB 00:00 2024-04-05T23:22:41.630 INFO:teuthology.orchestra.run.smithi016.stdout:(108/146): python3-markupsafe-0.23-19.el8.x86_6 332 kB/s | 39 kB 00:00 2024-04-05T23:22:41.632 INFO:teuthology.orchestra.run.smithi027.stdout:(108/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 328 kB/s | 44 kB 00:00 2024-04-05T23:22:41.658 INFO:teuthology.orchestra.run.smithi130.stdout:(93/146): automake-1.16.1-8.el8.noarch.rpm 5.6 MB/s | 714 kB 00:00 2024-04-05T23:22:41.672 INFO:teuthology.orchestra.run.smithi016.stdout:(109/146): bison-3.0.4-10.el8.x86_64.rpm 2.1 MB/s | 688 kB 00:00 2024-04-05T23:22:41.682 INFO:teuthology.orchestra.run.smithi027.stdout:(109/146): libaio-devel-0.3.112-1.el8.x86_64.rp 66 kB/s | 19 kB 00:00 2024-04-05T23:22:41.714 INFO:teuthology.orchestra.run.smithi016.stdout:(110/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2024-04-05T23:22:41.725 INFO:teuthology.orchestra.run.smithi130.stdout:(94/146): openblas-0.3.15-6.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-04-05T23:22:41.741 INFO:teuthology.orchestra.run.smithi027.stdout:(110/146): pcre2-devel-10.32-2.el8.x86_64.rpm 4.2 MB/s | 605 kB 00:00 2024-04-05T23:22:41.756 INFO:teuthology.orchestra.run.smithi016.stdout:(111/146): uuid-1.6.2-43.el8.x86_64.rpm 508 kB/s | 64 kB 00:00 2024-04-05T23:22:41.783 INFO:teuthology.orchestra.run.smithi027.stdout:(111/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.5 MB/s | 229 kB 00:00 2024-04-05T23:22:41.883 INFO:teuthology.orchestra.run.smithi027.stdout:(112/146): libselinux-devel-2.9-5.el8.x86_64.rp 1.4 MB/s | 200 kB 00:00 2024-04-05T23:22:41.900 INFO:teuthology.orchestra.run.smithi130.stdout:(95/146): thrift-0.13.0-2.el8.x86_64.rpm 751 kB/s | 1.7 MB 00:02 2024-04-05T23:22:41.917 INFO:teuthology.orchestra.run.smithi027.stdout:(113/146): keyutils-libs-devel-1.5.10-9.el8.x86 360 kB/s | 48 kB 00:00 2024-04-05T23:22:41.926 INFO:teuthology.orchestra.run.smithi130.stdout:(96/146): libpq-13.11-1.el8.x86_64.rpm 740 kB/s | 198 kB 00:00 2024-04-05T23:22:41.931 INFO:teuthology.orchestra.run.smithi016.stdout:(112/146): python3-jinja2-2.10.1-3.el8.noarch.r 3.0 MB/s | 538 kB 00:00 2024-04-05T23:22:41.956 INFO:teuthology.orchestra.run.smithi016.stdout:(113/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 364 kB/s | 103 kB 00:00 2024-04-05T23:22:41.959 INFO:teuthology.orchestra.run.smithi130.stdout:(97/146): jq-1.6-7.el8.x86_64.rpm 866 kB/s | 202 kB 00:00 2024-04-05T23:22:41.967 INFO:teuthology.orchestra.run.smithi027.stdout:(114/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 775 kB/s | 220 kB 00:00 2024-04-05T23:22:42.000 INFO:teuthology.orchestra.run.smithi027.stdout:(115/146): libsepol-devel-2.9-3.el8.x86_64.rpm 741 kB/s | 87 kB 00:00 2024-04-05T23:22:42.049 INFO:teuthology.orchestra.run.smithi016.stdout:(114/146): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2024-04-05T23:22:42.059 INFO:teuthology.orchestra.run.smithi027.stdout:(116/146): zlib-devel-1.2.11-18.el8_5.x86_64.rp 406 kB/s | 58 kB 00:00 2024-04-05T23:22:42.068 INFO:teuthology.orchestra.run.smithi130.stdout:(98/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-05T23:22:42.093 INFO:teuthology.orchestra.run.smithi130.stdout:(99/146): python3-pytz-2017.2-11.el8.noarch.rpm 282 kB/s | 54 kB 00:00 2024-04-05T23:22:42.107 INFO:teuthology.orchestra.run.smithi016.stdout:(115/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2024-04-05T23:22:42.118 INFO:teuthology.orchestra.run.smithi027.stdout:(117/146): libcom_err-devel-1.45.6-4.el8.x86_64 256 kB/s | 39 kB 00:00 2024-04-05T23:22:42.149 INFO:teuthology.orchestra.run.smithi016.stdout:(116/146): python3-numpy-f2py-1.14.3-10.el8.x86 2.2 MB/s | 225 kB 00:00 2024-04-05T23:22:42.172 INFO:teuthology.orchestra.run.smithi027.stdout:(118/146): fuse-2.9.7-17.el8.x86_64.rpm 485 kB/s | 83 kB 00:00 2024-04-05T23:22:42.219 INFO:teuthology.orchestra.run.smithi027.stdout:(119/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 645 kB 00:00 2024-04-05T23:22:42.233 INFO:teuthology.orchestra.run.smithi016.stdout:(117/146): python3-numpy-1.14.3-10.el8.x86_64.r 12 MB/s | 3.7 MB 00:00 2024-04-05T23:22:42.244 INFO:teuthology.orchestra.run.smithi027.stdout:(120/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-04-05T23:22:42.266 INFO:teuthology.orchestra.run.smithi016.stdout:(118/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2024-04-05T23:22:42.285 INFO:teuthology.orchestra.run.smithi027.stdout:(121/146): libuuid-devel-2.32.1-43.el8.x86_64.r 881 kB/s | 100 kB 00:00 2024-04-05T23:22:42.344 INFO:teuthology.orchestra.run.smithi027.stdout:(122/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.9 MB/s | 238 kB 00:00 2024-04-05T23:22:42.352 INFO:teuthology.orchestra.run.smithi130.stdout:(100/146): python3-scipy-1.0.0-21.module+el8.9. 33 MB/s | 14 MB 00:00 2024-04-05T23:22:42.369 INFO:teuthology.orchestra.run.smithi027.stdout:(123/146): xfsprogs-devel-5.0.0-12.el8.x86_64.r 579 kB/s | 73 kB 00:00 2024-04-05T23:22:42.385 INFO:teuthology.orchestra.run.smithi130.stdout:(101/146): postgresql-10.23-4.module+el8.9.0+21 4.7 MB/s | 1.5 MB 00:00 2024-04-05T23:22:42.388 INFO:teuthology.orchestra.run.smithi016.stdout:(119/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.0 MB/s | 157 kB 00:00 2024-04-05T23:22:42.427 INFO:teuthology.orchestra.run.smithi130.stdout:(102/146): postgresql-contrib-10.23-4.module+el 2.4 MB/s | 812 kB 00:00 2024-04-05T23:22:42.436 INFO:teuthology.orchestra.run.smithi027.stdout:(124/146): NetworkManager-initscripts-updown-1. 956 kB/s | 144 kB 00:00 2024-04-05T23:22:42.492 INFO:teuthology.orchestra.run.smithi016.stdout:(120/146): openblas-threads-0.3.15-6.el8.x86_64 15 MB/s | 5.2 MB 00:00 2024-04-05T23:22:42.503 INFO:teuthology.orchestra.run.smithi027.stdout:(125/146): elfutils-libelf-devel-0.189-3.el8.x8 388 kB/s | 62 kB 00:00 2024-04-05T23:22:42.519 INFO:teuthology.orchestra.run.smithi130.stdout:(103/146): libverto-devel-0.3.0-5.el8.x86_64.rp 134 kB/s | 18 kB 00:00 2024-04-05T23:22:42.534 INFO:teuthology.orchestra.run.smithi016.stdout:(121/146): automake-1.16.1-8.el8.noarch.rpm 4.8 MB/s | 714 kB 00:00 2024-04-05T23:22:42.554 INFO:teuthology.orchestra.run.smithi027.stdout:(126/146): krb5-devel-1.18.2-26.el8_9.x86_64.rp 3.0 MB/s | 562 kB 00:00 2024-04-05T23:22:42.569 INFO:teuthology.orchestra.run.smithi130.stdout:(104/146): python3-jwt-1.6.1-2.el8.noarch.rpm 300 kB/s | 43 kB 00:00 2024-04-05T23:22:42.621 INFO:teuthology.orchestra.run.smithi027.stdout:(127/146): openssl-devel-1.1.1k-12.el8_9.x86_64 13 MB/s | 2.3 MB 00:00 2024-04-05T23:22:42.637 INFO:teuthology.orchestra.run.smithi027.stdout:(128/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 352 kB/s | 47 kB 00:00 2024-04-05T23:22:42.659 INFO:teuthology.orchestra.run.smithi016.stdout:(122/146): openblas-0.3.15-6.el8.x86_64.rpm 13 MB/s | 5.0 MB 00:00 2024-04-05T23:22:42.661 INFO:teuthology.orchestra.run.smithi130.stdout:(105/146): postgresql-server-10.23-4.module+el8 16 MB/s | 5.1 MB 00:00 2024-04-05T23:22:42.684 INFO:teuthology.orchestra.run.smithi016.stdout:(123/146): libpq-13.11-1.el8.x86_64.rpm 1.0 MB/s | 198 kB 00:00 2024-04-05T23:22:42.686 INFO:teuthology.orchestra.run.smithi130.stdout:(106/146): python3-oauthlib-2.1.0-1.el8.noarch. 927 kB/s | 155 kB 00:00 2024-04-05T23:22:42.696 INFO:teuthology.orchestra.run.smithi027.stdout:(129/146): python3-cryptography-3.2.1-7.el8_9.x 3.8 MB/s | 559 kB 00:00 2024-04-05T23:22:42.709 INFO:teuthology.orchestra.run.smithi016.stdout:(124/146): jq-1.6-7.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2024-04-05T23:22:42.737 INFO:teuthology.orchestra.run.smithi130.stdout:(107/146): python3-pycparser-2.14-14.el8.noarch 653 kB/s | 109 kB 00:00 2024-04-05T23:22:42.837 INFO:teuthology.orchestra.run.smithi130.stdout:(108/146): libaio-devel-0.3.112-1.el8.x86_64.rp 124 kB/s | 19 kB 00:00 2024-04-05T23:22:42.870 INFO:teuthology.orchestra.run.smithi130.stdout:(109/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 329 kB/s | 44 kB 00:00 2024-04-05T23:22:42.883 INFO:teuthology.orchestra.run.smithi016.stdout:(125/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-04-05T23:22:42.918 INFO:teuthology.orchestra.run.smithi016.stdout:(126/146): python3-pytz-2017.2-11.el8.noarch.rp 209 kB/s | 54 kB 00:00 2024-04-05T23:22:42.954 INFO:teuthology.orchestra.run.smithi130.stdout:(110/146): python3-requests-oauthlib-1.0.0-1.el 146 kB/s | 43 kB 00:00 2024-04-05T23:22:43.004 INFO:teuthology.orchestra.run.smithi130.stdout:(111/146): pcre2-devel-10.32-2.el8.x86_64.rpm 3.5 MB/s | 605 kB 00:00 2024-04-05T23:22:43.038 INFO:teuthology.orchestra.run.smithi130.stdout:(112/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.3 MB/s | 229 kB 00:00 2024-04-05T23:22:43.077 INFO:teuthology.orchestra.run.smithi016.stdout:(127/146): postgresql-10.23-4.module+el8.9.0+21 7.8 MB/s | 1.5 MB 00:00 2024-04-05T23:22:43.080 INFO:teuthology.orchestra.run.smithi027.stdout:(130/146): python3-packaging-16.8-9.el8.noarch. 105 kB/s | 48 kB 00:00 2024-04-05T23:22:43.097 INFO:teuthology.orchestra.run.smithi130.stdout:(113/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.5 MB/s | 220 kB 00:00 2024-04-05T23:22:43.119 INFO:teuthology.orchestra.run.smithi016.stdout:(128/146): postgresql-contrib-10.23-4.module+el 4.0 MB/s | 812 kB 00:00 2024-04-05T23:22:43.172 INFO:teuthology.orchestra.run.smithi130.stdout:(114/146): libselinux-devel-2.9-5.el8.x86_64.rp 1.2 MB/s | 200 kB 00:00 2024-04-05T23:22:43.197 INFO:teuthology.orchestra.run.smithi130.stdout:(115/146): keyutils-libs-devel-1.5.10-9.el8.x86 303 kB/s | 48 kB 00:00 2024-04-05T23:22:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:(131/146): librados2-18.2.2-1173.g8919605b.el8. 6.1 MB/s | 3.7 MB 00:00 2024-04-05T23:22:43.264 INFO:teuthology.orchestra.run.smithi130.stdout:(116/146): libsepol-devel-2.9-3.el8.x86_64.rpm 521 kB/s | 87 kB 00:00 2024-04-05T23:22:43.264 INFO:teuthology.orchestra.run.smithi027.stdout:(132/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.2 MB/s | 232 kB 00:00 2024-04-05T23:22:43.289 INFO:teuthology.orchestra.run.smithi130.stdout:(117/146): zlib-devel-1.2.11-18.el8_5.x86_64.rp 493 kB/s | 58 kB 00:00 2024-04-05T23:22:43.294 INFO:teuthology.orchestra.run.smithi016.stdout:(129/146): python3-scipy-1.0.0-21.module+el8.9. 22 MB/s | 14 MB 00:00 2024-04-05T23:22:43.348 INFO:teuthology.orchestra.run.smithi130.stdout:(118/146): libcom_err-devel-1.45.6-4.el8.x86_64 256 kB/s | 39 kB 00:00 2024-04-05T23:22:43.389 INFO:teuthology.orchestra.run.smithi027.stdout:(133/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.0 MB/s | 303 kB 00:00 2024-04-05T23:22:43.406 INFO:teuthology.orchestra.run.smithi130.stdout:(119/146): fuse-2.9.7-17.el8.x86_64.rpm 585 kB/s | 83 kB 00:00 2024-04-05T23:22:43.414 INFO:teuthology.orchestra.run.smithi027.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 1.5 MB/s | 237 kB 00:00 2024-04-05T23:22:43.436 INFO:teuthology.orchestra.run.smithi016.stdout:(130/146): postgresql-server-10.23-4.module+el8 14 MB/s | 5.1 MB 00:00 2024-04-05T23:22:43.440 INFO:teuthology.orchestra.run.smithi130.stdout:(120/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-04-05T23:22:43.465 INFO:teuthology.orchestra.run.smithi130.stdout:(121/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.4 MB/s | 172 kB 00:00 2024-04-05T23:22:43.540 INFO:teuthology.orchestra.run.smithi027.stdout:(135/146): librbd1-18.2.2-1173.g8919605b.el8.x8 4.2 MB/s | 3.6 MB 00:00 2024-04-05T23:22:43.615 INFO:teuthology.orchestra.run.smithi130.stdout:(122/146): xfsprogs-devel-5.0.0-12.el8.x86_64.r 484 kB/s | 73 kB 00:00 2024-04-05T23:22:43.623 INFO:teuthology.orchestra.run.smithi027.stdout:(136/146): libsmartcols-2.32.1-43.el8.x86_64.rp 768 kB/s | 179 kB 00:00 2024-04-05T23:22:43.641 INFO:teuthology.orchestra.run.smithi130.stdout:(123/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 238 kB 00:00 2024-04-05T23:22:43.666 INFO:teuthology.orchestra.run.smithi130.stdout:(124/146): libuuid-devel-2.32.1-43.el8.x86_64.r 385 kB/s | 100 kB 00:00 2024-04-05T23:22:43.695 INFO:teuthology.orchestra.run.smithi016.stdout:(131/146): librbd1-18.2.2-1173.g8919605b.el8.x8 8.9 MB/s | 3.6 MB 00:00 2024-04-05T23:22:43.715 INFO:teuthology.orchestra.run.smithi027.stdout:(137/146): libuuid-2.32.1-43.el8.x86_64.rpm 329 kB/s | 99 kB 00:00 2024-04-05T23:22:43.762 INFO:teuthology.orchestra.run.smithi016.stdout:(132/146): elfutils-libelf-0.189-3.el8.x86_64.r 714 kB/s | 232 kB 00:00 2024-04-05T23:22:43.811 INFO:teuthology.orchestra.run.smithi027.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 9.1 MB/s | 2.5 MB 00:00 2024-04-05T23:22:43.841 INFO:teuthology.orchestra.run.smithi016.stdout:(133/146): librados2-18.2.2-1173.g8919605b.el8. 5.1 MB/s | 3.7 MB 00:00 2024-04-05T23:22:43.871 INFO:teuthology.orchestra.run.smithi016.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 2.1 MB/s | 237 kB 00:00 2024-04-05T23:22:43.899 INFO:teuthology.orchestra.run.smithi027.stdout:(139/146): util-linux-user-2.32.1-43.el8.x86_64 373 kB/s | 103 kB 00:00 2024-04-05T23:22:43.904 INFO:teuthology.orchestra.run.smithi016.stdout:(135/146): elfutils-libs-0.189-3.el8.x86_64.rpm 1.4 MB/s | 303 kB 00:00 2024-04-05T23:22:43.938 INFO:teuthology.orchestra.run.smithi016.stdout:(136/146): libsmartcols-2.32.1-43.el8.x86_64.rp 1.8 MB/s | 179 kB 00:00 2024-04-05T23:22:43.949 INFO:teuthology.orchestra.run.smithi027.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 3.6 MB/s | 854 kB 00:00 2024-04-05T23:22:43.950 INFO:teuthology.orchestra.run.smithi130.stdout:(125/146): elfutils-libelf-devel-0.189-3.el8.x8 200 kB/s | 62 kB 00:00 2024-04-05T23:22:43.963 INFO:teuthology.orchestra.run.smithi016.stdout:(137/146): libuuid-2.32.1-43.el8.x86_64.rpm 1.1 MB/s | 99 kB 00:00 2024-04-05T23:22:43.975 INFO:teuthology.orchestra.run.smithi027.stdout:(141/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2024-04-05T23:22:44.000 INFO:teuthology.orchestra.run.smithi027.stdout:(142/146): libfdisk-2.32.1-43.el8.x86_64.rpm 2.5 MB/s | 254 kB 00:00 2024-04-05T23:22:44.055 INFO:teuthology.orchestra.run.smithi016.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 16 MB/s | 2.5 MB 00:00 2024-04-05T23:22:44.081 INFO:teuthology.orchestra.run.smithi016.stdout:(139/146): util-linux-user-2.32.1-43.el8.x86_64 720 kB/s | 103 kB 00:00 2024-04-05T23:22:44.108 INFO:teuthology.orchestra.run.smithi130.stdout:(126/146): NetworkManager-initscripts-updown-1. 292 kB/s | 144 kB 00:00 2024-04-05T23:22:44.142 INFO:teuthology.orchestra.run.smithi130.stdout:(127/146): krb5-devel-1.18.2-26.el8_9.x86_64.rp 1.2 MB/s | 562 kB 00:00 2024-04-05T23:22:44.158 INFO:teuthology.orchestra.run.smithi027.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 3.3 MB/s | 711 kB 00:00 2024-04-05T23:22:44.173 INFO:teuthology.orchestra.run.smithi016.stdout:(140/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.8 MB/s | 220 kB 00:00 2024-04-05T23:22:44.192 INFO:teuthology.orchestra.run.smithi027.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 1.7 MB/s | 333 kB 00:00 2024-04-05T23:22:44.206 INFO:teuthology.orchestra.run.smithi016.stdout:(141/146): iproute-6.2.0-5.el8_9.x86_64.rpm 3.4 MB/s | 854 kB 00:00 2024-04-05T23:22:44.242 INFO:teuthology.orchestra.run.smithi130.stdout:(128/146): openssl-devel-1.1.1k-12.el8_9.x86_64 8.0 MB/s | 2.3 MB 00:00 2024-04-05T23:22:44.242 INFO:teuthology.orchestra.run.smithi027.stdout:(145/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 5.5 MB/s | 1.5 MB 00:00 2024-04-05T23:22:44.248 INFO:teuthology.orchestra.run.smithi016.stdout:(142/146): libfdisk-2.32.1-43.el8.x86_64.rpm 1.5 MB/s | 254 kB 00:00 2024-04-05T23:22:44.259 INFO:teuthology.orchestra.run.smithi130.stdout:(129/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 313 kB/s | 47 kB 00:00 2024-04-05T23:22:44.301 INFO:teuthology.orchestra.run.smithi130.stdout:(130/146): python3-cryptography-3.2.1-7.el8_9.x 3.4 MB/s | 559 kB 00:00 2024-04-05T23:22:44.351 INFO:teuthology.orchestra.run.smithi027.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.6 MB/s | 1.1 MB 00:00 2024-04-05T23:22:44.354 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-05T23:22:44.355 INFO:teuthology.orchestra.run.smithi027.stdout:Total 8.9 MB/s | 215 MB 00:24 2024-04-05T23:22:44.390 INFO:teuthology.orchestra.run.smithi016.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 3.2 MB/s | 711 kB 00:00 2024-04-05T23:22:44.441 INFO:teuthology.orchestra.run.smithi016.stdout:(144/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 6.3 MB/s | 1.5 MB 00:00 2024-04-05T23:22:44.541 INFO:teuthology.orchestra.run.smithi016.stdout:(145/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 1.1 MB/s | 333 kB 00:00 2024-04-05T23:22:44.584 INFO:teuthology.orchestra.run.smithi130.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 819 kB/s | 232 kB 00:00 2024-04-05T23:22:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 4.0 MB/s | 1.1 MB 00:00 2024-04-05T23:22:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-05T23:22:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:Total 9.2 MB/s | 215 MB 00:23 2024-04-05T23:22:44.768 INFO:teuthology.orchestra.run.smithi130.stdout:(132/146): librbd1-18.2.2-1173.g8919605b.el8.x8 7.0 MB/s | 3.6 MB 00:00 2024-04-05T23:22:44.793 INFO:teuthology.orchestra.run.smithi130.stdout:(133/146): elfutils-libs-0.189-3.el8.x86_64.rpm 1.4 MB/s | 303 kB 00:00 2024-04-05T23:22:44.944 INFO:teuthology.orchestra.run.smithi130.stdout:(134/146): libsmartcols-2.32.1-43.el8.x86_64.rp 1.2 MB/s | 179 kB 00:00 2024-04-05T23:22:44.969 INFO:teuthology.orchestra.run.smithi130.stdout:(135/146): libmount-2.32.1-43.el8.x86_64.rpm 1.2 MB/s | 237 kB 00:00 2024-04-05T23:22:45.136 INFO:teuthology.orchestra.run.smithi130.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 517 kB/s | 99 kB 00:00 2024-04-05T23:22:45.203 INFO:teuthology.orchestra.run.smithi130.stdout:(137/146): util-linux-2.32.1-43.el8.x86_64.rpm 11 MB/s | 2.5 MB 00:00 2024-04-05T23:22:45.261 INFO:teuthology.orchestra.run.smithi130.stdout:(138/146): util-linux-user-2.32.1-43.el8.x86_64 820 kB/s | 103 kB 00:00 2024-04-05T23:22:45.328 INFO:teuthology.orchestra.run.smithi130.stdout:(139/146): libblkid-2.32.1-43.el8.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-04-05T23:22:45.420 INFO:teuthology.orchestra.run.smithi130.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 3.8 MB/s | 854 kB 00:00 2024-04-05T23:22:45.470 INFO:teuthology.orchestra.run.smithi130.stdout:(141/146): libfdisk-2.32.1-43.el8.x86_64.rpm 1.7 MB/s | 254 kB 00:00 2024-04-05T23:22:45.490 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-05T23:22:45.604 INFO:teuthology.orchestra.run.smithi130.stdout:(142/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 3.8 MB/s | 711 kB 00:00 2024-04-05T23:22:45.762 INFO:teuthology.orchestra.run.smithi130.stdout:(143/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 5.0 MB/s | 1.5 MB 00:00 2024-04-05T23:22:45.796 INFO:teuthology.orchestra.run.smithi130.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 1.7 MB/s | 333 kB 00:00 2024-04-05T23:22:45.920 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-05T23:22:45.921 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-05T23:22:45.954 INFO:teuthology.orchestra.run.smithi130.stdout:(145/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.6 MB/s | 1.1 MB 00:00 2024-04-05T23:22:46.396 INFO:teuthology.orchestra.run.smithi130.stdout:(146/146): librados2-18.2.2-1173.g8919605b.el8. 1.7 MB/s | 3.7 MB 00:02 2024-04-05T23:22:46.401 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-04-05T23:22:46.401 INFO:teuthology.orchestra.run.smithi130.stdout:Total 7.3 MB/s | 215 MB 00:29 2024-04-05T23:22:46.825 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-05T23:22:47.198 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-05T23:22:47.209 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-05T23:22:47.302 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-05T23:22:47.302 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-05T23:22:47.514 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-04-05T23:22:47.910 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-04-05T23:22:47.910 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-04-05T23:22:48.954 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-05T23:22:48.967 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-05T23:22:49.122 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-04-05T23:22:49.133 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-04-05T23:22:53.183 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-05T23:22:53.326 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-04-05T23:22:53.713 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-05T23:22:53.951 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-05T23:22:54.154 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-04-05T23:22:54.325 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-05T23:22:54.446 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-05T23:22:54.616 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-05T23:22:54.668 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-04-05T23:22:54.721 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-05T23:22:54.753 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-05T23:22:54.802 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-04-05T23:22:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-04-05T23:22:54.959 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-04-05T23:22:55.063 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-05T23:22:55.142 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/162 2024-04-05T23:22:55.189 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-05T23:22:55.207 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/162 2024-04-05T23:22:55.278 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-05T23:22:55.448 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/162 2024-04-05T23:22:55.469 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-05T23:22:55.485 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-04-05T23:22:55.608 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-05T23:22:55.643 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-04-05T23:22:55.670 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-05T23:22:55.748 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-05T23:22:55.840 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-05T23:22:55.925 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-05T23:22:55.954 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-05T23:22:56.025 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-05T23:22:56.069 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-05T23:22:56.106 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-05T23:22:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-05T23:22:56.236 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-04-05T23:22:56.412 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/162 2024-04-05T23:22:56.419 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:18.2.2-1173.g8919605b.el8.x86_64 9/162 2024-04-05T23:22:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-04-05T23:22:56.484 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/162 2024-04-05T23:22:56.678 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-04-05T23:22:56.698 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/162 2024-04-05T23:22:56.815 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:18.2.2-1173.g8919605b.el8.x86_64 9/162 2024-04-05T23:22:56.887 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-05T23:22:56.939 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-04-05T23:22:57.008 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 10/162 2024-04-05T23:22:57.089 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-04-05T23:22:57.143 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 10/162 2024-04-05T23:22:57.289 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-05T23:22:57.323 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-05T23:22:57.374 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 11/162 2024-04-05T23:22:57.478 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 11/162 2024-04-05T23:22:57.505 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:18.2.2-1173.g8919605b.el8.x86_64 9/162 2024-04-05T23:22:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-05T23:22:57.834 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:18.2.2-1173.g8919605b.el8.x86_64 12/162 2024-04-05T23:22:57.898 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:18.2.2-1173.g8919605b.el8.x86_64 9/162 2024-04-05T23:22:58.028 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:18.2.2-1173.g8919605b.el8.x86_64 9/162 2024-04-05T23:22:58.062 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 10/162 2024-04-05T23:22:58.141 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-05T23:22:58.191 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 10/162 2024-04-05T23:22:58.226 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-05T23:22:58.379 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 11/162 2024-04-05T23:22:58.440 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-04-05T23:22:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:18.2.2-1173.g8919605b.el8.x86_64 9/162 2024-04-05T23:22:58.478 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 11/162 2024-04-05T23:22:58.647 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 10/162 2024-04-05T23:22:58.797 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:18.2.2-1173.g8919605b.el8.x86_64 12/162 2024-04-05T23:22:58.824 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 10/162 2024-04-05T23:22:58.904 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-04-05T23:22:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 11/162 2024-04-05T23:22:59.053 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-05T23:22:59.095 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-04-05T23:22:59.135 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-05T23:22:59.141 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 11/162 2024-04-05T23:22:59.385 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-04-05T23:22:59.520 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-04-05T23:22:59.522 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:18.2.2-1173.g8919605b.el8.x86_64 12/162 2024-04-05T23:22:59.640 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-04-05T23:22:59.754 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-04-05T23:22:59.814 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-05T23:22:59.848 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-04-05T23:22:59.925 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-05T23:22:59.949 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 19/162 2024-04-05T23:23:00.063 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 19/162 2024-04-05T23:23:00.175 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-04-05T23:23:00.206 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-04-05T23:23:00.224 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86 20/162 2024-04-05T23:23:00.291 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86 20/162 2024-04-05T23:23:00.449 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-04-05T23:23:00.469 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-04-05T23:23:00.507 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/162 2024-04-05T23:23:00.603 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 19/162 2024-04-05T23:23:00.639 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/162 2024-04-05T23:23:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-04-05T23:23:00.713 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 19/162 2024-04-05T23:23:00.832 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/162 2024-04-05T23:23:00.853 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86 20/162 2024-04-05T23:23:00.933 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86 20/162 2024-04-05T23:23:01.128 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/162 2024-04-05T23:23:01.140 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-04-05T23:23:01.248 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/162 2024-04-05T23:23:01.387 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-04-05T23:23:01.428 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/162 2024-04-05T23:23:01.479 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/162 2024-04-05T23:23:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 19/162 2024-04-05T23:23:01.661 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 19/162 2024-04-05T23:23:01.666 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-04-05T23:23:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86 20/162 2024-04-05T23:23:01.939 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86 20/162 2024-04-05T23:23:02.050 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/162 2024-04-05T23:23:02.121 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-04-05T23:23:02.245 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-04-05T23:23:02.334 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-04-05T23:23:02.429 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-04-05T23:23:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/162 2024-04-05T23:23:02.680 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-04-05T23:23:02.848 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-04-05T23:23:02.855 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-04-05T23:23:03.025 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-04-05T23:23:03.080 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/162 2024-04-05T23:23:03.181 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-04-05T23:23:03.258 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/162 2024-04-05T23:23:03.547 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/162 2024-04-05T23:23:03.723 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/162 2024-04-05T23:23:03.759 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/162 2024-04-05T23:23:03.887 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/162 2024-04-05T23:23:03.910 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-04-05T23:23:04.009 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/162 2024-04-05T23:23:04.130 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-04-05T23:23:04.158 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/162 2024-04-05T23:23:04.193 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/162 2024-04-05T23:23:04.340 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/162 2024-04-05T23:23:04.503 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/162 2024-04-05T23:23:04.595 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/162 2024-04-05T23:23:04.799 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-04-05T23:23:04.838 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-04-05T23:23:04.874 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/162 2024-04-05T23:23:05.022 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:18.2.2-1173.g8919605b.el8. 31/162 2024-04-05T23:23:05.048 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-04-05T23:23:05.138 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-04-05T23:23:05.162 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:18.2.2-1173.g8919605b.el 32/162 2024-04-05T23:23:05.292 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:18.2.2-1173.g8919605b.el8. 31/162 2024-04-05T23:23:05.438 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:18.2.2-1173.g8919605b.el 32/162 2024-04-05T23:23:05.462 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:18.2.2-1173.g8919605b.el8.x86_6 33/162 2024-04-05T23:23:05.704 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:18.2.2-1173.g8919605b.el8.x86_6 33/162 2024-04-05T23:23:05.911 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-04-05T23:23:06.163 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-04-05T23:23:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-04-05T23:23:06.526 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/162 2024-04-05T23:23:06.634 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/162 2024-04-05T23:23:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-04-05T23:23:06.862 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/162 2024-04-05T23:23:06.904 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:18.2.2-1173.g8919605b.el8. 31/162 2024-04-05T23:23:06.965 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/162 2024-04-05T23:23:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:18.2.2-1173.g8919605b.el 32/162 2024-04-05T23:23:07.157 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/162 2024-04-05T23:23:07.197 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/162 2024-04-05T23:23:07.307 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:18.2.2-1173.g8919605b.el8.x86_6 33/162 2024-04-05T23:23:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-04-05T23:23:07.418 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-04-05T23:23:07.533 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-04-05T23:23:09.159 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-04-05T23:23:09.232 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-04-05T23:23:09.503 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/162 2024-04-05T23:23:09.594 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/162 2024-04-05T23:23:09.692 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-04-05T23:23:09.721 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-04-05T23:23:09.837 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-04-05T23:23:09.922 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-04-05T23:23:10.088 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-04-05T23:23:10.090 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-04-05T23:23:10.157 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:18.2.2-1173.g8919605b.el8.x86_64 41/162 2024-04-05T23:23:10.288 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:18.2.2-1173.g8919605b.el8.x86_64 41/162 2024-04-05T23:23:10.324 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:18.2.2-1173.g8919605b.el8.x86_6 42/162 2024-04-05T23:23:10.472 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:18.2.2-1173.g8919605b.el8.x86_6 42/162 2024-04-05T23:23:10.518 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-04-05T23:23:10.598 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-04-05T23:23:10.698 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-04-05T23:23:10.778 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-04-05T23:23:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-05T23:23:10.899 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-05T23:23:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-05T23:23:11.095 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-05T23:23:11.130 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-04-05T23:23:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-04-05T23:23:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-04-05T23:23:11.371 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-04-05T23:23:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-04-05T23:23:11.551 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-05T23:23:11.605 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-04-05T23:23:11.773 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-05T23:23:12.014 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-05T23:23:12.049 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-04-05T23:23:12.236 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-05T23:23:12.312 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-04-05T23:23:12.318 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-05T23:23:12.336 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-05T23:23:12.533 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/162 2024-04-05T23:23:12.571 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-04-05T23:23:12.623 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-05T23:23:12.649 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-05T23:23:12.747 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/162 2024-04-05T23:23:12.796 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:18.2.2-1173.g8919605b.el8.x86_64 41/162 2024-04-05T23:23:12.866 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-16.8-9.el8.noarch 49/162 2024-04-05T23:23:12.923 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/162 2024-04-05T23:23:13.006 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:18.2.2-1173.g8919605b.el8.x86_6 42/162 2024-04-05T23:23:13.078 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.7-17.el8.x86_64 50/162 2024-04-05T23:23:13.122 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/162 2024-04-05T23:23:13.237 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-04-05T23:23:13.296 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 51/162 2024-04-05T23:23:13.346 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/162 2024-04-05T23:23:13.348 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-04-05T23:23:13.520 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 52/162 2024-04-05T23:23:13.621 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-05T23:23:13.636 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/162 2024-04-05T23:23:13.711 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 53/162 2024-04-05T23:23:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-05T23:23:13.871 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/162 2024-04-05T23:23:13.910 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-04-05T23:23:13.944 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 54/162 2024-04-05T23:23:14.065 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/162 2024-04-05T23:23:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-04-05T23:23:14.213 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/162 2024-04-05T23:23:14.294 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 55/162 2024-04-05T23:23:14.320 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-05T23:23:14.427 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/162 2024-04-05T23:23:14.577 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/162 2024-04-05T23:23:14.806 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/162 2024-04-05T23:23:14.844 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/162 2024-04-05T23:23:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-05T23:23:14.995 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 58/162 2024-04-05T23:23:15.246 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-05T23:23:15.262 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 59/162 2024-04-05T23:23:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-05T23:23:15.420 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-04-05T23:23:15.490 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 60/162 2024-04-05T23:23:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-04-05T23:23:15.690 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 60/162 2024-04-05T23:23:15.778 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/162 2024-04-05T23:23:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.6-7.el8.x86_64 50/162 2024-04-05T23:23:16.026 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 62/162 2024-04-05T23:23:16.036 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 61/162 2024-04-05T23:23:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.1-8.el8.noarch 51/162 2024-04-05T23:23:16.187 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/162 2024-04-05T23:23:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-43.el8.x86_64 52/162 2024-04-05T23:23:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 52/162 2024-04-05T23:23:16.323 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/162 2024-04-05T23:23:16.435 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 64/162 2024-04-05T23:23:16.530 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 63/162 2024-04-05T23:23:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/162 2024-04-05T23:23:16.697 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 65/162 2024-04-05T23:23:16.733 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-04-05T23:23:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 54/162 2024-04-05T23:23:16.910 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 66/162 2024-04-05T23:23:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 55/162 2024-04-05T23:23:16.997 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 65/162 2024-04-05T23:23:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.7-17.el8.x86_64 56/162 2024-04-05T23:23:17.155 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/162 2024-04-05T23:23:17.237 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 66/162 2024-04-05T23:23:17.349 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/162 2024-04-05T23:23:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 57/162 2024-04-05T23:23:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 67/162 2024-04-05T23:23:17.598 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/162 2024-04-05T23:23:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 58/162 2024-04-05T23:23:17.727 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 68/162 2024-04-05T23:23:17.817 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 70/162 2024-04-05T23:23:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 59/162 2024-04-05T23:23:17.897 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 70/162 2024-04-05T23:23:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 69/162 2024-04-05T23:23:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 60/162 2024-04-05T23:23:18.166 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : jq-1.6-7.el8.x86_64 71/162 2024-04-05T23:23:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 70/162 2024-04-05T23:23:18.389 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-04-05T23:23:18.402 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 61/162 2024-04-05T23:23:18.429 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : automake-1.16.1-8.el8.noarch 72/162 2024-04-05T23:23:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-04-05T23:23:18.609 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : uuid-1.6.2-43.el8.x86_64 73/162 2024-04-05T23:23:18.678 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 62/162 2024-04-05T23:23:18.683 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 73/162 2024-04-05T23:23:18.803 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-7.el8.x86_64 72/162 2024-04-05T23:23:18.860 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 74/162 2024-04-05T23:23:18.926 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 63/162 2024-04-05T23:23:19.066 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : automake-1.16.1-8.el8.noarch 73/162 2024-04-05T23:23:19.067 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 75/162 2024-04-05T23:23:19.099 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 64/162 2024-04-05T23:23:19.253 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 76/162 2024-04-05T23:23:19.280 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : uuid-1.6.2-43.el8.x86_64 74/162 2024-04-05T23:23:19.309 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 65/162 2024-04-05T23:23:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 74/162 2024-04-05T23:23:19.446 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/162 2024-04-05T23:23:19.557 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 75/162 2024-04-05T23:23:19.618 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-04-05T23:23:19.730 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 76/162 2024-04-05T23:23:19.738 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 66/162 2024-04-05T23:23:19.816 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-04-05T23:23:19.957 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 77/162 2024-04-05T23:23:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 67/162 2024-04-05T23:23:20.105 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-04-05T23:23:20.123 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-04-05T23:23:20.339 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-04-05T23:23:20.366 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-04-05T23:23:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 68/162 2024-04-05T23:23:20.547 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-04-05T23:23:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 69/162 2024-04-05T23:23:20.682 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-04-05T23:23:20.770 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-04-05T23:23:20.853 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 70/162 2024-04-05T23:23:20.954 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-04-05T23:23:21.009 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-04-05T23:23:21.139 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 71/162 2024-04-05T23:23:21.193 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-04-05T23:23:21.204 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-04-05T23:23:21.367 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-04-05T23:23:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 72/162 2024-04-05T23:23:21.459 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-04-05T23:23:21.617 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-04-05T23:23:21.648 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 73/162 2024-04-05T23:23:21.664 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-04-05T23:23:21.844 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-04-05T23:23:21.847 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-04-05T23:23:21.932 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 74/162 2024-04-05T23:23:22.071 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-04-05T23:23:22.099 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-04-05T23:23:22.141 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 75/162 2024-04-05T23:23:22.277 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-04-05T23:23:22.329 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-04-05T23:23:22.461 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-04-05T23:23:22.471 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 76/162 2024-04-05T23:23:22.497 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-04-05T23:23:22.657 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/162 2024-04-05T23:23:22.679 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-04-05T23:23:22.762 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-04-05T23:23:22.881 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-04-05T23:23:22.945 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-04-05T23:23:23.007 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-04-05T23:23:23.073 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-04-05T23:23:23.144 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-04-05T23:23:23.182 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-04-05T23:23:23.380 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-04-05T23:23:23.414 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-04-05T23:23:23.447 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-04-05T23:23:23.595 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-04-05T23:23:23.730 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-04-05T23:23:23.783 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-04-05T23:23:23.921 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-04-05T23:23:23.943 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-04-05T23:23:24.207 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-04-05T23:23:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-04-05T23:23:24.405 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-04-05T23:23:24.508 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-04-05T23:23:24.612 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-04-05T23:23:24.803 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-04-05T23:23:24.808 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-04-05T23:23:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-04-05T23:23:25.048 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-04-05T23:23:25.256 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-04-05T23:23:25.341 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-04-05T23:23:25.398 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-04-05T23:23:25.476 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-04-05T23:23:25.611 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-04-05T23:23:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-04-05T23:23:25.761 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-04-05T23:23:25.899 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-04-05T23:23:25.948 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-04-05T23:23:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-04-05T23:23:26.203 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-04-05T23:23:26.340 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-04-05T23:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-04-05T23:23:26.616 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-04-05T23:23:26.671 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-04-05T23:23:26.712 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-04-05T23:23:26.833 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-04-05T23:23:26.874 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 103/162 2024-04-05T23:23:26.951 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 103/162 2024-04-05T23:23:27.136 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:18.2.2-1173.g8919605b.el8.x86_64 104/162 2024-04-05T23:23:27.700 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-04-05T23:23:27.912 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 103/162 2024-04-05T23:23:27.985 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 103/162 2024-04-05T23:23:28.126 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-04-05T23:23:28.178 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:18.2.2-1173.g8919605b.el8.x86_64 104/162 2024-04-05T23:23:28.433 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-04-05T23:23:28.709 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-04-05T23:23:29.049 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-04-05T23:23:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-04-05T23:23:29.564 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-04-05T23:23:30.438 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-04-05T23:23:30.651 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 103/162 2024-04-05T23:23:30.750 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 103/162 2024-04-05T23:23:30.808 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 105/162 2024-04-05T23:23:30.970 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:18.2.2-1173.g8919605b.el8.x86_64 104/162 2024-04-05T23:23:31.633 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 105/162 2024-04-05T23:23:32.230 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 105/162 2024-04-05T23:23:32.508 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 106/162 2024-04-05T23:23:32.602 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 105/162 2024-04-05T23:23:32.604 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 106/162 2024-04-05T23:23:32.742 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 107/162 2024-04-05T23:23:32.845 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 105/162 2024-04-05T23:23:33.473 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 105/162 2024-04-05T23:23:33.766 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 106/162 2024-04-05T23:23:33.854 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 106/162 2024-04-05T23:23:34.039 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 107/162 2024-04-05T23:23:36.182 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 105/162 2024-04-05T23:23:36.401 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 105/162 2024-04-05T23:23:37.134 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 105/162 2024-04-05T23:23:38.251 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 106/162 2024-04-05T23:23:38.395 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 106/162 2024-04-05T23:23:38.535 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 107/162 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 107/162 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /run 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2024-04-05T23:23:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2024-04-05T23:23:52.071 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:23:52.423 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 108/162 2024-04-05T23:23:54.057 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 108/162 2024-04-05T23:23:54.229 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 109/162 2024-04-05T23:23:54.314 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 107/162 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /run 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2024-04-05T23:23:54.315 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:23:54.449 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 109/162 2024-04-05T23:23:54.637 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 110/162 2024-04-05T23:23:54.678 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 108/162 2024-04-05T23:23:55.112 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 110/162 2024-04-05T23:23:55.312 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 111/162 2024-04-05T23:23:55.331 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 111/162 2024-04-05T23:23:56.420 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 108/162 2024-04-05T23:23:56.588 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 109/162 2024-04-05T23:23:56.820 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 109/162 2024-04-05T23:23:57.000 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 110/162 2024-04-05T23:23:57.110 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:18.2.2-1173.g8919605b.el8.noarch 112/162 2024-04-05T23:23:57.256 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:18.2.2-1173.g8919605b.el8.noarch 112/162 2024-04-05T23:23:57.376 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1173.g8919605b.e 113/162 2024-04-05T23:23:57.506 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 110/162 2024-04-05T23:23:57.545 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1173.g8919605b. 114/162 2024-04-05T23:23:57.701 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 111/162 2024-04-05T23:23:57.720 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 111/162 2024-04-05T23:23:57.797 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noa 115/162 2024-04-05T23:23:58.832 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noa 115/162 2024-04-05T23:23:59.086 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.n 116/162 2024-04-05T23:23:59.595 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:18.2.2-1173.g8919605b.el8.noarch 112/162 2024-04-05T23:23:59.783 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:18.2.2-1173.g8919605b.el8.noarch 112/162 2024-04-05T23:23:59.923 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1173.g8919605b.e 113/162 2024-04-05T23:24:00.051 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.n 116/162 2024-04-05T23:24:00.161 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1173.g8919605b. 114/162 2024-04-05T23:24:00.194 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1173.g891 117/162 2024-04-05T23:24:00.268 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 107/162 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /run 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2024-04-05T23:24:00.269 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:24:00.288 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1173.g891 117/162 2024-04-05T23:24:00.352 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noa 115/162 2024-04-05T23:24:00.640 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1173.g8919605b.el 118/162 2024-04-05T23:24:00.641 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 108/162 2024-04-05T23:24:00.812 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 119/162 2024-04-05T23:24:00.851 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 119/162 2024-04-05T23:24:01.010 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 120/162 2024-04-05T23:24:01.023 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 120/162 2024-04-05T23:24:01.446 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noa 115/162 2024-04-05T23:24:01.696 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.n 116/162 2024-04-05T23:24:02.284 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:18.2.2-1173.g8919605b.el8.x86_64 121/162 2024-04-05T23:24:02.478 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 122/162 2024-04-05T23:24:02.513 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 122/162 2024-04-05T23:24:02.546 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 108/162 2024-04-05T23:24:02.694 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1173.g89196 123/162 2024-04-05T23:24:02.726 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.n 116/162 2024-04-05T23:24:02.738 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1173.g89196 123/162 2024-04-05T23:24:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 109/162 2024-04-05T23:24:02.891 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1173.g891 117/162 2024-04-05T23:24:02.946 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 124/162 2024-04-05T23:24:02.990 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1173.g891 117/162 2024-04-05T23:24:03.035 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 109/162 2024-04-05T23:24:03.294 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 124/162 2024-04-05T23:24:03.331 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 110/162 2024-04-05T23:24:03.350 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1173.g8919605b.el 118/162 2024-04-05T23:24:03.502 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 125/162 2024-04-05T23:24:03.521 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 119/162 2024-04-05T23:24:03.560 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 119/162 2024-04-05T23:24:03.841 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 120/162 2024-04-05T23:24:03.853 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 120/162 2024-04-05T23:24:03.906 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 110/162 2024-04-05T23:24:04.122 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 111/162 2024-04-05T23:24:04.166 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 111/162 2024-04-05T23:24:05.227 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:18.2.2-1173.g8919605b.el8.x86_64 121/162 2024-04-05T23:24:05.412 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 122/162 2024-04-05T23:24:05.448 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 122/162 2024-04-05T23:24:05.649 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1173.g89196 123/162 2024-04-05T23:24:05.694 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1173.g89196 123/162 2024-04-05T23:24:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 124/162 2024-04-05T23:24:06.270 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 124/162 2024-04-05T23:24:06.482 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 125/162 2024-04-05T23:24:07.300 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:18.2.2-1173.g8919605b.el8.noarch 112/162 2024-04-05T23:24:07.491 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:18.2.2-1173.g8919605b.el8.noarch 112/162 2024-04-05T23:24:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1173.g8919605b.e 113/162 2024-04-05T23:24:07.807 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1173.g8919605b. 114/162 2024-04-05T23:24:07.953 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 125/162 2024-04-05T23:24:08.040 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noa 115/162 2024-04-05T23:24:08.425 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:18.2.2-1173.g8919605b.el8.x86_64 126/162 2024-04-05T23:24:08.755 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 127/162 2024-04-05T23:24:08.972 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libtool-2.4.6-25.el8.x86_64 128/162 2024-04-05T23:24:09.305 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noa 115/162 2024-04-05T23:24:09.545 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.n 116/162 2024-04-05T23:24:09.655 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 128/162 2024-04-05T23:24:09.983 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/162 2024-04-05T23:24:10.244 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 130/162 2024-04-05T23:24:10.457 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 131/162 2024-04-05T23:24:10.703 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.n 116/162 2024-04-05T23:24:10.714 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-04-05T23:24:10.880 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1173.g891 117/162 2024-04-05T23:24:10.905 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-04-05T23:24:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1173.g891 117/162 2024-04-05T23:24:11.085 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:18.2.2-1173.g8919605b.el8.x86_ 134/162 2024-04-05T23:24:11.236 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 125/162 2024-04-05T23:24:11.445 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1173.g8919605b.el 118/162 2024-04-05T23:24:11.613 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 119/162 2024-04-05T23:24:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 119/162 2024-04-05T23:24:11.733 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:18.2.2-1173.g8919605b.el8.x86_64 126/162 2024-04-05T23:24:11.838 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 120/162 2024-04-05T23:24:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 120/162 2024-04-05T23:24:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 127/162 2024-04-05T23:24:12.359 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libtool-2.4.6-25.el8.x86_64 128/162 2024-04-05T23:24:13.102 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 128/162 2024-04-05T23:24:13.290 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:18.2.2-1173.g8919605b.el8.x86_64 121/162 2024-04-05T23:24:13.342 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-05T23:24:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/162 2024-04-05T23:24:13.487 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 122/162 2024-04-05T23:24:13.550 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 122/162 2024-04-05T23:24:13.645 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-05T23:24:13.695 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-05T23:24:13.730 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1173.g89196 123/162 2024-04-05T23:24:13.763 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 130/162 2024-04-05T23:24:13.803 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1173.g89196 123/162 2024-04-05T23:24:13.946 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 131/162 2024-04-05T23:24:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 124/162 2024-04-05T23:24:14.084 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-04-05T23:24:14.205 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-04-05T23:24:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 124/162 2024-04-05T23:24:14.411 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-04-05T23:24:14.443 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-04-05T23:24:14.591 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:18.2.2-1173.g8919605b.el8.x86_ 134/162 2024-04-05T23:24:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 125/162 2024-04-05T23:24:14.654 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-04-05T23:24:14.825 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-top-2:18.2.2-1173.g8919605b.el8.noarch 139/162 2024-04-05T23:24:15.069 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 140/162 2024-04-05T23:24:15.356 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:18.2.2-1173.g8919605b.el8.x86_64 141/162 2024-04-05T23:24:15.630 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-04-05T23:24:15.811 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-04-05T23:24:15.822 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-04-05T23:24:15.990 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 144/162 2024-04-05T23:24:16.199 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flex-2.6.1-9.el8.x86_64 145/162 2024-04-05T23:24:16.281 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 145/162 2024-04-05T23:24:16.494 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : bison-3.0.4-10.el8.x86_64 146/162 2024-04-05T23:24:16.508 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 146/162 2024-04-05T23:24:16.680 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-04-05T23:24:16.846 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-04-05T23:24:17.039 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-04-05T23:24:17.040 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-05T23:24:17.105 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-05T23:24:17.297 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-05T23:24:17.297 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-05T23:24:17.553 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-05T23:24:17.554 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-05T23:24:17.600 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-05T23:24:17.638 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-05T23:24:17.781 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-05T23:24:17.976 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-04-05T23:24:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-04-05T23:24:18.138 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-04-05T23:24:18.350 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-04-05T23:24:18.404 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-04-05T23:24:18.404 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-05T23:24:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-04-05T23:24:18.610 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-05T23:24:18.610 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-04-05T23:24:18.832 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-top-2:18.2.2-1173.g8919605b.el8.noarch 139/162 2024-04-05T23:24:18.877 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-04-05T23:24:18.877 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-05T23:24:19.034 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 140/162 2024-04-05T23:24:19.110 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-05T23:24:19.111 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-05T23:24:19.335 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-05T23:24:19.335 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-05T23:24:19.353 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:18.2.2-1173.g8919605b.el8.x86_64 141/162 2024-04-05T23:24:19.593 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-05T23:24:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-04-05T23:24:19.735 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-04-05T23:24:19.736 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-05T23:24:19.777 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 125/162 2024-04-05T23:24:19.849 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-04-05T23:24:19.861 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-04-05T23:24:20.024 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 144/162 2024-04-05T23:24:20.222 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flex-2.6.1-9.el8.x86_64 145/162 2024-04-05T23:24:20.308 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 145/162 2024-04-05T23:24:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : bison-3.0.4-10.el8.x86_64 146/162 2024-04-05T23:24:20.556 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 146/162 2024-04-05T23:24:20.694 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-05T23:24:20.694 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:18.2.2-1173.g8919605b.el8.x86_64 1/162 2024-04-05T23:24:20.694 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 2/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 3/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 4/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1173.g89196 5/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 6/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 7/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 8/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 9/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 10/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 11/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:18.2.2-1173.g8919605b.el8.x86_64 12/162 2024-04-05T23:24:20.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 13/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:18.2.2-1173.g8919605b.el8.x86_ 14/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 15/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 16/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:18.2.2-1173.g8919605b.el8.x86_6 17/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86 18/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 19/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1173.g8919605b.el 20/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:18.2.2-1173.g8919605b.el8. 21/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:18.2.2-1173.g8919605b.el8.x86_6 22/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:18.2.2-1173.g8919605b.el8.x86_64 23/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:18.2.2-1173.g8919605b.el8.x86_64 24/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:18.2.2-1173.g8919605b.el8.x86_64 25/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 26/162 2024-04-05T23:24:20.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 27/162 2024-04-05T23:24:20.697 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:18.2.2-1173.g8919605b.el8.x86_64 28/162 2024-04-05T23:24:20.697 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1173.g8919605b. 29/162 2024-04-05T23:24:20.697 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noa 30/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.n 31/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1173.g891 32/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1173.g8919605b.el 33/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 34/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1173.g8919605b.e 35/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 36/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:18.2.2-1173.g8919605b.el8.noarch 37/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-top-2:18.2.2-1173.g8919605b.el8.noarch 38/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-04-05T23:24:20.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-04-05T23:24:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-04-05T23:24:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 72/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 73/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 74/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 75/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 76/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 77/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 78/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : bison-3.0.4-10.el8.x86_64 79/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flex-2.6.1-9.el8.x86_64 80/162 2024-04-05T23:24:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 81/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 82/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 83/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 84/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 85/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 87/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 88/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 89/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 90/162 2024-04-05T23:24:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 91/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 92/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 93/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 94/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : automake-1.16.1-8.el8.noarch 95/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpq-13.11-1.el8.x86_64 96/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : jq-1.6-7.el8.x86_64 97/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 98/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 99/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 100/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 101/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 102/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 103/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 104/162 2024-04-05T23:24:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 106/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 107/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 108/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 109/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 110/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 111/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 112/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 113/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 114/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 115/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 116/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 117/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 118/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 119/162 2024-04-05T23:24:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 120/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 121/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 122/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 123/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 124/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 125/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 126/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 127/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 129/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 130/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:18.2.2-1173.g8919605b.el8.x86_64 131/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 133/162 2024-04-05T23:24:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-04-05T23:24:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-04-05T23:24:20.719 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-04-05T23:24:20.757 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-04-05T23:24:20.924 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-04-05T23:24:21.110 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-04-05T23:24:21.110 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-05T23:24:21.128 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:18.2.2-1173.g8919605b.el8.x86_64 126/162 2024-04-05T23:24:21.400 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-05T23:24:21.401 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-05T23:24:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-04-05T23:24:21.668 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-05T23:24:21.668 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-05T23:24:21.739 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-04-05T23:24:21.954 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-05T23:24:22.157 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-04-05T23:24:22.171 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 129/162 2024-04-05T23:24:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-04-05T23:24:22.509 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-04-05T23:24:22.616 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-04-05T23:24:22.616 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-05T23:24:22.769 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-04-05T23:24:22.800 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-04-05T23:24:22.833 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-05T23:24:22.833 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-04-05T23:24:23.011 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-04-05T23:24:23.067 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-04-05T23:24:23.067 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-05T23:24:23.259 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-04-05T23:24:23.308 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-05T23:24:23.308 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-05T23:24:23.487 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:18.2.2-1173.g8919605b.el8.x86_ 134/162 2024-04-05T23:24:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-05T23:24:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-05T23:24:23.783 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-05T23:24:23.983 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-04-05T23:24:23.983 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:18.2.2-1173.g8919605b.el8.x86_64 1/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 2/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 3/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 4/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1173.g89196 5/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 6/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 7/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 8/162 2024-04-05T23:24:25.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 9/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 10/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 11/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:18.2.2-1173.g8919605b.el8.x86_64 12/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 13/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:18.2.2-1173.g8919605b.el8.x86_ 14/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 15/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 16/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:18.2.2-1173.g8919605b.el8.x86_6 17/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86 18/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 19/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1173.g8919605b.el 20/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:18.2.2-1173.g8919605b.el8. 21/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:18.2.2-1173.g8919605b.el8.x86_6 22/162 2024-04-05T23:24:25.120 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:18.2.2-1173.g8919605b.el8.x86_64 23/162 2024-04-05T23:24:25.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:18.2.2-1173.g8919605b.el8.x86_64 24/162 2024-04-05T23:24:25.121 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:18.2.2-1173.g8919605b.el8.x86_64 25/162 2024-04-05T23:24:25.133 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 26/162 2024-04-05T23:24:25.133 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 27/162 2024-04-05T23:24:25.133 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:18.2.2-1173.g8919605b.el8.x86_64 28/162 2024-04-05T23:24:25.133 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1173.g8919605b. 29/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noa 30/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.n 31/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1173.g891 32/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1173.g8919605b.el 33/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 34/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1173.g8919605b.e 35/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 36/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:18.2.2-1173.g8919605b.el8.noarch 37/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-top-2:18.2.2-1173.g8919605b.el8.noarch 38/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-04-05T23:24:25.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-04-05T23:24:25.135 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-04-05T23:24:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 83/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 84/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 85/162 2024-04-05T23:24:25.137 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 86/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 90/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 91/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 92/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 93/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : automake-1.16.1-8.el8.noarch 94/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpq-13.11-1.el8.x86_64 95/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-7.el8.x86_64 96/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 97/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 98/162 2024-04-05T23:24:25.138 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 99/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 100/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 101/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 102/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 103/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 104/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 105/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 106/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 107/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 109/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 110/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 112/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 113/162 2024-04-05T23:24:25.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 114/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 115/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 117/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 118/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 119/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 120/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 121/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 122/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 123/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 124/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 125/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 126/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 128/162 2024-04-05T23:24:25.140 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 129/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 130/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:18.2.2-1173.g8919605b.el8.x86_64 131/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 133/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-04-05T23:24:25.141 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-04-05T23:24:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-04-05T23:24:25.143 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-04-05T23:24:25.143 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-04-05T23:24:27.162 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-05T23:24:27.496 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-05T23:24:27.562 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-05T23:24:27.952 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-04-05T23:24:28.291 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-04-05T23:24:28.566 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-04-05T23:24:28.754 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:18.2.2-1173.g8919605b.el8.noarch 139/162 2024-04-05T23:24:29.032 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 140/162 2024-04-05T23:24:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:18.2.2-1173.g8919605b.el8.x86_64 141/162 2024-04-05T23:24:29.707 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-04-05T23:24:29.935 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-04-05T23:24:30.056 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-04-05T23:24:30.261 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-04-05T23:24:30.294 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-04-05T23:24:30.461 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-04-05T23:24:30.486 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-04-05T23:24:30.640 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-04-05T23:24:30.850 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-04-05T23:24:31.022 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-04-05T23:24:31.126 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-04-05T23:24:31.126 INFO:teuthology.orchestra.run.smithi130.stdout:Installed products updated. 2024-04-05T23:24:31.166 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-04-05T23:24:31.166 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-05T23:24:31.465 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-05T23:24:31.465 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-05T23:24:31.722 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:24:31.722 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2024-04-05T23:24:31.722 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-04-05T23:24:31.722 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-04-05T23:24:31.722 INFO:teuthology.orchestra.run.smithi130.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-04-05T23:24:31.722 INFO:teuthology.orchestra.run.smithi130.stdout: libblkid-2.32.1-43.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: libmount-2.32.1-43.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-2.32.1-43.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-2.32.1-43.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-04-05T23:24:31.723 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: automake-1.16.1-8.el8.noarch 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: bison-3.0.4-10.el8.x86_64 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.724 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:31.725 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: flex-2.6.1-9.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: fuse-2.9.7-17.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: jq-1.6-7.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-05T23:24:31.726 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: libpq-13.11-1.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: libtool-2.4.6-25.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-04-05T23:24:31.727 INFO:teuthology.orchestra.run.smithi130.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-05T23:24:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-05T23:24:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-05T23:24:31.730 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-2.6-1.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-05T23:24:31.731 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-05T23:24:31.732 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: uuid-1.6.2-43.el8.x86_64 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:24:31.733 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-04-05T23:24:31.751 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-05T23:24:31.751 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-05T23:24:31.930 DEBUG:teuthology.parallel:result is None 2024-04-05T23:24:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-05T23:24:32.264 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-04-05T23:24:32.449 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-04-05T23:24:32.705 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-04-05T23:24:32.705 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-05T23:24:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-05T23:24:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-04-05T23:24:33.230 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-04-05T23:24:33.231 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-05T23:24:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-05T23:24:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-05T23:24:34.713 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-05T23:24:34.714 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-05T23:24:34.955 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-05T23:24:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-04-05T23:24:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-05T23:24:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-05T23:24:35.726 INFO:teuthology.orchestra.run.smithi016.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-05T23:24:35.726 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:24:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:18.2.2-1173.g8919605b.el8.x86_64 1/162 2024-04-05T23:24:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 2/162 2024-04-05T23:24:36.443 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 3/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 4/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1173.g89196 5/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 6/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 7/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 8/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 9/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 10/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 11/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:18.2.2-1173.g8919605b.el8.x86_64 12/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 13/162 2024-04-05T23:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:18.2.2-1173.g8919605b.el8.x86_ 14/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 15/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 16/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:18.2.2-1173.g8919605b.el8.x86_6 17/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86 18/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 19/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1173.g8919605b.el 20/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:18.2.2-1173.g8919605b.el8. 21/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:18.2.2-1173.g8919605b.el8.x86_6 22/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:18.2.2-1173.g8919605b.el8.x86_64 23/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:18.2.2-1173.g8919605b.el8.x86_64 24/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:18.2.2-1173.g8919605b.el8.x86_64 25/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 26/162 2024-04-05T23:24:36.446 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 27/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:18.2.2-1173.g8919605b.el8.x86_64 28/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1173.g8919605b. 29/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noa 30/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.n 31/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1173.g891 32/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1173.g8919605b.el 33/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 34/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1173.g8919605b.e 35/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 36/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:18.2.2-1173.g8919605b.el8.noarch 37/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:18.2.2-1173.g8919605b.el8.noarch 38/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-04-05T23:24:36.447 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-04-05T23:24:36.448 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-04-05T23:24:36.449 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-04-05T23:24:36.450 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-04-05T23:24:36.450 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-04-05T23:24:36.450 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-04-05T23:24:36.450 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 72/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 73/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 75/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 76/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 78/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 80/162 2024-04-05T23:24:36.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 81/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 82/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 83/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 84/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 85/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 86/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 87/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 88/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 89/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 90/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 91/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 92/162 2024-04-05T23:24:36.452 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 93/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 94/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 95/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 96/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 97/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 98/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 99/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 100/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 103/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 104/162 2024-04-05T23:24:36.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 105/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.0.4-10.el8.x86_64 106/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.1-9.el8.x86_64 107/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 109/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 110/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 111/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 113/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 114/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 115/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 116/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 117/162 2024-04-05T23:24:36.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 118/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 119/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 120/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 121/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.1-8.el8.noarch 122/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpq-13.11-1.el8.x86_64 123/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.6-7.el8.x86_64 124/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 125/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 126/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 127/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 128/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 129/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 130/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:18.2.2-1173.g8919605b.el8.x86_64 131/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-04-05T23:24:36.455 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 133/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-04-05T23:24:36.456 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-04-05T23:24:36.457 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-04-05T23:24:36.457 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-04-05T23:24:36.457 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-04-05T23:24:36.457 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-04-05T23:24:36.457 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-04-05T23:24:36.457 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-04-05T23:24:36.457 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-04-05T23:24:36.457 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-04-05T23:24:36.458 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-04-05T23:24:36.458 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-04-05T23:24:36.458 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-04-05T23:24:36.458 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-04-05T23:24:36.458 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-04-05T23:24:36.458 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-04-05T23:24:36.458 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-04-05T23:24:36.936 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-04-05T23:24:36.936 INFO:teuthology.orchestra.run.smithi027.stdout:Installed products updated. 2024-04-05T23:24:38.366 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:24:38.366 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: libblkid-2.32.1-43.el8.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: libmount-2.32.1-43.el8.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-2.32.1-43.el8.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-04-05T23:24:38.367 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-2.32.1-43.el8.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: automake-1.16.1-8.el8.noarch 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: bison-3.0.4-10.el8.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.368 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.369 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: flex-2.6.1-9.el8.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: fuse-2.9.7-17.el8.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.6-7.el8.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-04-05T23:24:38.370 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: libpq-13.11-1.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-05T23:24:38.371 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: libtool-2.4.6-25.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-04-05T23:24:38.372 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-05T23:24:38.374 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-2.6-1.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-05T23:24:38.375 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-05T23:24:38.376 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-05T23:24:38.377 INFO:teuthology.orchestra.run.smithi027.stdout: uuid-1.6.2-43.el8.x86_64 2024-04-05T23:24:38.378 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-04-05T23:24:38.378 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-05T23:24:38.378 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-04-05T23:24:38.378 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-05T23:24:38.378 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-05T23:24:40.192 DEBUG:teuthology.parallel:result is None 2024-04-05T23:24:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-04-05T23:24:47.454 INFO:teuthology.orchestra.run.smithi016.stdout:Installed products updated. 2024-04-05T23:24:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:24:48.131 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-04-05T23:24:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-04-05T23:24:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-04-05T23:24:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-04-05T23:24:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: libblkid-2.32.1-43.el8.x86_64 2024-04-05T23:24:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-04-05T23:24:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: libmount-2.32.1-43.el8.x86_64 2024-04-05T23:24:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-2.32.1-43.el8.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-2.32.1-43.el8.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.1-8.el8.noarch 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.0.4-10.el8.x86_64 2024-04-05T23:24:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:18.2.2-1173.g8919605b.el8.noarch 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.1-9.el8.x86_64 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.7-17.el8.x86_64 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: jq-1.6-7.el8.x86_64 2024-04-05T23:24:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: libpq-13.11-1.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-05T23:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-25.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-05T23:24:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-05T23:24:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-05T23:24:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-05T23:24:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-2.6-1.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-05T23:24:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:18.2.2-1173.g8919605b.el8.x86_64 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-05T23:24:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-05T23:24:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-43.el8.x86_64 2024-04-05T23:24:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-04-05T23:24:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-05T23:24:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-04-05T23:24:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:24:48.142 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-05T23:24:48.561 DEBUG:teuthology.parallel:result is None 2024-04-05T23:24:48.561 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:24:48.783 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-05T23:24:48.836 INFO:teuthology.orchestra.run.smithi016.stdout:18.2.2-1173.g8919605b.el8 2024-04-05T23:24:48.836 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1173.g8919605b.el8 2024-04-05T23:24:48.836 INFO:teuthology.task.install:The correct ceph version 18.2.2-1173.g8919605b is installed. 2024-04-05T23:24:48.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:24:49.072 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-05T23:24:49.101 INFO:teuthology.orchestra.run.smithi027.stdout:18.2.2-1173.g8919605b.el8 2024-04-05T23:24:49.102 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1173.g8919605b.el8 2024-04-05T23:24:49.102 INFO:teuthology.task.install:The correct ceph version 18.2.2-1173.g8919605b is installed. 2024-04-05T23:24:49.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:24:49.327 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-05T23:24:49.355 INFO:teuthology.orchestra.run.smithi130.stdout:18.2.2-1173.g8919605b.el8 2024-04-05T23:24:49.356 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1173.g8919605b.el8 2024-04-05T23:24:49.356 INFO:teuthology.task.install:The correct ceph version 18.2.2-1173.g8919605b is installed. 2024-04-05T23:24:49.358 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-05T23:24:49.358 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:24:49.358 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-05T23:24:49.452 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:24:49.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-05T23:24:49.504 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:24:49.504 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-05T23:24:49.552 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-05T23:24:49.553 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:24:49.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-04-05T23:24:49.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-05T23:24:49.726 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:24:49.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-04-05T23:24:49.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-05T23:24:49.832 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:24:49.832 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2024-04-05T23:24:49.865 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-05T23:24:49.937 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-05T23:24:49.937 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:24:49.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-05T23:24:50.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-05T23:24:50.107 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:24:50.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-05T23:24:50.141 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-05T23:24:50.212 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:24:50.212 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-05T23:24:50.245 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-05T23:24:50.315 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-05T23:24:50.316 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:24:50.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-04-05T23:24:50.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-05T23:24:50.485 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:24:50.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-04-05T23:24:50.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-05T23:24:50.591 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:24:50.591 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2024-04-05T23:24:50.623 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-05T23:24:50.695 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-05T23:24:50.805 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-05T23:24:50.806 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:24:50.806 INFO:tasks.cephadm:Cluster fsid is b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:24:50.806 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-05T23:24:50.806 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.16', 'mon.b': '172.21.15.27', 'mon.c': '172.21.15.130'} 2024-04-05T23:24:50.807 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-04-05T23:24:50.807 INFO:tasks.cephadm:First mgr is x 2024-04-05T23:24:50.807 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-05T23:24:50.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-04-05T23:24:50.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-04-05T23:24:50.909 DEBUG:teuthology.orchestra.run.smithi130:> sudo hostname $(hostname -s) 2024-04-05T23:24:50.942 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-05T23:24:50.942 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:24:51.075 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/78290/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.2-1173-g8919605b', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.2-1173.g8919605b'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-04-05 14:10:52.055812', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri4-testing-2024-04-04-1128-reef', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-05T23:24:51.210 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-yuri4-testing-2024-04-04-1128-reef, sha1 8919605b0a5dc1f1650c8aec3051c77c0e3e240e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:24:51.212 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/x86_64/flavors/default/cephadm 2024-04-05T23:24:51.212 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-05T23:24:51.672 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Apr 5 23:24 /home/ubuntu/cephtest/cephadm 2024-04-05T23:24:51.673 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-05T23:24:52.375 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Apr 5 23:24 /home/ubuntu/cephtest/cephadm 2024-04-05T23:24:52.375 DEBUG:teuthology.orchestra.run.smithi130:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-05T23:24:52.772 INFO:teuthology.orchestra.run.smithi130.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Apr 5 23:24 /home/ubuntu/cephtest/cephadm 2024-04-05T23:24:52.772 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-05T23:24:52.822 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-05T23:24:52.844 DEBUG:teuthology.orchestra.run.smithi130:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-05T23:24:52.883 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e on all hosts... 2024-04-05T23:24:52.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e pull 2024-04-05T23:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e pull 2024-04-05T23:24:52.889 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e pull 2024-04-05T23:24:53.793 INFO:teuthology.orchestra.run.smithi130.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e... 2024-04-05T23:24:53.932 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e... 2024-04-05T23:24:54.935 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e... 2024-04-05T23:25:23.924 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-05T23:25:23.925 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)", 2024-04-05T23:25:23.925 INFO:teuthology.orchestra.run.smithi130.stdout: "image_id": "067bdb2497923b6516e58a75510884d6c4a2f4bafeca4e7c8488b4cf365cae8b", 2024-04-05T23:25:23.925 INFO:teuthology.orchestra.run.smithi130.stdout: "repo_digests": [ 2024-04-05T23:25:23.925 INFO:teuthology.orchestra.run.smithi130.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da" 2024-04-05T23:25:23.925 INFO:teuthology.orchestra.run.smithi130.stdout: ] 2024-04-05T23:25:23.925 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-05T23:25:24.501 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-05T23:25:24.501 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)", 2024-04-05T23:25:24.501 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "067bdb2497923b6516e58a75510884d6c4a2f4bafeca4e7c8488b4cf365cae8b", 2024-04-05T23:25:24.501 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-04-05T23:25:24.501 INFO:teuthology.orchestra.run.smithi027.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da" 2024-04-05T23:25:24.502 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-05T23:25:24.502 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-05T23:25:27.513 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:25:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)", 2024-04-05T23:25:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "067bdb2497923b6516e58a75510884d6c4a2f4bafeca4e7c8488b4cf365cae8b", 2024-04-05T23:25:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-04-05T23:25:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da" 2024-04-05T23:25:27.513 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:25:27.513 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:25:27.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-04-05T23:25:27.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-04-05T23:25:27.647 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph 2024-04-05T23:25:27.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-04-05T23:25:27.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-04-05T23:25:27.783 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /etc/ceph 2024-04-05T23:25:27.815 INFO:tasks.cephadm:Writing seed config... 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-05T23:25:27.817 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-05T23:25:27.818 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-05T23:25:27.819 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-05T23:25:27.820 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-05T23:25:27.820 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-05T23:25:27.820 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-05T23:25:27.820 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-05T23:25:27.820 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-05T23:25:27.820 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-05T23:25:27.820 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-05T23:25:27.821 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:25:27.821 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-05T23:25:27.862 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b3471b30-f3a3-11ee-b649-cb9ed24678a4 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-05T23:25:27.862 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.a.service 2024-04-05T23:25:27.907 DEBUG:teuthology.orchestra.run.smithi016:mgr.x> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.x.service 2024-04-05T23:25:27.951 INFO:tasks.cephadm:Bootstrapping... 2024-04-05T23:25:27.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e -v bootstrap --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-05T23:25:27.971 INFO:journalctl@ceph.mon.a.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:25:28.017 INFO:journalctl@ceph.mgr.x.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:25:28.176 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-05T23:25:28.176 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e', '-v', 'bootstrap', '--fsid', 'b3471b30-f3a3-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-04-05T23:25:28.222 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.6.1 2024-04-05T23:25:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-05T23:25:28.223 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-04-05T23:25:28.267 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.6.1 2024-04-05T23:25:28.267 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-04-05T23:25:28.267 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-04-05T23:25:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-05T23:25:28.281 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-05T23:25:28.293 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-05T23:25:28.293 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-05T23:25:28.306 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-04-05T23:25:28.318 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-04-05T23:25:28.319 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-04-05T23:25:28.319 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-04-05T23:25:28.361 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.6.1 2024-04-05T23:25:28.362 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 4.6.1 is present 2024-04-05T23:25:28.362 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-04-05T23:25:28.362 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-04-05T23:25:28.374 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-05T23:25:28.375 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-05T23:25:28.387 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-05T23:25:28.387 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-05T23:25:28.399 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-04-05T23:25:28.412 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-04-05T23:25:28.412 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-04-05T23:25:28.412 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-04-05T23:25:28.412 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:25:28.412 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 139644376590824 on /run/cephadm/b3471b30-f3a3-11ee-b649-cb9ed24678a4.lock 2024-04-05T23:25:28.412 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 139644376590824 acquired on /run/cephadm/b3471b30-f3a3-11ee-b649-cb9ed24678a4.lock 2024-04-05T23:25:28.413 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-04-05T23:25:28.413 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-04-05T23:25:28.413 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-05T23:25:28.421 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2024-04-05T23:25:28.421 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-04-05T23:25:28.428 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-05T23:25:28.428 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-04-05T23:25:28.428 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1788sec hoplimit 64 pref medium 2024-04-05T23:25:28.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-05T23:25:28.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-05T23:25:28.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-05T23:25:28.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-05T23:25:28.434 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-04-05T23:25:28.435 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-05T23:25:28.435 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-04-05T23:25:28.435 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-04-05T23:25:28.435 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-05T23:25:28.436 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-05T23:25:28.437 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e... 2024-04-05T23:25:28.860 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 067bdb2497923b6516e58a75510884d6c4a2f4bafeca4e7c8488b4cf365cae8b 2024-04-05T23:25:28.860 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e... 2024-04-05T23:25:28.861 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-04-05T23:25:28.861 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:b8f36992b3dd8f10d7089deb1fd5dac73b9d9c86ead8d07ff63bacb405e05b14 2024-04-05T23:25:28.861 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:77070339c81bcb5c53bbbf3f31a47f7187e5393cb0e096452852a31192d97570 2024-04-05T23:25:28.861 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:067bdb2497923b6516e58a75510884d6c4a2f4bafeca4e7c8488b4cf365cae8b 2024-04-05T23:25:28.861 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-05T23:25:29.578 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable) 2024-04-05T23:25:29.578 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable) 2024-04-05T23:25:29.579 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-04-05T23:25:30.176 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-04-05T23:25:30.177 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-04-05T23:25:30.759 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQBqiBBmtrVKGxAA0kt58O3nGRypevPMKKhPJA== 2024-04-05T23:25:31.371 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQBriBBmaQH7AhAAGIK3hfTTFrdCTjckp6Lw4A== 2024-04-05T23:25:31.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQBriBBmq6+sJxAAAnq7r8rKv9klZ0b0ftWyXQ== 2024-04-05T23:25:31.981 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for a [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = pacific 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:25:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-04-05T23:25:33.218 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.a on 2024-04-05T23:25:33.461 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-05T23:25:33.655 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-05T23:25:33.890 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4.target. 2024-04-05T23:25:33.890 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4.target. 2024-04-05T23:25:34.248 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.a 2024-04-05T23:25:34.248 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.a.service: Unit ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.a.service not loaded. 2024-04-05T23:25:34.452 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4.target.wants/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@.service. 2024-04-05T23:25:34.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:34 smithi016 systemd[1]: Starting Ceph mon.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:25:35.213 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T23:25:35.213 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-05T23:25:35.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:35 smithi016 bash[37720]: 1a3aad2f5a1d37438a30cd638828334060a233e057ebd73d06c0148281486fa3 2024-04-05T23:25:35.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:35 smithi016 systemd[1]: Started Ceph mon.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:25:35.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:35 smithi016 ceph-mon[37758]: mkfs b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:25:35.231 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T23:25:35.231 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-05T23:25:35.231 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-05T23:25:35.231 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-04-05T23:25:35.232 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-04-05T23:25:35.232 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-04-05T23:25:35.432 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:35 smithi016 ceph-mon[37758]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-05T23:25:36.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-04-05T23:25:36.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:25:36.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-05T23:25:36.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:25:36.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-04-05T23:25:36.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.591226s) 2024-04-05T23:25:36.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-05T23:25:36.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-05T23:25:36.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:25:36.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-04-05T23:25:36.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-05T23:25:36.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-05T23:25:36.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-05T23:25:36.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-04-05T23:25:36.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:25:36.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.731+0000 7f5f02400700 1 Processor -- start 2024-04-05T23:25:36.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.732+0000 7f5f02400700 1 -- start start 2024-04-05T23:25:36.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.732+0000 7f5f02400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:36.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.732+0000 7f5f02400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc1070b0 con 0x7f5efc1066c0 2024-04-05T23:25:36.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.733+0000 7f5efb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:36.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.733+0000 7f5efb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42084/0 (socket says 172.21.15.16:42084) 2024-04-05T23:25:36.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.733+0000 7f5efb400700 1 -- 172.21.15.16:0/842165784 learned_addr learned my addr 172.21.15.16:0/842165784 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:36.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.733+0000 7f5efb400700 1 -- 172.21.15.16:0/842165784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc1078c0 con 0x7f5efc1066c0 2024-04-05T23:25:36.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.733+0000 7f5efb400700 1 --2- 172.21.15.16:0/842165784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc106ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0098a0 tx=0x7f5eec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adccdfce13c1db03 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:36.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.734+0000 7f5efa000700 1 -- 172.21.15.16:0/842165784 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5eec004160 con 0x7f5efc1066c0 2024-04-05T23:25:36.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.734+0000 7f5efa000700 1 -- 172.21.15.16:0/842165784 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec0042c0 con 0x7f5efc1066c0 2024-04-05T23:25:36.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.734+0000 7f5efa000700 1 -- 172.21.15.16:0/842165784 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5eec0045a0 con 0x7f5efc1066c0 2024-04-05T23:25:36.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.734+0000 7f5f02400700 1 -- 172.21.15.16:0/842165784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 msgr2=0x7f5efc106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:36.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.734+0000 7f5f02400700 1 --2- 172.21.15.16:0/842165784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc106ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0098a0 tx=0x7f5eec009bb0 comp rx=0 tx=0).stop 2024-04-05T23:25:36.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.734+0000 7f5f02400700 1 -- 172.21.15.16:0/842165784 shutdown_connections 2024-04-05T23:25:36.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.734+0000 7f5f02400700 1 --2- 172.21.15.16:0/842165784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc106ae0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:36.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.734+0000 7f5f02400700 1 -- 172.21.15.16:0/842165784 >> 172.21.15.16:0/842165784 conn(0x7f5efc101c60 msgr2=0x7f5efc1040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:36.179 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.735+0000 7f5f02400700 1 -- 172.21.15.16:0/842165784 shutdown_connections 2024-04-05T23:25:36.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.735+0000 7f5f02400700 1 -- 172.21.15.16:0/842165784 wait complete. 2024-04-05T23:25:36.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.736+0000 7f5f02400700 1 Processor -- start 2024-04-05T23:25:36.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.736+0000 7f5f02400700 1 -- start start 2024-04-05T23:25:36.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.737+0000 7f5f02400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc19be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:36.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.737+0000 7f5f02400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc19c340 con 0x7f5efc1066c0 2024-04-05T23:25:36.180 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.737+0000 7f5efb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc19be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:36.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.737+0000 7f5efb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc19be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42094/0 (socket says 172.21.15.16:42094) 2024-04-05T23:25:36.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.737+0000 7f5efb400700 1 -- 172.21.15.16:0/4269477105 learned_addr learned my addr 172.21.15.16:0/4269477105 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:36.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.737+0000 7f5efb400700 1 -- 172.21.15.16:0/4269477105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec009550 con 0x7f5efc1066c0 2024-04-05T23:25:36.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.738+0000 7f5efb400700 1 --2- 172.21.15.16:0/4269477105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc19be00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5eec004000 tx=0x7f5eec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:36.181 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.738+0000 7f5ef3e00700 1 -- 172.21.15.16:0/4269477105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5eec011630 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.738+0000 7f5ef3e00700 1 -- 172.21.15.16:0/4269477105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec011790 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.738+0000 7f5ef3e00700 1 -- 172.21.15.16:0/4269477105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5eec011a30 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.738+0000 7f5f02400700 1 -- 172.21.15.16:0/4269477105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5efc19c540 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.738+0000 7f5f02400700 1 -- 172.21.15.16:0/4269477105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5efc19c9e0 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.738+0000 7f5ef3e00700 1 -- 172.21.15.16:0/4269477105 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f5eec022070 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.738+0000 7f5ef3e00700 1 -- 172.21.15.16:0/4269477105 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5eec01c790 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.739+0000 7f5f02400700 1 -- 172.21.15.16:0/4269477105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5efc107c30 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.742+0000 7f5ef3e00700 1 -- 172.21.15.16:0/4269477105 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f5eec01bcf0 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.819+0000 7f5f02400700 1 -- 172.21.15.16:0/4269477105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f5efc19f4a0 con 0x7f5efc1066c0 2024-04-05T23:25:36.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.819+0000 7f5ef3e00700 1 -- 172.21.15.16:0/4269477105 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f5eec01b450 con 0x7f5efc1066c0 2024-04-05T23:25:36.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.820+0000 7f5f02400700 1 -- 172.21.15.16:0/4269477105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 msgr2=0x7f5efc19be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:36.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.820+0000 7f5f02400700 1 --2- 172.21.15.16:0/4269477105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc19be00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5eec004000 tx=0x7f5eec004750 comp rx=0 tx=0).stop 2024-04-05T23:25:36.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.820+0000 7f5f02400700 1 -- 172.21.15.16:0/4269477105 shutdown_connections 2024-04-05T23:25:36.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.820+0000 7f5f02400700 1 --2- 172.21.15.16:0/4269477105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5efc1066c0 0x7f5efc19be00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:36.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.820+0000 7f5f02400700 1 -- 172.21.15.16:0/4269477105 >> 172.21.15.16:0/4269477105 conn(0x7f5efc101c60 msgr2=0x7f5efc078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:36.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.821+0000 7f5f02400700 1 -- 172.21.15.16:0/4269477105 shutdown_connections 2024-04-05T23:25:36.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:35.821+0000 7f5f02400700 1 -- 172.21.15.16:0/4269477105 wait complete. 2024-04-05T23:25:36.184 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-04-05T23:25:36.184 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-04-05T23:25:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:36 smithi016 ceph-mon[37758]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-05T23:25:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:36 smithi016 ceph-mon[37758]: monmap e1: 1 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T23:25:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:36 smithi016 ceph-mon[37758]: fsmap 2024-04-05T23:25:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:36 smithi016 ceph-mon[37758]: osdmap e1: 0 total, 0 up, 0 in 2024-04-05T23:25:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:36 smithi016 ceph-mon[37758]: mgrmap e1: no daemons active 2024-04-05T23:25:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:36 smithi016 ceph-mon[37758]: from='client.? 172.21.15.16:0/4269477105' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-05T23:25:37.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.621+0000 7fc176800700 1 Processor -- start 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.622+0000 7fc176800700 1 -- start start 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.622+0000 7fc176800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc170106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.622+0000 7fc176800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1701070b0 con 0x7fc1701066c0 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.623+0000 7fc16f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc170106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.623+0000 7fc16f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc170106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42106/0 (socket says 172.21.15.16:42106) 2024-04-05T23:25:37.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.623+0000 7fc16f400700 1 -- 172.21.15.16:0/852653734 learned_addr learned my addr 172.21.15.16:0/852653734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.623+0000 7fc16f400700 1 -- 172.21.15.16:0/852653734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1701078c0 con 0x7fc1701066c0 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.623+0000 7fc16f400700 1 --2- 172.21.15.16:0/852653734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc170106ae0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc1600098a0 tx=0x7fc160009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ff8bfac08363f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc16ea00700 1 -- 172.21.15.16:0/852653734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc160004160 con 0x7fc1701066c0 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc16ea00700 1 -- 172.21.15.16:0/852653734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1600042c0 con 0x7fc1701066c0 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc16ea00700 1 -- 172.21.15.16:0/852653734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1600045a0 con 0x7fc1701066c0 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc176800700 1 -- 172.21.15.16:0/852653734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 msgr2=0x7fc170106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc176800700 1 --2- 172.21.15.16:0/852653734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc170106ae0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc1600098a0 tx=0x7fc160009bb0 comp rx=0 tx=0).stop 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc176800700 1 -- 172.21.15.16:0/852653734 shutdown_connections 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc176800700 1 --2- 172.21.15.16:0/852653734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc170106ae0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc176800700 1 -- 172.21.15.16:0/852653734 >> 172.21.15.16:0/852653734 conn(0x7fc170101c60 msgr2=0x7fc1701040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc176800700 1 -- 172.21.15.16:0/852653734 shutdown_connections 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.624+0000 7fc176800700 1 -- 172.21.15.16:0/852653734 wait complete. 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.625+0000 7fc176800700 1 Processor -- start 2024-04-05T23:25:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.626+0000 7fc176800700 1 -- start start 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.626+0000 7fc176800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc17019be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.626+0000 7fc176800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17019c360 con 0x7fc1701066c0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.626+0000 7fc16f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc17019be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.626+0000 7fc16f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc17019be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42114/0 (socket says 172.21.15.16:42114) 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.626+0000 7fc16f400700 1 -- 172.21.15.16:0/3434728863 learned_addr learned my addr 172.21.15.16:0/3434728863 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.627+0000 7fc16f400700 1 -- 172.21.15.16:0/3434728863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc160009550 con 0x7fc1701066c0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.627+0000 7fc16f400700 1 --2- 172.21.15.16:0/3434728863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc17019be20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc160004000 tx=0x7fc160004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.627+0000 7fc16cc00700 1 -- 172.21.15.16:0/3434728863 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc160011630 con 0x7fc1701066c0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.627+0000 7fc16cc00700 1 -- 172.21.15.16:0/3434728863 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc160011790 con 0x7fc1701066c0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.627+0000 7fc16cc00700 1 -- 172.21.15.16:0/3434728863 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc160011a30 con 0x7fc1701066c0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.627+0000 7fc176800700 1 -- 172.21.15.16:0/3434728863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17019c560 con 0x7fc1701066c0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.627+0000 7fc176800700 1 -- 172.21.15.16:0/3434728863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17019ca00 con 0x7fc1701066c0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.627+0000 7fc16cc00700 1 -- 172.21.15.16:0/3434728863 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fc160022070 con 0x7fc1701066c0 2024-04-05T23:25:37.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.628+0000 7fc16cc00700 1 -- 172.21.15.16:0/3434728863 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc16001c450 con 0x7fc1701066c0 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.628+0000 7fc176800700 1 -- 172.21.15.16:0/3434728863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc170195d80 con 0x7fc1701066c0 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.631+0000 7fc16cc00700 1 -- 172.21.15.16:0/3434728863 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7fc16001b450 con 0x7fc1701066c0 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.704+0000 7fc176800700 1 -- 172.21.15.16:0/3434728863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fc170107ee0 con 0x7fc1701066c0 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.708+0000 7fc16cc00700 1 -- 172.21.15.16:0/3434728863 <== mon.0 v2:172.21.15.16:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc16001b5f0 con 0x7fc1701066c0 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.708+0000 7fc16cc00700 1 -- 172.21.15.16:0/3434728863 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fc1600375f0 con 0x7fc1701066c0 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.709+0000 7fc176800700 1 -- 172.21.15.16:0/3434728863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 msgr2=0x7fc17019be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.709+0000 7fc176800700 1 --2- 172.21.15.16:0/3434728863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc17019be20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc160004000 tx=0x7fc160004750 comp rx=0 tx=0).stop 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.709+0000 7fc176800700 1 -- 172.21.15.16:0/3434728863 shutdown_connections 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.709+0000 7fc176800700 1 --2- 172.21.15.16:0/3434728863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1701066c0 0x7fc17019be20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.709+0000 7fc176800700 1 -- 172.21.15.16:0/3434728863 >> 172.21.15.16:0/3434728863 conn(0x7fc170101c60 msgr2=0x7fc170192ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.709+0000 7fc176800700 1 -- 172.21.15.16:0/3434728863 shutdown_connections 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:36.709+0000 7fc176800700 1 -- 172.21.15.16:0/3434728863 wait complete. 2024-04-05T23:25:37.082 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-04-05T23:25:37.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:37 smithi016 ceph-mon[37758]: from='client.? 172.21.15.16:0/3434728863' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-05T23:25:37.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:37 smithi016 ceph-mon[37758]: from='client.? 172.21.15.16:0/3434728863' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-05T23:25:37.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.478+0000 7f3eddc00700 1 Processor -- start 2024-04-05T23:25:37.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.479+0000 7f3eddc00700 1 -- start start 2024-04-05T23:25:37.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.479+0000 7f3eddc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed8079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:37.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.479+0000 7f3eddc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed80797e0 con 0x7f3ed807adb0 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.479+0000 7f3ed6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed8079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.479+0000 7f3ed6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed8079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42122/0 (socket says 172.21.15.16:42122) 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.479+0000 7f3ed6a00700 1 -- 172.21.15.16:0/1425642408 learned_addr learned my addr 172.21.15.16:0/1425642408 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.480+0000 7f3ed6a00700 1 -- 172.21.15.16:0/1425642408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed8079920 con 0x7f3ed807adb0 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.480+0000 7f3ed6a00700 1 --2- 172.21.15.16:0/1425642408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed8079210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80098a0 tx=0x7f3ec8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d3d4d875281130 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.480+0000 7f3ed5600700 1 -- 172.21.15.16:0/1425642408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ec8004160 con 0x7f3ed807adb0 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.480+0000 7f3ed5600700 1 -- 172.21.15.16:0/1425642408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ec80042c0 con 0x7f3ed807adb0 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.480+0000 7f3ed5600700 1 -- 172.21.15.16:0/1425642408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ec8011a00 con 0x7f3ed807adb0 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.481+0000 7f3eddc00700 1 -- 172.21.15.16:0/1425642408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 msgr2=0x7f3ed8079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.481+0000 7f3eddc00700 1 --2- 172.21.15.16:0/1425642408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed8079210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80098a0 tx=0x7f3ec8009bb0 comp rx=0 tx=0).stop 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.481+0000 7f3eddc00700 1 -- 172.21.15.16:0/1425642408 shutdown_connections 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.481+0000 7f3eddc00700 1 --2- 172.21.15.16:0/1425642408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed8079210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:37.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.481+0000 7f3eddc00700 1 -- 172.21.15.16:0/1425642408 >> 172.21.15.16:0/1425642408 conn(0x7f3ed8101c60 msgr2=0x7f3ed81040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.481+0000 7f3eddc00700 1 -- 172.21.15.16:0/1425642408 shutdown_connections 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.481+0000 7f3eddc00700 1 -- 172.21.15.16:0/1425642408 wait complete. 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.482+0000 7f3eddc00700 1 Processor -- start 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.482+0000 7f3eddc00700 1 -- start start 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.482+0000 7f3eddc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.482+0000 7f3eddc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed819cbd0 con 0x7f3ed807adb0 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.483+0000 7f3ed6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.483+0000 7f3ed6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42130/0 (socket says 172.21.15.16:42130) 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.483+0000 7f3ed6a00700 1 -- 172.21.15.16:0/3693853285 learned_addr learned my addr 172.21.15.16:0/3693853285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.483+0000 7f3ed6a00700 1 -- 172.21.15.16:0/3693853285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec8009550 con 0x7f3ed807adb0 2024-04-05T23:25:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.483+0000 7f3ed6a00700 1 --2- 172.21.15.16:0/3693853285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed819c690 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009130 tx=0x7f3ec8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.483+0000 7f3ecf400700 1 -- 172.21.15.16:0/3693853285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ec8003ab0 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.484+0000 7f3ecf400700 1 -- 172.21.15.16:0/3693853285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ec8003c10 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.484+0000 7f3eddc00700 1 -- 172.21.15.16:0/3693853285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed819cdd0 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.484+0000 7f3ecf400700 1 -- 172.21.15.16:0/3693853285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ec80247d0 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.484+0000 7f3eddc00700 1 -- 172.21.15.16:0/3693853285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed819d270 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.484+0000 7f3ecf400700 1 -- 172.21.15.16:0/3693853285 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f3ec8024d50 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.484+0000 7f3ecf400700 1 -- 172.21.15.16:0/3693853285 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3ec80193c0 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.485+0000 7f3eddc00700 1 -- 172.21.15.16:0/3693853285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed810ca90 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.488+0000 7f3ecf400700 1 -- 172.21.15.16:0/3693853285 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f3ec801abb0 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.560+0000 7f3eddc00700 1 -- 172.21.15.16:0/3693853285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3ed810cc80 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.560+0000 7f3ecf400700 1 -- 172.21.15.16:0/3693853285 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f3ec801ad50 con 0x7f3ed807adb0 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.561+0000 7f3eddc00700 1 -- 172.21.15.16:0/3693853285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 msgr2=0x7f3ed819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:37.891 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.561+0000 7f3eddc00700 1 --2- 172.21.15.16:0/3693853285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed819c690 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009130 tx=0x7f3ec8011ad0 comp rx=0 tx=0).stop 2024-04-05T23:25:37.892 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.561+0000 7f3eddc00700 1 -- 172.21.15.16:0/3693853285 shutdown_connections 2024-04-05T23:25:37.892 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.561+0000 7f3eddc00700 1 --2- 172.21.15.16:0/3693853285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ed807adb0 0x7f3ed819c690 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:37.892 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.561+0000 7f3eddc00700 1 -- 172.21.15.16:0/3693853285 >> 172.21.15.16:0/3693853285 conn(0x7f3ed8101c60 msgr2=0x7f3ed81028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:37.892 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.562+0000 7f3eddc00700 1 -- 172.21.15.16:0/3693853285 shutdown_connections 2024-04-05T23:25:37.892 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:37.562+0000 7f3eddc00700 1 -- 172.21.15.16:0/3693853285 wait complete. 2024-04-05T23:25:37.892 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-04-05T23:25:38.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:37 smithi016 systemd[1]: Stopping Ceph mon.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:25:38.498 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:38 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a[37735]: 2024-04-05T23:25:38.198+0000 7f6e69200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:25:38.498 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:38 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a[37735]: 2024-04-05T23:25:38.198+0000 7f6e69200700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-05T23:25:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:38 smithi016 bash[38088]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a 2024-04-05T23:25:39.203 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:38 smithi016 systemd[1]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.a.service: Succeeded. 2024-04-05T23:25:39.204 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:38 smithi016 systemd[1]: Stopped Ceph mon.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:25:39.204 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:38 smithi016 systemd[1]: Starting Ceph mon.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:25:39.285 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-05T23:25:39.525 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 bash[38211]: a904b918c1131e191261a85577ea7a11e56fbd3bcae0a7ea47618acf80c1d221 2024-04-05T23:25:39.525 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 systemd[1]: Started Ceph mon.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:25:39.525 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: set uid:gid to 167:167 (ceph:ceph) 2024-04-05T23:25:39.525 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable), process ceph-mon, pid 7 2024-04-05T23:25:39.525 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: pidfile_write: ignore empty --pid-file 2024-04-05T23:25:39.525 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: load: jerasure load: lrc 2024-04-05T23:25:39.525 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: RocksDB version: 7.9.2 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Git sha 0 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Compile date 2024-04-05 12:53:35 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: DB SUMMARY 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: DB Session ID: IRB6Q196WRYX2GNXCWC2 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: CURRENT file: CURRENT 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: IDENTITY file: IDENTITY 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98282 ; 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.error_if_exists: 0 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.create_if_missing: 0 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.paranoid_checks: 1 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-05T23:25:39.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.env: 0x55db4683f7a0 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.fs: PosixFileSystem 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.info_log: 0x55db48467540 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_file_opening_threads: 16 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.statistics: (nil) 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.use_fsync: 0 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_log_file_size: 0 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.keep_log_file_num: 1000 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.recycle_log_file_num: 0 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.allow_fallocate: 1 2024-04-05T23:25:39.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.allow_mmap_reads: 0 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.allow_mmap_writes: 0 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.use_direct_reads: 0 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.create_missing_column_families: 0 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.db_log_dir: 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.wal_dir: 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.advise_random_on_open: 1 2024-04-05T23:25:39.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.db_write_buffer_size: 0 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.write_buffer_manager: 0x55db476fc320 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.rate_limiter: (nil) 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.wal_recovery_mode: 2 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.enable_thread_tracking: 0 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.enable_pipelined_write: 0 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.unordered_write: 0 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-05T23:25:39.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.row_cache: None 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.wal_filter: None 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.allow_ingest_behind: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.two_write_queues: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.manual_wal_flush: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.wal_compression: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.atomic_flush: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.log_readahead_size: 0 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-05T23:25:39.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.best_efforts_recovery: 0 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.allow_data_in_errors: 0 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.db_host_id: __hostname__ 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_background_jobs: 2 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_background_compactions: -1 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_subcompactions: 1 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_total_wal_size: 0 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-05T23:25:39.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_open_files: -1 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bytes_per_sync: 0 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_readahead_size: 0 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_background_flushes: -1 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Compression algorithms supported: 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: kZSTD supported: 0 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: kXpressCompression supported: 0 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: kLZ4HCCompression supported: 1 2024-04-05T23:25:39.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: kZlibCompression supported: 1 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: kSnappyCompression supported: 1 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: kLZ4Compression supported: 1 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: kBZip2Compression supported: 0 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.merge_operator: 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_filter: None 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_filter_factory: None 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.sst_partitioner_factory: None 2024-04-05T23:25:39.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55db48467640) 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-04-05T23:25:39.534 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x55db4777d1f0 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-04-05T23:25:39.535 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-04-05T23:25:39.536 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.write_buffer_size: 33554432 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_write_buffer_number: 2 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression: NoCompression 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression: Disabled 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.prefix_extractor: nullptr 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.num_levels: 7 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-05T23:25:39.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-05T23:25:39.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-05T23:25:39.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression_opts.level: 32767 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression_opts.strategy: 0 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression_opts.enabled: false 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-05T23:25:39.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.target_file_size_base: 67108864 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-05T23:25:39.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.arena_block_size: 1048576 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.disable_auto_compactions: 0 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-05T23:25:39.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.table_properties_collectors: 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.inplace_update_support: 0 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.bloom_locality: 0 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.max_successive_merges: 0 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.paranoid_file_checks: 0 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.force_consistency_checks: 1 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.report_bg_io_stats: 0 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.ttl: 2592000 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-05T23:25:39.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.enable_blob_files: false 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.min_blob_size: 0 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.blob_file_size: 268435456 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.blob_file_starting_level: 0 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0a85aecb-ec50-4b05-8ac4-4c957afd5ae2 2024-04-05T23:25:39.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712359539310885, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-05T23:25:39.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-05T23:25:39.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712359539312710, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93609, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 311, "table_properties": {"data_size": 91600, "index_size": 291, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13802, "raw_average_key_size": 50, "raw_value_size": 84098, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 274, "num_filter_entries": 274, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712359539, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0a85aecb-ec50-4b05-8ac4-4c957afd5ae2", "db_session_id": "IRB6Q196WRYX2GNXCWC2", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-05T23:25:39.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712359539312854, "job": 1, "event": "recovery_finished"} 2024-04-05T23:25:39.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-05T23:25:39.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-05T23:25:39.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55db4781c000 2024-04-05T23:25:39.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: DB pointer 0x55db4780a000 2024-04-05T23:25:39.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-05T23:25:39.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 93.25 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T23:25:39.546 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 93.25 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T23:25:39.547 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 12.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 12.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x55db4777d1f0#7 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mon.a@-1(???) e1 preinit fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mon.a@-1(???).mds e1 new map 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mon.a@-1(???).mds e1 print_map 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T23:25:39.548 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: monmap e1: 1 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: fsmap 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: osdmap e1: 0 total, 0 up, 0 in 2024-04-05T23:25:39.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:39 smithi016 ceph-mon[38248]: mgrmap e1: no daemons active 2024-04-05T23:25:41.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:40 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1861328324' entity='client.admin' 2024-04-05T23:25:41.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.735+0000 7f1fe4c00700 1 Processor -- start 2024-04-05T23:25:41.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.735+0000 7f1fe4c00700 1 -- start start 2024-04-05T23:25:41.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.736+0000 7f1fe4c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe0079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:41.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.736+0000 7f1fe4c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00797e0 con 0x7f1fe007adb0 2024-04-05T23:25:41.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.736+0000 7f1fdde00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe0079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:41.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.736+0000 7f1fdde00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe0079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42144/0 (socket says 172.21.15.16:42144) 2024-04-05T23:25:41.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.736+0000 7f1fdde00700 1 -- 172.21.15.16:0/3669645478 learned_addr learned my addr 172.21.15.16:0/3669645478 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:41.335 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.737+0000 7f1fdde00700 1 -- 172.21.15.16:0/3669645478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe0079920 con 0x7f1fe007adb0 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.737+0000 7f1fdde00700 1 --2- 172.21.15.16:0/3669645478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe0079210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1fd00098a0 tx=0x7f1fd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e846ee2f70f80e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.737+0000 7f1fdd400700 1 -- 172.21.15.16:0/3669645478 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fd0004160 con 0x7f1fe007adb0 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.737+0000 7f1fdd400700 1 -- 172.21.15.16:0/3669645478 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fd00042c0 con 0x7f1fe007adb0 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.737+0000 7f1fdd400700 1 -- 172.21.15.16:0/3669645478 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fd0011a00 con 0x7f1fe007adb0 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.738+0000 7f1fe4c00700 1 -- 172.21.15.16:0/3669645478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 msgr2=0x7f1fe0079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.738+0000 7f1fe4c00700 1 --2- 172.21.15.16:0/3669645478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe0079210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1fd00098a0 tx=0x7f1fd0009bb0 comp rx=0 tx=0).stop 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.738+0000 7f1fe4c00700 1 -- 172.21.15.16:0/3669645478 shutdown_connections 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.738+0000 7f1fe4c00700 1 --2- 172.21.15.16:0/3669645478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe0079210 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.738+0000 7f1fe4c00700 1 -- 172.21.15.16:0/3669645478 >> 172.21.15.16:0/3669645478 conn(0x7f1fe0101c60 msgr2=0x7f1fe01040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:41.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.738+0000 7f1fe4c00700 1 -- 172.21.15.16:0/3669645478 shutdown_connections 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.738+0000 7f1fe4c00700 1 -- 172.21.15.16:0/3669645478 wait complete. 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.739+0000 7f1fe4c00700 1 Processor -- start 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.739+0000 7f1fe4c00700 1 -- start start 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.740+0000 7f1fe4c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.740+0000 7f1fe4c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe019cbc0 con 0x7f1fe007adb0 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.740+0000 7f1fdde00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.740+0000 7f1fdde00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe019c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42158/0 (socket says 172.21.15.16:42158) 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.740+0000 7f1fdde00700 1 -- 172.21.15.16:0/1861328324 learned_addr learned my addr 172.21.15.16:0/1861328324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.740+0000 7f1fdde00700 1 -- 172.21.15.16:0/1861328324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd0009550 con 0x7f1fe007adb0 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.741+0000 7f1fdde00700 1 --2- 172.21.15.16:0/1861328324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe019c680 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009130 tx=0x7f1fd0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.741+0000 7f1fd6a00700 1 -- 172.21.15.16:0/1861328324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fd0003ab0 con 0x7f1fe007adb0 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.741+0000 7f1fd6a00700 1 -- 172.21.15.16:0/1861328324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fd0003c10 con 0x7f1fe007adb0 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.741+0000 7f1fe4c00700 1 -- 172.21.15.16:0/1861328324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe019cdc0 con 0x7f1fe007adb0 2024-04-05T23:25:41.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.741+0000 7f1fd6a00700 1 -- 172.21.15.16:0/1861328324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1fd00247d0 con 0x7f1fe007adb0 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.741+0000 7f1fe4c00700 1 -- 172.21.15.16:0/1861328324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe019d260 con 0x7f1fe007adb0 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.741+0000 7f1fd6a00700 1 -- 172.21.15.16:0/1861328324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f1fd0024d50 con 0x7f1fe007adb0 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.741+0000 7f1fd6a00700 1 -- 172.21.15.16:0/1861328324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1fd0019490 con 0x7f1fe007adb0 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.742+0000 7f1fe4c00700 1 -- 172.21.15.16:0/1861328324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe010ca90 con 0x7f1fe007adb0 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.745+0000 7f1fd6a00700 1 -- 172.21.15.16:0/1861328324 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f1fd001abb0 con 0x7f1fe007adb0 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.818+0000 7f1fe4c00700 1 -- 172.21.15.16:0/1861328324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f1fe019faf0 con 0x7f1fe007adb0 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.821+0000 7f1fd6a00700 1 -- 172.21.15.16:0/1861328324 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1fd001ad50 con 0x7f1fe007adb0 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.822+0000 7f1fe4c00700 1 -- 172.21.15.16:0/1861328324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 msgr2=0x7f1fe019c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.822+0000 7f1fe4c00700 1 --2- 172.21.15.16:0/1861328324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe019c680 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009130 tx=0x7f1fd0011ad0 comp rx=0 tx=0).stop 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.822+0000 7f1fe4c00700 1 -- 172.21.15.16:0/1861328324 shutdown_connections 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.822+0000 7f1fe4c00700 1 --2- 172.21.15.16:0/1861328324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe007adb0 0x7f1fe019c680 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.822+0000 7f1fe4c00700 1 -- 172.21.15.16:0/1861328324 >> 172.21.15.16:0/1861328324 conn(0x7f1fe0101c60 msgr2=0x7f1fe01028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.822+0000 7f1fe4c00700 1 -- 172.21.15.16:0/1861328324 shutdown_connections 2024-04-05T23:25:41.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:39.822+0000 7f1fe4c00700 1 -- 172.21.15.16:0/1861328324 wait complete. 2024-04-05T23:25:41.339 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-05T23:25:41.371 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-05T23:25:41.371 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-04-05T23:25:41.372 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-05T23:25:41.373 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-05T23:25:41.760 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.x 2024-04-05T23:25:41.760 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.x.service: Unit ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.x.service not loaded. 2024-04-05T23:25:41.957 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4.target.wants/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.x.service → /etc/systemd/system/ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@.service. 2024-04-05T23:25:42.084 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:41 smithi016 systemd[1]: Starting Ceph mgr.x for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:25:42.511 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T23:25:42.511 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-05T23:25:42.527 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T23:25:42.527 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-05T23:25:42.527 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-05T23:25:42.527 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-04-05T23:25:42.544 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T23:25:42.545 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-05T23:25:42.563 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T23:25:42.563 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-05T23:25:42.563 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-05T23:25:42.563 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-05T23:25:42.563 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-04-05T23:25:42.563 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-04-05T23:25:42.743 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:42 smithi016 bash[38485]: 9e4e09a1a78c461efb4dfd383c4dc6486571d2922f2fc359e5642f8beac48cf6 2024-04-05T23:25:42.744 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:42 smithi016 systemd[1]: Started Ceph mgr.x for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:25:43.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/934394848' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T23:25:43.308 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:43.060+0000 7f9dd3b1f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-05T23:25:43.308 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:43.194+0000 7f9dd3b1f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-05T23:25:43.308 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:43.315+0000 7f9dd3b1f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-05T23:25:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:25:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-05T23:25:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:25:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T23:25:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T23:25:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T23:25:43.439 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T23:25:43.453 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:43.453 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T23:25:43.454 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T23:25:43.454 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-05T23:25:43.454 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:25:43.454 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T23:25:43.454 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-05T23:25:43.455 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:25:43.455 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-05T23:25:43.455 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T23:25:43.455 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:43.455 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-05T23:25:43.456 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T23:25:43.456 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:43.456 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T23:25:43.456 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:43.456 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T23:25:43.456 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T23:25:43.457 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T23:25:43.457 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T23:25:43.457 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T23:25:43.457 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T23:25:43.457 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:43.457 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T23:25:43.458 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T23:25:43.458 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T23:25:43.458 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T23:25:43.458 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T23:25:43.458 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T23:25:43.458 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T23:25:43.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T23:25:43.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T23:25:43.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:43.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T23:25:43.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:43.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T23:25:43.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T23:25:43.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:43.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T23:25:43.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-05T23:25:43.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T23:25:43.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T23:25:43.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T23:25:43.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T23:25:43.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T23:25:43.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:25:43.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T23:25:43.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:43.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T23:25:43.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:43.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T23:25:35.231583+0000", 2024-04-05T23:25:43.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T23:25:43.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:43.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T23:25:43.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-05T23:25:43.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.025+0000 7f5cbc600700 1 Processor -- start 2024-04-05T23:25:43.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.025+0000 7f5cbc600700 1 -- start start 2024-04-05T23:25:43.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.025+0000 7f5cbc600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb4079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:43.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.025+0000 7f5cbc600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb40797e0 con 0x7f5cb407adb0 2024-04-05T23:25:43.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.026+0000 7f5cb9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb4079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:43.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.026+0000 7f5cb9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb4079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43834/0 (socket says 172.21.15.16:43834) 2024-04-05T23:25:43.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.026+0000 7f5cb9c00700 1 -- 172.21.15.16:0/593396878 learned_addr learned my addr 172.21.15.16:0/593396878 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:43.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.026+0000 7f5cb9c00700 1 -- 172.21.15.16:0/593396878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb4079920 con 0x7f5cb407adb0 2024-04-05T23:25:43.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.027+0000 7f5cb9c00700 1 --2- 172.21.15.16:0/593396878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb4079210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80098a0 tx=0x7f5ca8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530763b5482a0ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:43.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.027+0000 7f5cb3e00700 1 -- 172.21.15.16:0/593396878 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca8004160 con 0x7f5cb407adb0 2024-04-05T23:25:43.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.027+0000 7f5cb3e00700 1 -- 172.21.15.16:0/593396878 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ca80042c0 con 0x7f5cb407adb0 2024-04-05T23:25:43.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.027+0000 7f5cb3e00700 1 -- 172.21.15.16:0/593396878 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca8011a00 con 0x7f5cb407adb0 2024-04-05T23:25:43.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.027+0000 7f5cbc600700 1 -- 172.21.15.16:0/593396878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 msgr2=0x7f5cb4079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:43.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.027+0000 7f5cbc600700 1 --2- 172.21.15.16:0/593396878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb4079210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80098a0 tx=0x7f5ca8009bb0 comp rx=0 tx=0).stop 2024-04-05T23:25:43.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.028+0000 7f5cbc600700 1 -- 172.21.15.16:0/593396878 shutdown_connections 2024-04-05T23:25:43.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.028+0000 7f5cbc600700 1 --2- 172.21.15.16:0/593396878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb4079210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:43.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.028+0000 7f5cbc600700 1 -- 172.21.15.16:0/593396878 >> 172.21.15.16:0/593396878 conn(0x7f5cb4101c60 msgr2=0x7f5cb41040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:43.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.028+0000 7f5cbc600700 1 -- 172.21.15.16:0/593396878 shutdown_connections 2024-04-05T23:25:43.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.028+0000 7f5cbc600700 1 -- 172.21.15.16:0/593396878 wait complete. 2024-04-05T23:25:43.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.029+0000 7f5cbc600700 1 Processor -- start 2024-04-05T23:25:43.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.029+0000 7f5cbc600700 1 -- start start 2024-04-05T23:25:43.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.029+0000 7f5cbc600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:43.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.029+0000 7f5cbc600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb419cbd0 con 0x7f5cb407adb0 2024-04-05T23:25:43.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.029+0000 7f5cb9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:43.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.030+0000 7f5cb9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43836/0 (socket says 172.21.15.16:43836) 2024-04-05T23:25:43.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.030+0000 7f5cb9c00700 1 -- 172.21.15.16:0/934394848 learned_addr learned my addr 172.21.15.16:0/934394848 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.030+0000 7f5cb9c00700 1 -- 172.21.15.16:0/934394848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca8009550 con 0x7f5cb407adb0 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.030+0000 7f5cb9c00700 1 --2- 172.21.15.16:0/934394848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb419c690 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009130 tx=0x7f5ca8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.030+0000 7f5cb2000700 1 -- 172.21.15.16:0/934394848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca8003ab0 con 0x7f5cb407adb0 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.030+0000 7f5cb2000700 1 -- 172.21.15.16:0/934394848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ca8003c10 con 0x7f5cb407adb0 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.030+0000 7f5cb2000700 1 -- 172.21.15.16:0/934394848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca80247d0 con 0x7f5cb407adb0 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.030+0000 7f5cbc600700 1 -- 172.21.15.16:0/934394848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb419cdd0 con 0x7f5cb407adb0 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.031+0000 7f5cbc600700 1 -- 172.21.15.16:0/934394848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb419d270 con 0x7f5cb407adb0 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.031+0000 7f5cb2000700 1 -- 172.21.15.16:0/934394848 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f5ca8024d00 con 0x7f5cb407adb0 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.031+0000 7f5cb2000700 1 -- 172.21.15.16:0/934394848 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5ca80193c0 con 0x7f5cb407adb0 2024-04-05T23:25:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.032+0000 7f5cbc600700 1 -- 172.21.15.16:0/934394848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb410ca90 con 0x7f5cb407adb0 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.034+0000 7f5cb2000700 1 -- 172.21.15.16:0/934394848 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f5ca801abb0 con 0x7f5cb407adb0 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.112+0000 7f5cbc600700 1 -- 172.21.15.16:0/934394848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5cb410cc80 con 0x7f5cb407adb0 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.113+0000 7f5cb2000700 1 -- 172.21.15.16:0/934394848 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f5ca801ad50 con 0x7f5cb407adb0 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.114+0000 7f5cbc600700 1 -- 172.21.15.16:0/934394848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 msgr2=0x7f5cb419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.114+0000 7f5cbc600700 1 --2- 172.21.15.16:0/934394848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb419c690 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009130 tx=0x7f5ca8011ad0 comp rx=0 tx=0).stop 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.114+0000 7f5cbc600700 1 -- 172.21.15.16:0/934394848 shutdown_connections 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.114+0000 7f5cbc600700 1 --2- 172.21.15.16:0/934394848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cb407adb0 0x7f5cb419c690 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.114+0000 7f5cbc600700 1 -- 172.21.15.16:0/934394848 >> 172.21.15.16:0/934394848 conn(0x7f5cb4101c60 msgr2=0x7f5cb41028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.114+0000 7f5cbc600700 1 -- 172.21.15.16:0/934394848 shutdown_connections 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:43.114+0000 7f5cbc600700 1 -- 172.21.15.16:0/934394848 wait complete. 2024-04-05T23:25:43.467 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-04-05T23:25:43.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:43.430+0000 7f9dd3b1f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-05T23:25:44.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:44 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:44.321+0000 7f9dd3b1f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:25:44.893 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:44 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:44.642+0000 7f9dd3b1f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-05T23:25:44.893 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:44 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:44.757+0000 7f9dd3b1f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-05T23:25:44.894 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:44 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:44.900+0000 7f9dd3b1f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-05T23:25:45.269 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:45 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:45.101+0000 7f9dd3b1f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-05T23:25:45.269 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:45 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:45.276+0000 7f9dd3b1f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-05T23:25:45.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:45 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:45.391+0000 7f9dd3b1f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-05T23:25:46.174 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:46 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2357638869' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T23:25:46.174 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:45 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:45.917+0000 7f9dd3b1f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-05T23:25:46.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:25:46.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-05T23:25:46.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:25:46.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T23:25:46.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T23:25:46.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T23:25:46.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T23:25:46.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:46.336 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T23:25:46.337 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T23:25:46.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T23:25:46.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T23:25:35.231583+0000", 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T23:25:46.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.888+0000 7f8cb7400700 1 Processor -- start 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.888+0000 7f8cb7400700 1 -- start start 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.889+0000 7f8cb7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb0079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.889+0000 7f8cb7400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00797e0 con 0x7f8cb007adb0 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.889+0000 7f8cb4a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb0079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.889+0000 7f8cb4a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb0079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43838/0 (socket says 172.21.15.16:43838) 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.889+0000 7f8cb4a00700 1 -- 172.21.15.16:0/3847507541 learned_addr learned my addr 172.21.15.16:0/3847507541 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.890+0000 7f8cb4a00700 1 -- 172.21.15.16:0/3847507541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb0079920 con 0x7f8cb007adb0 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.890+0000 7f8cb4a00700 1 --2- 172.21.15.16:0/3847507541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb0079210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40098a0 tx=0x7f8ca4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89677873aac62bc server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.890+0000 7f8caf400700 1 -- 172.21.15.16:0/3847507541 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ca4004160 con 0x7f8cb007adb0 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.890+0000 7f8caf400700 1 -- 172.21.15.16:0/3847507541 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ca40042c0 con 0x7f8cb007adb0 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.890+0000 7f8caf400700 1 -- 172.21.15.16:0/3847507541 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ca4011a00 con 0x7f8cb007adb0 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.891+0000 7f8cb7400700 1 -- 172.21.15.16:0/3847507541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 msgr2=0x7f8cb0079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:46.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.891+0000 7f8cb7400700 1 --2- 172.21.15.16:0/3847507541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb0079210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40098a0 tx=0x7f8ca4009bb0 comp rx=0 tx=0).stop 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.891+0000 7f8cb7400700 1 -- 172.21.15.16:0/3847507541 shutdown_connections 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.891+0000 7f8cb7400700 1 --2- 172.21.15.16:0/3847507541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb0079210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.891+0000 7f8cb7400700 1 -- 172.21.15.16:0/3847507541 >> 172.21.15.16:0/3847507541 conn(0x7f8cb0101c60 msgr2=0x7f8cb01040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.891+0000 7f8cb7400700 1 -- 172.21.15.16:0/3847507541 shutdown_connections 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.891+0000 7f8cb7400700 1 -- 172.21.15.16:0/3847507541 wait complete. 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.893+0000 7f8cb7400700 1 Processor -- start 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.893+0000 7f8cb7400700 1 -- start start 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.893+0000 7f8cb7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.894+0000 7f8cb7400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb019cbd0 con 0x7f8cb007adb0 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.894+0000 7f8cb4a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.894+0000 7f8cb4a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb019c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43852/0 (socket says 172.21.15.16:43852) 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.894+0000 7f8cb4a00700 1 -- 172.21.15.16:0/2357638869 learned_addr learned my addr 172.21.15.16:0/2357638869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.894+0000 7f8cb4a00700 1 -- 172.21.15.16:0/2357638869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca4009550 con 0x7f8cb007adb0 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.894+0000 7f8cb4a00700 1 --2- 172.21.15.16:0/2357638869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb019c690 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009130 tx=0x7f8ca4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:46.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.895+0000 7f8cad600700 1 -- 172.21.15.16:0/2357638869 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ca4003ab0 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.895+0000 7f8cad600700 1 -- 172.21.15.16:0/2357638869 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ca4003c10 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.895+0000 7f8cad600700 1 -- 172.21.15.16:0/2357638869 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ca40247d0 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.895+0000 7f8cb7400700 1 -- 172.21.15.16:0/2357638869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb019cdd0 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.895+0000 7f8cb7400700 1 -- 172.21.15.16:0/2357638869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb019d270 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.895+0000 7f8cad600700 1 -- 172.21.15.16:0/2357638869 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f8ca4024d00 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.895+0000 7f8cad600700 1 -- 172.21.15.16:0/2357638869 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8ca40193c0 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.896+0000 7f8cb7400700 1 -- 172.21.15.16:0/2357638869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb010ca90 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.899+0000 7f8cad600700 1 -- 172.21.15.16:0/2357638869 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f8ca401abb0 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.989+0000 7f8cb7400700 1 -- 172.21.15.16:0/2357638869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8cb010cc80 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.990+0000 7f8cad600700 1 -- 172.21.15.16:0/2357638869 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f8ca401ad50 con 0x7f8cb007adb0 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.991+0000 7f8cb7400700 1 -- 172.21.15.16:0/2357638869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 msgr2=0x7f8cb019c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.991+0000 7f8cb7400700 1 --2- 172.21.15.16:0/2357638869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb019c690 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009130 tx=0x7f8ca4011ad0 comp rx=0 tx=0).stop 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.991+0000 7f8cb7400700 1 -- 172.21.15.16:0/2357638869 shutdown_connections 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.991+0000 7f8cb7400700 1 --2- 172.21.15.16:0/2357638869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb007adb0 0x7f8cb019c690 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:46.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.991+0000 7f8cb7400700 1 -- 172.21.15.16:0/2357638869 >> 172.21.15.16:0/2357638869 conn(0x7f8cb0101c60 msgr2=0x7f8cb01028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:46.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.991+0000 7f8cb7400700 1 -- 172.21.15.16:0/2357638869 shutdown_connections 2024-04-05T23:25:46.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:45.991+0000 7f8cb7400700 1 -- 172.21.15.16:0/2357638869 wait complete. 2024-04-05T23:25:46.344 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-04-05T23:25:46.690 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:46 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:46.438+0000 7f9dd3b1f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-05T23:25:47.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:46 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:46.697+0000 7f9dd3b1f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-05T23:25:47.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:46 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:46.824+0000 7f9dd3b1f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-05T23:25:48.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:47 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:47.820+0000 7f9dd3b1f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-05T23:25:48.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:47 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:47.951+0000 7f9dd3b1f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-05T23:25:48.485 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:48 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:48.151+0000 7f9dd3b1f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-05T23:25:49.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:25:49.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-05T23:25:49.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:25:49.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T23:25:49.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T23:25:49.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T23:25:49.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T23:25:49.174 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:48 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2432155188' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T23:25:49.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:49.175 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:49 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:49.036+0000 7f9dd3b1f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-05T23:25:49.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T23:25:49.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T23:25:49.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-05T23:25:49.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:25:49.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T23:25:49.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T23:25:49.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T23:25:49.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T23:25:49.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T23:25:49.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T23:25:49.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:49.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T23:25:49.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:49.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T23:25:35.231583+0000", 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:25:49.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.747+0000 7f1de6400700 1 Processor -- start 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.747+0000 7f1de6400700 1 -- start start 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.748+0000 7f1de6400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de0079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.748+0000 7f1de6400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00797e0 con 0x7f1de007adb0 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.748+0000 7f1ddf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de0079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.748+0000 7f1ddf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de0079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43854/0 (socket says 172.21.15.16:43854) 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.748+0000 7f1ddf400700 1 -- 172.21.15.16:0/4152782660 learned_addr learned my addr 172.21.15.16:0/4152782660 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.749+0000 7f1ddf400700 1 -- 172.21.15.16:0/4152782660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de0079920 con 0x7f1de007adb0 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.749+0000 7f1ddf400700 1 --2- 172.21.15.16:0/4152782660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de0079210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00098a0 tx=0x7f1dd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a977af5b3e80d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.749+0000 7f1dde000700 1 -- 172.21.15.16:0/4152782660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dd0004160 con 0x7f1de007adb0 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.749+0000 7f1dde000700 1 -- 172.21.15.16:0/4152782660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1dd00042c0 con 0x7f1de007adb0 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.749+0000 7f1dde000700 1 -- 172.21.15.16:0/4152782660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dd0011a00 con 0x7f1de007adb0 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.750+0000 7f1de6400700 1 -- 172.21.15.16:0/4152782660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 msgr2=0x7f1de0079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:49.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.750+0000 7f1de6400700 1 --2- 172.21.15.16:0/4152782660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de0079210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00098a0 tx=0x7f1dd0009bb0 comp rx=0 tx=0).stop 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.750+0000 7f1de6400700 1 -- 172.21.15.16:0/4152782660 shutdown_connections 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.750+0000 7f1de6400700 1 --2- 172.21.15.16:0/4152782660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de0079210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.750+0000 7f1de6400700 1 -- 172.21.15.16:0/4152782660 >> 172.21.15.16:0/4152782660 conn(0x7f1de0101c60 msgr2=0x7f1de01040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.750+0000 7f1de6400700 1 -- 172.21.15.16:0/4152782660 shutdown_connections 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.750+0000 7f1de6400700 1 -- 172.21.15.16:0/4152782660 wait complete. 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.751+0000 7f1de6400700 1 Processor -- start 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.751+0000 7f1de6400700 1 -- start start 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.751+0000 7f1de6400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.751+0000 7f1de6400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de019cbd0 con 0x7f1de007adb0 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.752+0000 7f1ddf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.752+0000 7f1ddf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de019c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43864/0 (socket says 172.21.15.16:43864) 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.752+0000 7f1ddf400700 1 -- 172.21.15.16:0/2432155188 learned_addr learned my addr 172.21.15.16:0/2432155188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.752+0000 7f1ddf400700 1 -- 172.21.15.16:0/2432155188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd0009550 con 0x7f1de007adb0 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.752+0000 7f1ddf400700 1 --2- 172.21.15.16:0/2432155188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de019c690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009130 tx=0x7f1dd0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:25:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.753+0000 7f1dd7e00700 1 -- 172.21.15.16:0/2432155188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dd0003ab0 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.753+0000 7f1dd7e00700 1 -- 172.21.15.16:0/2432155188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1dd0003c10 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.753+0000 7f1de6400700 1 -- 172.21.15.16:0/2432155188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de019cdd0 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.753+0000 7f1dd7e00700 1 -- 172.21.15.16:0/2432155188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dd00247d0 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.753+0000 7f1de6400700 1 -- 172.21.15.16:0/2432155188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de019d270 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.753+0000 7f1dd7e00700 1 -- 172.21.15.16:0/2432155188 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f1dd0024d00 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.753+0000 7f1dd7e00700 1 -- 172.21.15.16:0/2432155188 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1dd00193c0 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.754+0000 7f1de6400700 1 -- 172.21.15.16:0/2432155188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de010ca90 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.757+0000 7f1dd7e00700 1 -- 172.21.15.16:0/2432155188 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f1dd001abb0 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.857+0000 7f1de6400700 1 -- 172.21.15.16:0/2432155188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1de010cc80 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.858+0000 7f1dd7e00700 1 -- 172.21.15.16:0/2432155188 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f1dd001ad50 con 0x7f1de007adb0 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.859+0000 7f1de6400700 1 -- 172.21.15.16:0/2432155188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 msgr2=0x7f1de019c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.859+0000 7f1de6400700 1 --2- 172.21.15.16:0/2432155188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de019c690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009130 tx=0x7f1dd0011ad0 comp rx=0 tx=0).stop 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.859+0000 7f1de6400700 1 -- 172.21.15.16:0/2432155188 shutdown_connections 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.859+0000 7f1de6400700 1 --2- 172.21.15.16:0/2432155188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de007adb0 0x7f1de019c690 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:25:49.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.859+0000 7f1de6400700 1 -- 172.21.15.16:0/2432155188 >> 172.21.15.16:0/2432155188 conn(0x7f1de0101c60 msgr2=0x7f1de01028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:25:49.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.859+0000 7f1de6400700 1 -- 172.21.15.16:0/2432155188 shutdown_connections 2024-04-05T23:25:49.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:48.859+0000 7f1de6400700 1 -- 172.21.15.16:0/2432155188 wait complete. 2024-04-05T23:25:49.191 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (3/15)... 2024-04-05T23:25:49.546 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:49 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:49.287+0000 7f9dd3b1f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-05T23:25:49.833 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:49 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:49.553+0000 7f9dd3b1f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-05T23:25:51.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:50.739+0000 7f9dd3b1f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-05T23:25:51.538 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:51.125+0000 7f9dd3b1f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:25:51.538 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:51.263+0000 7f9dd3b1f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-05T23:25:51.833 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:25:51.545+0000 7f9dd3b1f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: Activating manager daemon x 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: mgrmap e2: x(active, starting, since 0.00498634s) 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: Manager daemon x is now available 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' 2024-04-05T23:25:51.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' 2024-04-05T23:25:51.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:51 smithi016 ceph-mon[38248]: from='mgr.14100 172.21.15.16:0/1968578582' entity='mgr.x' 2024-04-05T23:25:53.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:53 smithi016 ceph-mon[38248]: mgrmap e3: x(active, since 1.0101s) 2024-04-05T23:25:54.822 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:54 smithi016 ceph-mon[38248]: mgrmap e4: x(active, since 2s) 2024-04-05T23:25:55.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:25:55 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2219365011' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T23:26:00.879 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:26:00.879 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-05T23:26:00.879 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:26:00.879 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T23:26:00.880 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T23:26:00.880 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T23:26:00.880 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T23:26:00.880 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:26:00.880 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T23:26:00.880 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T23:26:00.880 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-05T23:26:00.880 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T23:26:00.882 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T23:26:00.883 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T23:26:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T23:26:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T23:26:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T23:26:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T23:26:00.893 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T23:25:35.231583+0000", 2024-04-05T23:26:00.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.495+0000 7f26cf600700 1 Processor -- start 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.495+0000 7f26cf600700 1 -- start start 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.496+0000 7f26cf600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c8079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.496+0000 7f26cf600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80797e0 con 0x7f26c807adb0 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.496+0000 7f26ccc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c8079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.496+0000 7f26ccc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c8079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36492/0 (socket says 172.21.15.16:36492) 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.496+0000 7f26ccc00700 1 -- 172.21.15.16:0/3435269353 learned_addr learned my addr 172.21.15.16:0/3435269353 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.497+0000 7f26ccc00700 1 -- 172.21.15.16:0/3435269353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c8079920 con 0x7f26c807adb0 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.497+0000 7f26ccc00700 1 --2- 172.21.15.16:0/3435269353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c8079210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0098a0 tx=0x7f26bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fefe96ddfcf22fc server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.497+0000 7f26c7400700 1 -- 172.21.15.16:0/3435269353 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26bc004160 con 0x7f26c807adb0 2024-04-05T23:26:00.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.497+0000 7f26c7400700 1 -- 172.21.15.16:0/3435269353 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26bc0042c0 con 0x7f26c807adb0 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.497+0000 7f26c7400700 1 -- 172.21.15.16:0/3435269353 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26bc011a00 con 0x7f26c807adb0 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.498+0000 7f26cf600700 1 -- 172.21.15.16:0/3435269353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 msgr2=0x7f26c8079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.498+0000 7f26cf600700 1 --2- 172.21.15.16:0/3435269353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c8079210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0098a0 tx=0x7f26bc009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.498+0000 7f26cf600700 1 -- 172.21.15.16:0/3435269353 shutdown_connections 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.498+0000 7f26cf600700 1 --2- 172.21.15.16:0/3435269353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c8079210 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.498+0000 7f26cf600700 1 -- 172.21.15.16:0/3435269353 >> 172.21.15.16:0/3435269353 conn(0x7f26c8101c60 msgr2=0x7f26c81040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.498+0000 7f26cf600700 1 -- 172.21.15.16:0/3435269353 shutdown_connections 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.498+0000 7f26cf600700 1 -- 172.21.15.16:0/3435269353 wait complete. 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.499+0000 7f26cf600700 1 Processor -- start 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.499+0000 7f26cf600700 1 -- start start 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.499+0000 7f26cf600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.499+0000 7f26cf600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c819cbd0 con 0x7f26c807adb0 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.500+0000 7f26ccc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.500+0000 7f26ccc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36508/0 (socket says 172.21.15.16:36508) 2024-04-05T23:26:00.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.500+0000 7f26ccc00700 1 -- 172.21.15.16:0/2219365011 learned_addr learned my addr 172.21.15.16:0/2219365011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.500+0000 7f26ccc00700 1 -- 172.21.15.16:0/2219365011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc009550 con 0x7f26c807adb0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.500+0000 7f26ccc00700 1 --2- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c819c690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f26bc009130 tx=0x7f26bc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.500+0000 7f26c5600700 1 -- 172.21.15.16:0/2219365011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26bc003ab0 con 0x7f26c807adb0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.500+0000 7f26c5600700 1 -- 172.21.15.16:0/2219365011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26bc003c10 con 0x7f26c807adb0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.500+0000 7f26c5600700 1 -- 172.21.15.16:0/2219365011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26bc0247d0 con 0x7f26c807adb0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.501+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c819cdd0 con 0x7f26c807adb0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.501+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c819d270 con 0x7f26c807adb0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.502+0000 7f26c5600700 1 -- 172.21.15.16:0/2219365011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 45530+0+0 (secure 0 0 0) 0x7f26bc01abb0 con 0x7f26c807adb0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.502+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c810ca90 con 0x7f26c807adb0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.502+0000 7f26c5600700 1 --2- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f26b0038700 0x7f26b003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.502+0000 7f26c5600700 1 -- 172.21.15.16:0/2219365011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f26bc04c3e0 con 0x7f26c807adb0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.503+0000 7f26c7e00700 1 --2- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f26b0038700 0x7f26b003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.503+0000 7f26c7e00700 1 --2- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f26b0038700 0x7f26b003abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f26b4006fd0 tx=0x7f26b4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:00.897 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.508+0000 7f26c5600700 1 -- 172.21.15.16:0/2219365011 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f26bc018710 con 0x7f26c807adb0 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.806+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f26c810cd00 con 0x7f26c807adb0 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.807+0000 7f26c5600700 1 -- 172.21.15.16:0/2219365011 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f26bc018350 con 0x7f26c807adb0 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f26b0038700 msgr2=0x7f26b003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 --2- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f26b0038700 0x7f26b003abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f26b4006fd0 tx=0x7f26b4006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 msgr2=0x7f26c819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 --2- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c819c690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f26bc009130 tx=0x7f26bc011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 shutdown_connections 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 --2- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f26b0038700 0x7f26b003abc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 --2- 172.21.15.16:0/2219365011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c807adb0 0x7f26c819c690 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 >> 172.21.15.16:0/2219365011 conn(0x7f26c8101c60 msgr2=0x7f26c81028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 shutdown_connections 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:25:54.809+0000 7f26cf600700 1 -- 172.21.15.16:0/2219365011 wait complete. 2024-04-05T23:26:00.898 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-04-05T23:26:01.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:01 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3729833687' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-05T23:26:01.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.292+0000 7f23db800700 1 Processor -- start 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.292+0000 7f23db800700 1 -- start start 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.293+0000 7f23db800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d4106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.293+0000 7f23db800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d41070b0 con 0x7f23d41066c0 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.293+0000 7f23d8e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d4106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.293+0000 7f23d8e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d4106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36514/0 (socket says 172.21.15.16:36514) 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.293+0000 7f23d8e00700 1 -- 172.21.15.16:0/2057957449 learned_addr learned my addr 172.21.15.16:0/2057957449 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.294+0000 7f23d8e00700 1 -- 172.21.15.16:0/2057957449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d41078c0 con 0x7f23d41066c0 2024-04-05T23:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.294+0000 7f23d8e00700 1 --2- 172.21.15.16:0/2057957449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d4106ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f23c80098a0 tx=0x7f23c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5faab45c01cca08 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.294+0000 7f23d3400700 1 -- 172.21.15.16:0/2057957449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23c8004160 con 0x7f23d41066c0 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.294+0000 7f23d3400700 1 -- 172.21.15.16:0/2057957449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23c80042c0 con 0x7f23d41066c0 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.294+0000 7f23d3400700 1 -- 172.21.15.16:0/2057957449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23c8011a00 con 0x7f23d41066c0 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.295+0000 7f23db800700 1 -- 172.21.15.16:0/2057957449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 msgr2=0x7f23d4106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.295+0000 7f23db800700 1 --2- 172.21.15.16:0/2057957449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d4106ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f23c80098a0 tx=0x7f23c8009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.295+0000 7f23db800700 1 -- 172.21.15.16:0/2057957449 shutdown_connections 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.295+0000 7f23db800700 1 --2- 172.21.15.16:0/2057957449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d4106ae0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.295+0000 7f23db800700 1 -- 172.21.15.16:0/2057957449 >> 172.21.15.16:0/2057957449 conn(0x7f23d4101c60 msgr2=0x7f23d41040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.295+0000 7f23db800700 1 -- 172.21.15.16:0/2057957449 shutdown_connections 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.295+0000 7f23db800700 1 -- 172.21.15.16:0/2057957449 wait complete. 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.296+0000 7f23db800700 1 Processor -- start 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.296+0000 7f23db800700 1 -- start start 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.296+0000 7f23db800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.296+0000 7f23db800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d419cbf0 con 0x7f23d41066c0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.296+0000 7f23d8e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.296+0000 7f23d8e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36524/0 (socket says 172.21.15.16:36524) 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.297+0000 7f23d8e00700 1 -- 172.21.15.16:0/3729833687 learned_addr learned my addr 172.21.15.16:0/3729833687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.297+0000 7f23d8e00700 1 -- 172.21.15.16:0/3729833687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8009550 con 0x7f23d41066c0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.297+0000 7f23d8e00700 1 --2- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d419c6b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009fb0 tx=0x7f23c8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.297+0000 7f23d1600700 1 -- 172.21.15.16:0/3729833687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23c8003a40 con 0x7f23d41066c0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.297+0000 7f23d1600700 1 -- 172.21.15.16:0/3729833687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23c8003ba0 con 0x7f23d41066c0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.297+0000 7f23d1600700 1 -- 172.21.15.16:0/3729833687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23c801a750 con 0x7f23d41066c0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.297+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d419cdf0 con 0x7f23d41066c0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.298+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d419d290 con 0x7f23d41066c0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.299+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d410aaf0 con 0x7f23d41066c0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.299+0000 7f23d1600700 1 -- 172.21.15.16:0/3729833687 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 45530+0+0 (secure 0 0 0) 0x7f23c8024b60 con 0x7f23d41066c0 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.299+0000 7f23d1600700 1 --2- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f23bc0386b0 0x7f23bc03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:01.910 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.299+0000 7f23d1600700 1 -- 172.21.15.16:0/3729833687 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f23c804c310 con 0x7f23d41066c0 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.300+0000 7f23d3e00700 1 --2- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f23bc0386b0 0x7f23bc03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.300+0000 7f23d3e00700 1 --2- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f23bc0386b0 0x7f23bc03ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f23c0006fd0 tx=0x7f23c0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.304+0000 7f23d1600700 1 -- 172.21.15.16:0/3729833687 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f23c8024430 con 0x7f23d41066c0 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.515+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f23d419fb20 con 0x7f23d41066c0 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.516+0000 7f23d1600700 1 -- 172.21.15.16:0/3729833687 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f23c80187c0 con 0x7f23d41066c0 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.518+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f23bc0386b0 msgr2=0x7f23bc03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.518+0000 7f23db800700 1 --2- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f23bc0386b0 0x7f23bc03ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f23c0006fd0 tx=0x7f23c0006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.518+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 msgr2=0x7f23d419c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.518+0000 7f23db800700 1 --2- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d419c6b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009fb0 tx=0x7f23c8011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.518+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 shutdown_connections 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.518+0000 7f23db800700 1 --2- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f23bc0386b0 0x7f23bc03ab70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.518+0000 7f23db800700 1 --2- 172.21.15.16:0/3729833687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23d41066c0 0x7f23d419c6b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.518+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 >> 172.21.15.16:0/3729833687 conn(0x7f23d4101c60 msgr2=0x7f23d41028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.518+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 shutdown_connections 2024-04-05T23:26:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:01.519+0000 7f23db800700 1 -- 172.21.15.16:0/3729833687 wait complete. 2024-04-05T23:26:01.912 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-04-05T23:26:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:02 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1092098134' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-05T23:26:02.914 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:02 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: ignoring --setuser ceph since I am not root 2024-04-05T23:26:02.914 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:02 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: ignoring --setgroup ceph since I am not root 2024-04-05T23:26:03.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.326+0000 7f0014800700 1 Processor -- start 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.326+0000 7f0014800700 1 -- start start 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.327+0000 7f0014800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.327+0000 7f0014800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c1070b0 con 0x7f000c1066c0 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.327+0000 7f0011e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.327+0000 7f0011e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56092/0 (socket says 172.21.15.16:56092) 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.327+0000 7f0011e00700 1 -- 172.21.15.16:0/1014682813 learned_addr learned my addr 172.21.15.16:0/1014682813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.328+0000 7f0011e00700 1 -- 172.21.15.16:0/1014682813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c1078c0 con 0x7f000c1066c0 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.328+0000 7f0011e00700 1 --2- 172.21.15.16:0/1014682813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c106ae0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f00000098a0 tx=0x7f0000009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ba9ad22d2b36dd server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.328+0000 7f0010a00700 1 -- 172.21.15.16:0/1014682813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0000004160 con 0x7f000c1066c0 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.328+0000 7f0010a00700 1 -- 172.21.15.16:0/1014682813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00000042c0 con 0x7f000c1066c0 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.328+0000 7f0010a00700 1 -- 172.21.15.16:0/1014682813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0000011a00 con 0x7f000c1066c0 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.329+0000 7f0014800700 1 -- 172.21.15.16:0/1014682813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 msgr2=0x7f000c106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:03.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.329+0000 7f0014800700 1 --2- 172.21.15.16:0/1014682813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c106ae0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f00000098a0 tx=0x7f0000009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.329+0000 7f0014800700 1 -- 172.21.15.16:0/1014682813 shutdown_connections 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.329+0000 7f0014800700 1 --2- 172.21.15.16:0/1014682813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c106ae0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.329+0000 7f0014800700 1 -- 172.21.15.16:0/1014682813 >> 172.21.15.16:0/1014682813 conn(0x7f000c101c60 msgr2=0x7f000c1040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.329+0000 7f0014800700 1 -- 172.21.15.16:0/1014682813 shutdown_connections 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.329+0000 7f0014800700 1 -- 172.21.15.16:0/1014682813 wait complete. 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.330+0000 7f0014800700 1 Processor -- start 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.330+0000 7f0014800700 1 -- start start 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.330+0000 7f0014800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c07aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.330+0000 7f0014800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c07af70 con 0x7f000c1066c0 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.330+0000 7f0011e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c07aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.330+0000 7f0011e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c07aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56102/0 (socket says 172.21.15.16:56102) 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.330+0000 7f0011e00700 1 -- 172.21.15.16:0/1092098134 learned_addr learned my addr 172.21.15.16:0/1092098134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.331+0000 7f0011e00700 1 -- 172.21.15.16:0/1092098134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0000009550 con 0x7f000c1066c0 2024-04-05T23:26:03.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.331+0000 7f0011e00700 1 --2- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c07aa30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0000009130 tx=0x7f0000011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.331+0000 7f0006a00700 1 -- 172.21.15.16:0/1092098134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0000003ab0 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.331+0000 7f0006a00700 1 -- 172.21.15.16:0/1092098134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0000003c10 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.331+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000c0791b0 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.331+0000 7f0006a00700 1 -- 172.21.15.16:0/1092098134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00000247d0 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.331+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c079650 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.332+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c062510 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.332+0000 7f0006a00700 1 -- 172.21.15.16:0/1092098134 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 45530+0+0 (secure 0 0 0) 0x7f000001abb0 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.333+0000 7f0006a00700 1 --2- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7efff40386b0 0x7efff403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.333+0000 7f0006a00700 1 -- 172.21.15.16:0/1092098134 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f000004c470 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.333+0000 7f0011400700 1 --2- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7efff40386b0 0x7efff403ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.334+0000 7f0011400700 1 --2- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7efff40386b0 0x7efff403ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efffc006fd0 tx=0x7efffc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.338+0000 7f0006a00700 1 -- 172.21.15.16:0/1092098134 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0000018ba0 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.607+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f000c10aaf0 con 0x7f000c1066c0 2024-04-05T23:26:03.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.660+0000 7f0006a00700 1 -- 172.21.15.16:0/1092098134 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 5) v1 ==== 45541+0+0 (secure 0 0 0) 0x7f000002d770 con 0x7f000c1066c0 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.662+0000 7f0006a00700 1 -- 172.21.15.16:0/1092098134 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f000004c1d0 con 0x7f000c1066c0 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7efff40386b0 msgr2=0x7efff403ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 --2- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7efff40386b0 0x7efff403ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efffc006fd0 tx=0x7efffc006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 msgr2=0x7f000c07aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 --2- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c07aa30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0000009130 tx=0x7f0000011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 shutdown_connections 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 --2- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7efff40386b0 0x7efff403ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 --2- 172.21.15.16:0/1092098134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000c1066c0 0x7f000c07aa30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 >> 172.21.15.16:0/1092098134 conn(0x7f000c101c60 msgr2=0x7f000c102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 shutdown_connections 2024-04-05T23:26:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:02.665+0000 7f0014800700 1 -- 172.21.15.16:0/1092098134 wait complete. 2024-04-05T23:26:03.532 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:03.289+0000 7f7180a7d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-05T23:26:03.532 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:03.416+0000 7f7180a7d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-05T23:26:03.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:03 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1092098134' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-05T23:26:03.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:03 smithi016 ceph-mon[38248]: mgrmap e5: x(active, since 11s) 2024-04-05T23:26:03.833 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:03.538+0000 7f7180a7d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-05T23:26:03.834 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:03.652+0000 7f7180a7d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-05T23:26:04.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-05T23:26:04.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-05T23:26:04.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-05T23:26:04.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-05T23:26:04.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-05T23:26:04.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-05T23:26:04.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.468+0000 7f44a7e00700 1 Processor -- start 2024-04-05T23:26:04.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.468+0000 7f44a7e00700 1 -- start start 2024-04-05T23:26:04.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.469+0000 7f44a7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a0106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:04.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.469+0000 7f44a7e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a01070b0 con 0x7f44a01066c0 2024-04-05T23:26:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.469+0000 7f44a5400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a0106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.469+0000 7f44a5400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a0106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56128/0 (socket says 172.21.15.16:56128) 2024-04-05T23:26:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.469+0000 7f44a5400700 1 -- 172.21.15.16:0/1013817053 learned_addr learned my addr 172.21.15.16:0/1013817053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.469+0000 7f44a5400700 1 -- 172.21.15.16:0/1013817053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a01078c0 con 0x7f44a01066c0 2024-04-05T23:26:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.470+0000 7f44a5400700 1 --2- 172.21.15.16:0/1013817053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a0106ae0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f44940098a0 tx=0x7f4494009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4e2a41243a88c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.470+0000 7f449fe00700 1 -- 172.21.15.16:0/1013817053 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4494004160 con 0x7f44a01066c0 2024-04-05T23:26:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.470+0000 7f449fe00700 1 -- 172.21.15.16:0/1013817053 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44940042c0 con 0x7f44a01066c0 2024-04-05T23:26:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.470+0000 7f449fe00700 1 -- 172.21.15.16:0/1013817053 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4494011a00 con 0x7f44a01066c0 2024-04-05T23:26:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.471+0000 7f44a7e00700 1 -- 172.21.15.16:0/1013817053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 msgr2=0x7f44a0106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.471+0000 7f44a7e00700 1 --2- 172.21.15.16:0/1013817053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a0106ae0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f44940098a0 tx=0x7f4494009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.471+0000 7f44a7e00700 1 -- 172.21.15.16:0/1013817053 shutdown_connections 2024-04-05T23:26:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.471+0000 7f44a7e00700 1 --2- 172.21.15.16:0/1013817053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a0106ae0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.471+0000 7f44a7e00700 1 -- 172.21.15.16:0/1013817053 >> 172.21.15.16:0/1013817053 conn(0x7f44a0101c60 msgr2=0x7f44a01040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.471+0000 7f44a7e00700 1 -- 172.21.15.16:0/1013817053 shutdown_connections 2024-04-05T23:26:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.471+0000 7f44a7e00700 1 -- 172.21.15.16:0/1013817053 wait complete. 2024-04-05T23:26:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.473+0000 7f44a7e00700 1 Processor -- start 2024-04-05T23:26:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.473+0000 7f44a7e00700 1 -- start start 2024-04-05T23:26:04.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.473+0000 7f44a7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a007aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.473+0000 7f44a7e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a007af70 con 0x7f44a01066c0 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.474+0000 7f44a5400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a007aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.474+0000 7f44a5400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a007aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56140/0 (socket says 172.21.15.16:56140) 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.474+0000 7f44a5400700 1 -- 172.21.15.16:0/3055506006 learned_addr learned my addr 172.21.15.16:0/3055506006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.474+0000 7f44a5400700 1 -- 172.21.15.16:0/3055506006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4494009550 con 0x7f44a01066c0 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.474+0000 7f44a5400700 1 --2- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a007aa30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4494009130 tx=0x7f4494011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.474+0000 7f449e000700 1 -- 172.21.15.16:0/3055506006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4494003ab0 con 0x7f44a01066c0 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.474+0000 7f449e000700 1 -- 172.21.15.16:0/3055506006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4494003c10 con 0x7f44a01066c0 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.474+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a00791b0 con 0x7f44a01066c0 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.475+0000 7f449e000700 1 -- 172.21.15.16:0/3055506006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44940247d0 con 0x7f44a01066c0 2024-04-05T23:26:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.475+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a0079650 con 0x7f44a01066c0 2024-04-05T23:26:04.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.476+0000 7f449e000700 1 -- 172.21.15.16:0/3055506006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 45541+0+0 (secure 0 0 0) 0x7f449401a430 con 0x7f44a01066c0 2024-04-05T23:26:04.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.476+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a0067f50 con 0x7f44a01066c0 2024-04-05T23:26:04.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.476+0000 7f449e000700 1 --2- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f4488038750 0x7f448803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:04.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.476+0000 7f449e000700 1 -- 172.21.15.16:0/3055506006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f449404c4a0 con 0x7f44a01066c0 2024-04-05T23:26:04.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.477+0000 7f44a4a00700 1 -- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f4488038750 msgr2=0x7f448803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3856233973 2024-04-05T23:26:04.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.477+0000 7f44a4a00700 1 --2- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f4488038750 0x7f448803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T23:26:04.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.481+0000 7f449e000700 1 -- 172.21.15.16:0/3055506006 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4494018b40 con 0x7f44a01066c0 2024-04-05T23:26:04.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.678+0000 7f44a4a00700 1 -- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f4488038750 msgr2=0x7f448803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3856233973 2024-04-05T23:26:04.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.678+0000 7f44a4a00700 1 --2- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f4488038750 0x7f448803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T23:26:04.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.752+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f44a010aaf0 con 0x7f44a01066c0 2024-04-05T23:26:04.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.752+0000 7f449e000700 1 -- 172.21.15.16:0/3055506006 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f4494018b40 con 0x7f44a01066c0 2024-04-05T23:26:04.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.754+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f4488038750 msgr2=0x7f448803ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:26:04.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.754+0000 7f44a7e00700 1 --2- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f4488038750 0x7f448803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:04.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.754+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 msgr2=0x7f44a007aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:04.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.754+0000 7f44a7e00700 1 --2- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a007aa30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4494009130 tx=0x7f4494011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.755+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 shutdown_connections 2024-04-05T23:26:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.755+0000 7f44a7e00700 1 --2- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7f4488038750 0x7f448803ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.755+0000 7f44a7e00700 1 --2- 172.21.15.16:0/3055506006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44a01066c0 0x7f44a007aa30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.755+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 >> 172.21.15.16:0/3055506006 conn(0x7f44a0101c60 msgr2=0x7f44a01028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.755+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 shutdown_connections 2024-04-05T23:26:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:03.755+0000 7f44a7e00700 1 -- 172.21.15.16:0/3055506006 wait complete. 2024-04-05T23:26:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-04-05T23:26:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-04-05T23:26:04.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:04 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3055506006' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-05T23:26:04.833 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:04 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:04.530+0000 7f7180a7d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:26:05.114 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:04 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:04.857+0000 7f7180a7d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-05T23:26:05.115 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:04 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:04.973+0000 7f7180a7d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-05T23:26:05.115 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:05 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:05.120+0000 7f7180a7d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-05T23:26:05.490 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:05 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:05.323+0000 7f7180a7d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-05T23:26:05.490 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:05 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:05.496+0000 7f7180a7d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-05T23:26:05.833 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:05 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:05.612+0000 7f7180a7d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-05T23:26:06.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:06 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:06.094+0000 7f7180a7d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-05T23:26:06.849 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:06 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:06.596+0000 7f7180a7d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-05T23:26:07.333 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:06 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:06.855+0000 7f7180a7d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-05T23:26:07.333 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:06 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:06.982+0000 7f7180a7d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-05T23:26:08.294 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:07 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:07.968+0000 7f7180a7d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-05T23:26:08.294 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:08 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:08.098+0000 7f7180a7d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-05T23:26:08.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:08 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:08.299+0000 7f7180a7d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-05T23:26:09.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:09 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:09.139+0000 7f7180a7d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-05T23:26:09.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:09 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:09.386+0000 7f7180a7d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-05T23:26:10.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:09 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:09.649+0000 7f7180a7d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-05T23:26:11.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:10 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:10.829+0000 7f7180a7d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-05T23:26:11.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:11.214+0000 7f7180a7d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:26:11.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:11.336+0000 7f7180a7d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-05T23:26:12.086 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:11.597+0000 7f7180a7d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-05T23:26:12.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: Active manager daemon x restarted 2024-04-05T23:26:12.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: Activating manager daemon x 2024-04-05T23:26:12.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: osdmap e2: 0 total, 0 up, 0 in 2024-04-05T23:26:12.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: mgrmap e6: x(active, starting, since 0.00413051s) 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: Manager daemon x is now available 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:26:12.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:11 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.638+0000 7fa48ae00700 1 Processor -- start 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.639+0000 7fa48ae00700 1 -- start start 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.639+0000 7fa48ae00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa484106af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.639+0000 7fa48ae00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4841070c0 con 0x7fa4841066d0 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.639+0000 7fa483e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa484106af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:13.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.639+0000 7fa483e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa484106af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56142/0 (socket says 172.21.15.16:56142) 2024-04-05T23:26:13.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.639+0000 7fa483e00700 1 -- 172.21.15.16:0/2221210096 learned_addr learned my addr 172.21.15.16:0/2221210096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:13.047 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:12 smithi016 ceph-mon[38248]: Found migration_current of "None". Setting to last migration. 2024-04-05T23:26:13.047 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:12 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-05T23:26:13.047 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:12 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-05T23:26:13.047 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:12 smithi016 ceph-mon[38248]: mgrmap e7: x(active, since 1.00686s) 2024-04-05T23:26:13.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.640+0000 7fa483e00700 1 -- 172.21.15.16:0/2221210096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4841078d0 con 0x7fa4841066d0 2024-04-05T23:26:13.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.640+0000 7fa483e00700 1 --2- 172.21.15.16:0/2221210096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa484106af0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa4740098a0 tx=0x7fa474009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a28e85b1ed3082 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:13.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.640+0000 7fa482a00700 1 -- 172.21.15.16:0/2221210096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa474004160 con 0x7fa4841066d0 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.640+0000 7fa482a00700 1 -- 172.21.15.16:0/2221210096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4740042c0 con 0x7fa4841066d0 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.640+0000 7fa482a00700 1 -- 172.21.15.16:0/2221210096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa474011a00 con 0x7fa4841066d0 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.641+0000 7fa48ae00700 1 -- 172.21.15.16:0/2221210096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 msgr2=0x7fa484106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.641+0000 7fa48ae00700 1 --2- 172.21.15.16:0/2221210096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa484106af0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa4740098a0 tx=0x7fa474009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.641+0000 7fa48ae00700 1 -- 172.21.15.16:0/2221210096 shutdown_connections 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.641+0000 7fa48ae00700 1 --2- 172.21.15.16:0/2221210096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa484106af0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.641+0000 7fa48ae00700 1 -- 172.21.15.16:0/2221210096 >> 172.21.15.16:0/2221210096 conn(0x7fa484101c90 msgr2=0x7fa4841040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.641+0000 7fa48ae00700 1 -- 172.21.15.16:0/2221210096 shutdown_connections 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.641+0000 7fa48ae00700 1 -- 172.21.15.16:0/2221210096 wait complete. 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.642+0000 7fa48ae00700 1 Processor -- start 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.643+0000 7fa48ae00700 1 -- start start 2024-04-05T23:26:13.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.643+0000 7fa48ae00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa48419a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.643+0000 7fa48ae00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48419a9b0 con 0x7fa4841066d0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.643+0000 7fa483e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa48419a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.643+0000 7fa483e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa48419a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56146/0 (socket says 172.21.15.16:56146) 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.643+0000 7fa483e00700 1 -- 172.21.15.16:0/1779337358 learned_addr learned my addr 172.21.15.16:0/1779337358 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.643+0000 7fa483e00700 1 -- 172.21.15.16:0/1779337358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474009550 con 0x7fa4841066d0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.644+0000 7fa483e00700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa48419a470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa474009130 tx=0x7fa474011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.644+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa474003ab0 con 0x7fa4841066d0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.644+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa474003c10 con 0x7fa4841066d0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.644+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4740247d0 con 0x7fa4841066d0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.644+0000 7fa48ae00700 1 -- 172.21.15.16:0/1779337358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48419abb0 con 0x7fa4841066d0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.644+0000 7fa48ae00700 1 -- 172.21.15.16:0/1779337358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48419b050 con 0x7fa4841066d0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.645+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 45541+0+0 (secure 0 0 0) 0x7fa47401a430 con 0x7fa4841066d0 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.646+0000 7fa480c00700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 0x7fa46803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.646+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 --> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa46803b320 con 0x7fa468038750 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.646+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa47404c450 con 0x7fa4841066d0 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.646+0000 7fa483400700 1 -- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 msgr2=0x7fa46803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3856233973 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.646+0000 7fa483400700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 0x7fa46803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.847+0000 7fa483400700 1 -- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 msgr2=0x7fa46803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3856233973 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:04.847+0000 7fa483400700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 0x7fa46803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:05.248+0000 7fa483400700 1 -- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 msgr2=0x7fa46803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3856233973 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:05.248+0000 7fa483400700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 0x7fa46803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:06.049+0000 7fa483400700 1 -- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 msgr2=0x7fa46803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3856233973 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:06.049+0000 7fa483400700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 0x7fa46803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:07.651+0000 7fa483400700 1 -- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 msgr2=0x7fa46803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3856233973 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:07.651+0000 7fa483400700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 0x7fa46803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-05T23:26:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:10.854+0000 7fa483400700 1 -- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 msgr2=0x7fa46803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3856233973 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:10.854+0000 7fa483400700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 0x7fa46803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:11.602+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 6) v1 ==== 45308+0+0 (secure 0 0 0) 0x7fa47401abb0 con 0x7fa4841066d0 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:11.602+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 msgr2=0x7fa46803ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:11.602+0000 7fa480c00700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 0x7fa46803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.603+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 7) v1 ==== 45435+0+0 (secure 0 0 0) 0x7fa47404e200 con 0x7fa4841066d0 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.604+0000 7fa480c00700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fa46803bfe0 0x7fa46803e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.604+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa46803b320 con 0x7fa46803bfe0 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.605+0000 7fa483400700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fa46803bfe0 0x7fa46803e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.607+0000 7fa483400700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fa46803bfe0 0x7fa46803e3d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa470003a10 tx=0x7fa4700092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.607+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 <== mgr.14120 v2:172.21.15.16:6800/2429779560 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fa46803b320 con 0x7fa46803bfe0 2024-04-05T23:26:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.614+0000 7fa48ae00700 1 -- 172.21.15.16:0/1779337358 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa484067f50 con 0x7fa46803bfe0 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.614+0000 7fa480c00700 1 -- 172.21.15.16:0/1779337358 <== mgr.14120 v2:172.21.15.16:6800/2429779560 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa484067f50 con 0x7fa46803bfe0 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 -- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fa46803bfe0 msgr2=0x7fa46803e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fa46803bfe0 0x7fa46803e3d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa470003a10 tx=0x7fa4700092b0 comp rx=0 tx=0).stop 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 -- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 msgr2=0x7fa48419a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa48419a470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa474009130 tx=0x7fa474011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 -- 172.21.15.16:0/1779337358 shutdown_connections 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4841066d0 0x7fa48419a470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fa46803bfe0 0x7fa46803e3d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 --2- 172.21.15.16:0/1779337358 >> [v2:172.21.15.16:6800/3856233973,v1:172.21.15.16:6801/3856233973] conn(0x7fa468038750 0x7fa46803ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 -- 172.21.15.16:0/1779337358 >> 172.21.15.16:0/1779337358 conn(0x7fa484101c90 msgr2=0x7fa484102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 -- 172.21.15.16:0/1779337358 shutdown_connections 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:12.615+0000 7fa48ae00700 1 -- 172.21.15.16:0/1779337358 wait complete. 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-04-05T23:26:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-04-05T23:26:14.028 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:13 smithi016 ceph-mon[38248]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-05T23:26:14.029 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:13 smithi016 ceph-mon[38248]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-05T23:26:14.029 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:13 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:14.029 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:13 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:14.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.461+0000 7fce71600700 1 Processor -- start 2024-04-05T23:26:14.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.461+0000 7fce71600700 1 -- start start 2024-04-05T23:26:14.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.461+0000 7fce71600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:14.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.461+0000 7fce71600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0797e0 con 0x7fce6c07adb0 2024-04-05T23:26:14.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.462+0000 7fce6aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.462+0000 7fce6aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38784/0 (socket says 172.21.15.16:38784) 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.462+0000 7fce6aa00700 1 -- 172.21.15.16:0/3909923585 learned_addr learned my addr 172.21.15.16:0/3909923585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.462+0000 7fce6aa00700 1 -- 172.21.15.16:0/3909923585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c079920 con 0x7fce6c07adb0 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.462+0000 7fce6aa00700 1 --2- 172.21.15.16:0/3909923585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c079210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0098a0 tx=0x7fce5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9bb8841270eef1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce69600700 1 -- 172.21.15.16:0/3909923585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce5c004160 con 0x7fce6c07adb0 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce69600700 1 -- 172.21.15.16:0/3909923585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce5c0042c0 con 0x7fce6c07adb0 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce69600700 1 -- 172.21.15.16:0/3909923585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce5c011a00 con 0x7fce6c07adb0 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce71600700 1 -- 172.21.15.16:0/3909923585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 msgr2=0x7fce6c079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce71600700 1 --2- 172.21.15.16:0/3909923585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c079210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0098a0 tx=0x7fce5c009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce71600700 1 -- 172.21.15.16:0/3909923585 shutdown_connections 2024-04-05T23:26:14.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce71600700 1 --2- 172.21.15.16:0/3909923585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c079210 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce71600700 1 -- 172.21.15.16:0/3909923585 >> 172.21.15.16:0/3909923585 conn(0x7fce6c101c60 msgr2=0x7fce6c1040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce71600700 1 -- 172.21.15.16:0/3909923585 shutdown_connections 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.463+0000 7fce71600700 1 -- 172.21.15.16:0/3909923585 wait complete. 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.464+0000 7fce71600700 1 Processor -- start 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.464+0000 7fce71600700 1 -- start start 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.465+0000 7fce71600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.465+0000 7fce71600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c19cbc0 con 0x7fce6c07adb0 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.465+0000 7fce6aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.465+0000 7fce6aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38790/0 (socket says 172.21.15.16:38790) 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.465+0000 7fce6aa00700 1 -- 172.21.15.16:0/1911745446 learned_addr learned my addr 172.21.15.16:0/1911745446 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.465+0000 7fce6aa00700 1 -- 172.21.15.16:0/1911745446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5c009550 con 0x7fce6c07adb0 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.466+0000 7fce6aa00700 1 --2- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c19c680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009130 tx=0x7fce5c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:14.215 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.466+0000 7fce63400700 1 -- 172.21.15.16:0/1911745446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce5c003ab0 con 0x7fce6c07adb0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.466+0000 7fce63400700 1 -- 172.21.15.16:0/1911745446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce5c003c10 con 0x7fce6c07adb0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.466+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c19cdc0 con 0x7fce6c07adb0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.466+0000 7fce63400700 1 -- 172.21.15.16:0/1911745446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce5c0247d0 con 0x7fce6c07adb0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.466+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c19d260 con 0x7fce6c07adb0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.467+0000 7fce63400700 1 -- 172.21.15.16:0/1911745446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 7) v1 ==== 45435+0+0 (secure 0 0 0) 0x7fce5c01abb0 con 0x7fce6c07adb0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.467+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c10ca90 con 0x7fce6c07adb0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.468+0000 7fce63400700 1 --2- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fce500385d0 0x7fce5003aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.468+0000 7fce63400700 1 -- 172.21.15.16:0/1911745446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fce5c04c660 con 0x7fce6c07adb0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.468+0000 7fce6a000700 1 --2- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fce500385d0 0x7fce5003aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.469+0000 7fce6a000700 1 --2- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fce500385d0 0x7fce5003aa90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fce58006fd0 tx=0x7fce58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.473+0000 7fce63400700 1 -- 172.21.15.16:0/1911745446 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fce5c018920 con 0x7fce6c07adb0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.702+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fce6c102fe0 con 0x7fce500385d0 2024-04-05T23:26:14.216 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.717+0000 7fce63400700 1 -- 172.21.15.16:0/1911745446 <== mgr.14120 v2:172.21.15.16:6800/2429779560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fce6c102fe0 con 0x7fce500385d0 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.719+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fce500385d0 msgr2=0x7fce5003aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.719+0000 7fce71600700 1 --2- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fce500385d0 0x7fce5003aa90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fce58006fd0 tx=0x7fce58006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.719+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 msgr2=0x7fce6c19c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.719+0000 7fce71600700 1 --2- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c19c680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009130 tx=0x7fce5c011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.719+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 shutdown_connections 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.719+0000 7fce71600700 1 --2- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fce500385d0 0x7fce5003aa90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.719+0000 7fce71600700 1 --2- 172.21.15.16:0/1911745446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6c07adb0 0x7fce6c19c680 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.719+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 >> 172.21.15.16:0/1911745446 conn(0x7fce6c101c60 msgr2=0x7fce6c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.720+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 shutdown_connections 2024-04-05T23:26:14.217 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:13.720+0000 7fce71600700 1 -- 172.21.15.16:0/1911745446 wait complete. 2024-04-05T23:26:15.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:14 smithi016 ceph-mon[38248]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:15.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:14 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:15.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:14 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:26:15.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:14 smithi016 ceph-mon[38248]: mgrmap e8: x(active, since 2s) 2024-04-05T23:26:15.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-05T23:26:15.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.616+0000 7f9c03a00700 1 Processor -- start 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.616+0000 7f9c03a00700 1 -- start start 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.616+0000 7f9c03a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.616+0000 7f9c03a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc074660 con 0x7f9bfc104f90 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.617+0000 7f9c01000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.617+0000 7f9c01000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38796/0 (socket says 172.21.15.16:38796) 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.617+0000 7f9c01000700 1 -- 172.21.15.16:0/1961395102 learned_addr learned my addr 172.21.15.16:0/1961395102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.617+0000 7f9c01000700 1 -- 172.21.15.16:0/1961395102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc1058f0 con 0x7f9bfc104f90 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.617+0000 7f9c01000700 1 --2- 172.21.15.16:0/1961395102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc1053b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00098a0 tx=0x7f9bf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bff1f97650f569 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9bfb400700 1 -- 172.21.15.16:0/1961395102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bf0004160 con 0x7f9bfc104f90 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9bfb400700 1 -- 172.21.15.16:0/1961395102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bf00042c0 con 0x7f9bfc104f90 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9bfb400700 1 -- 172.21.15.16:0/1961395102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bf0011a00 con 0x7f9bfc104f90 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9c03a00700 1 -- 172.21.15.16:0/1961395102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 msgr2=0x7f9bfc1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:15.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9c03a00700 1 --2- 172.21.15.16:0/1961395102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc1053b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00098a0 tx=0x7f9bf0009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9c03a00700 1 -- 172.21.15.16:0/1961395102 shutdown_connections 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9c03a00700 1 --2- 172.21.15.16:0/1961395102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc1053b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9c03a00700 1 -- 172.21.15.16:0/1961395102 >> 172.21.15.16:0/1961395102 conn(0x7f9bfc100b50 msgr2=0x7f9bfc102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9c03a00700 1 -- 172.21.15.16:0/1961395102 shutdown_connections 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.618+0000 7f9c03a00700 1 -- 172.21.15.16:0/1961395102 wait complete. 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.619+0000 7f9c03a00700 1 Processor -- start 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.620+0000 7f9c03a00700 1 -- start start 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.620+0000 7f9c03a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.620+0000 7f9c03a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc19cb70 con 0x7f9bfc104f90 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.620+0000 7f9c01000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.620+0000 7f9c01000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc19c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38800/0 (socket says 172.21.15.16:38800) 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.620+0000 7f9c01000700 1 -- 172.21.15.16:0/3097566588 learned_addr learned my addr 172.21.15.16:0/3097566588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:15.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.620+0000 7f9c01000700 1 -- 172.21.15.16:0/3097566588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf0009550 con 0x7f9bfc104f90 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.621+0000 7f9c01000700 1 --2- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc19c630 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009130 tx=0x7f9bf0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.621+0000 7f9bf9600700 1 -- 172.21.15.16:0/3097566588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bf0003b70 con 0x7f9bfc104f90 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.621+0000 7f9bf9600700 1 -- 172.21.15.16:0/3097566588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bf0003cd0 con 0x7f9bfc104f90 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.621+0000 7f9bf9600700 1 -- 172.21.15.16:0/3097566588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bf00197d0 con 0x7f9bfc104f90 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.621+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bfc19cd70 con 0x7f9bfc104f90 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.621+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bfc19d210 con 0x7f9bfc104f90 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.622+0000 7f9bf9600700 1 -- 172.21.15.16:0/3097566588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45541+0+0 (secure 0 0 0) 0x7f9bf001abb0 con 0x7f9bfc104f90 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.622+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bfc196360 con 0x7f9bfc104f90 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.622+0000 7f9bf9600700 1 --2- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9be4038750 0x7f9be403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.623+0000 7f9bf9600700 1 -- 172.21.15.16:0/3097566588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9bf004c9b0 con 0x7f9bfc104f90 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.623+0000 7f9bfbe00700 1 --2- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9be4038750 0x7f9be403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.623+0000 7f9bfbe00700 1 --2- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9be4038750 0x7f9be403ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9be8006fd0 tx=0x7f9be8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:15.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.628+0000 7f9bf9600700 1 -- 172.21.15.16:0/3097566588 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9bf001a430 con 0x7f9bfc104f90 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.840+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bfc02cfb0 con 0x7f9be4038750 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.841+0000 7f9bf9600700 1 -- 172.21.15.16:0/3097566588 <== mgr.14120 v2:172.21.15.16:6800/2429779560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f9bfc02cfb0 con 0x7f9be4038750 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.843+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9be4038750 msgr2=0x7f9be403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.843+0000 7f9c03a00700 1 --2- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9be4038750 0x7f9be403ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9be8006fd0 tx=0x7f9be8006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.843+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 msgr2=0x7f9bfc19c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.843+0000 7f9c03a00700 1 --2- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc19c630 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009130 tx=0x7f9bf0011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.844+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 shutdown_connections 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.844+0000 7f9c03a00700 1 --2- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9be4038750 0x7f9be403ac10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.844+0000 7f9c03a00700 1 --2- 172.21.15.16:0/3097566588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bfc104f90 0x7f9bfc19c630 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.844+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 >> 172.21.15.16:0/3097566588 conn(0x7f9bfc100b50 msgr2=0x7f9bfc1934a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.844+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 shutdown_connections 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:14.844+0000 7f9c03a00700 1 -- 172.21.15.16:0/3097566588 wait complete. 2024-04-05T23:26:15.195 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-04-05T23:26:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:15 smithi016 ceph-mon[38248]: [05/Apr/2024:23:26:14] ENGINE Bus STARTING 2024-04-05T23:26:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:15 smithi016 ceph-mon[38248]: [05/Apr/2024:23:26:14] ENGINE Serving on https://172.21.15.16:7150 2024-04-05T23:26:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:15 smithi016 ceph-mon[38248]: [05/Apr/2024:23:26:14] ENGINE Serving on http://172.21.15.16:8765 2024-04-05T23:26:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:15 smithi016 ceph-mon[38248]: [05/Apr/2024:23:26:14] ENGINE Bus STARTED 2024-04-05T23:26:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:15 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:26:17.052 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.573+0000 7f7336400700 1 Processor -- start 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.573+0000 7f7336400700 1 -- start start 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.573+0000 7f7336400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f7330106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.573+0000 7f7336400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73301070b0 con 0x7f73301066c0 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.574+0000 7f732f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f7330106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.574+0000 7f732f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f7330106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38812/0 (socket says 172.21.15.16:38812) 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.574+0000 7f732f400700 1 -- 172.21.15.16:0/1762587100 learned_addr learned my addr 172.21.15.16:0/1762587100 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.574+0000 7f732f400700 1 -- 172.21.15.16:0/1762587100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73301078c0 con 0x7f73301066c0 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.574+0000 7f732f400700 1 --2- 172.21.15.16:0/1762587100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f7330106ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f73200098a0 tx=0x7f7320009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b501e1c4498c53 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.574+0000 7f732e000700 1 -- 172.21.15.16:0/1762587100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7320004160 con 0x7f73301066c0 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.575+0000 7f732e000700 1 -- 172.21.15.16:0/1762587100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73200042c0 con 0x7f73301066c0 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.575+0000 7f732e000700 1 -- 172.21.15.16:0/1762587100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7320011a00 con 0x7f73301066c0 2024-04-05T23:26:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.575+0000 7f7336400700 1 -- 172.21.15.16:0/1762587100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 msgr2=0x7f7330106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.575+0000 7f7336400700 1 --2- 172.21.15.16:0/1762587100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f7330106ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f73200098a0 tx=0x7f7320009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.575+0000 7f7336400700 1 -- 172.21.15.16:0/1762587100 shutdown_connections 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.575+0000 7f7336400700 1 --2- 172.21.15.16:0/1762587100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f7330106ae0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.575+0000 7f7336400700 1 -- 172.21.15.16:0/1762587100 >> 172.21.15.16:0/1762587100 conn(0x7f7330101c60 msgr2=0x7f73301040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.575+0000 7f7336400700 1 -- 172.21.15.16:0/1762587100 shutdown_connections 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.575+0000 7f7336400700 1 -- 172.21.15.16:0/1762587100 wait complete. 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.576+0000 7f7336400700 1 Processor -- start 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.576+0000 7f7336400700 1 -- start start 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.577+0000 7f7336400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f733007aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.577+0000 7f7336400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733007af70 con 0x7f73301066c0 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.577+0000 7f732f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f733007aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.577+0000 7f732f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f733007aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38820/0 (socket says 172.21.15.16:38820) 2024-04-05T23:26:17.054 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.577+0000 7f732f400700 1 -- 172.21.15.16:0/2750176451 learned_addr learned my addr 172.21.15.16:0/2750176451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.577+0000 7f732f400700 1 -- 172.21.15.16:0/2750176451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7320009550 con 0x7f73301066c0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.578+0000 7f732f400700 1 --2- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f733007aa30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7320009130 tx=0x7f7320011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.578+0000 7f7327e00700 1 -- 172.21.15.16:0/2750176451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7320003ab0 con 0x7f73301066c0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.578+0000 7f7327e00700 1 -- 172.21.15.16:0/2750176451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7320003c10 con 0x7f73301066c0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.578+0000 7f7327e00700 1 -- 172.21.15.16:0/2750176451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73200247d0 con 0x7f73301066c0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.578+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73300791b0 con 0x7f73301066c0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.578+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7330079650 con 0x7f73301066c0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.579+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7330067f50 con 0x7f73301066c0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.579+0000 7f7327e00700 1 -- 172.21.15.16:0/2750176451 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45541+0+0 (secure 0 0 0) 0x7f732001abb0 con 0x7f73301066c0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.579+0000 7f7327e00700 1 --2- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f7314038750 0x7f731403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.580+0000 7f7327e00700 1 -- 172.21.15.16:0/2750176451 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f732004c850 con 0x7f73301066c0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.580+0000 7f732ea00700 1 --2- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f7314038750 0x7f731403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.580+0000 7f732ea00700 1 --2- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f7314038750 0x7f731403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f731c006fd0 tx=0x7f731c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.584+0000 7f7327e00700 1 -- 172.21.15.16:0/2750176451 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7320018530 con 0x7f73301066c0 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:15.799+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7330079c90 con 0x7f7314038750 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.693+0000 7f7327e00700 1 -- 172.21.15.16:0/2750176451 <== mgr.14120 v2:172.21.15.16:6800/2429779560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7330079c90 con 0x7f7314038750 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.695+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f7314038750 msgr2=0x7f731403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.695+0000 7f7336400700 1 --2- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f7314038750 0x7f731403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f731c006fd0 tx=0x7f731c006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.695+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 msgr2=0x7f733007aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.695+0000 7f7336400700 1 --2- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f733007aa30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7320009130 tx=0x7f7320011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.695+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 shutdown_connections 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.695+0000 7f7336400700 1 --2- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f7314038750 0x7f731403ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.695+0000 7f7336400700 1 --2- 172.21.15.16:0/2750176451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73301066c0 0x7f733007aa30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.695+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 >> 172.21.15.16:0/2750176451 conn(0x7f7330101c60 msgr2=0x7f73301028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.696+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 shutdown_connections 2024-04-05T23:26:17.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:16.696+0000 7f7336400700 1 -- 172.21.15.16:0/2750176451 wait complete. 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-mon[38248]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: Generating public/private rsa key pair. 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: Your identification has been saved in /tmp/tmpvtjh4cbs/key. 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: Your public key has been saved in /tmp/tmpvtjh4cbs/key.pub. 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: The key fingerprint is: 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: SHA256:onGJaXJB/rQN3CEIDO1WhF2jSyU9ijafJPikaezDxdE ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: The key's randomart image is: 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: +---[RSA 3072]----+ 2024-04-05T23:26:17.057 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: |.+.=++= . | 2024-04-05T23:26:17.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: | +o+=o+ . | 2024-04-05T23:26:17.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: | o o*.+.. | 2024-04-05T23:26:17.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: |. OooE = | 2024-04-05T23:26:17.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: |.Bo=O.* S | 2024-04-05T23:26:17.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: |oo.*o+ . | 2024-04-05T23:26:17.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: |+ . . | 2024-04-05T23:26:17.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: | + | 2024-04-05T23:26:17.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: | . | 2024-04-05T23:26:17.058 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:16 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: +----[SHA256]-----+ 2024-04-05T23:26:18.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDBD+3pbLOJbhKPHoBxGja0yX3uIAk9Q0MvND3y4qdMzRrdb3aRvbQfisjNiN9kwZxA858fNYNXW9r9+ZiqbsT7WO4cdRNI97q1SVS75ukiGX4COU35tWQbKDm7r2NqXSWWRc32mWUw803JgzAzx/0baqdhjdkz9W96Hb7Wl9wA+oA7O/KrqyZ8Yo5MdgniKzuPp/l45hEs0OMiobY7hAzNRI9CEV4038Bra+WdGhZ2avTscSZDyPze47KGxy1zCguepU6AjwBNcZ5Q6jmXkrwBFQOuROFuFaXPTShp6SgstrhGxhb+VuYLyLL5Jp+T+7XeD8OCiRMbL3XmWE4BrDxOaAZHiOyXSWJkXyYUF36D9KJFbqXGV5lzqSnD22JmFaEwGKDXuFppOTe7eyImdRCZ6/Mp7/AdqUQ2UjA7N+IWAeSpy6I+Cvou7xNtrwhqZzNgfzQ2e/Mq2/FZEkjFaNVLfzJ38L/zkq1VByOawEkSJMkv2EVelU5LWQa/uCw0eM= ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.465+0000 7fd43ec00700 1 Processor -- start 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.466+0000 7fd43ec00700 1 -- start start 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.466+0000 7fd43ec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd438106af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.466+0000 7fd43ec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4381070c0 con 0x7fd4381066d0 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.466+0000 7fd437e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd438106af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.467+0000 7fd437e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd438106af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38832/0 (socket says 172.21.15.16:38832) 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.467+0000 7fd437e00700 1 -- 172.21.15.16:0/2814956765 learned_addr learned my addr 172.21.15.16:0/2814956765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.467+0000 7fd437e00700 1 -- 172.21.15.16:0/2814956765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4381078d0 con 0x7fd4381066d0 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.467+0000 7fd437e00700 1 --2- 172.21.15.16:0/2814956765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd438106af0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4280098a0 tx=0x7fd428009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d19b54111304f88 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.467+0000 7fd436a00700 1 -- 172.21.15.16:0/2814956765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd428004160 con 0x7fd4381066d0 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.467+0000 7fd436a00700 1 -- 172.21.15.16:0/2814956765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4280042c0 con 0x7fd4381066d0 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.467+0000 7fd436a00700 1 -- 172.21.15.16:0/2814956765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd428011a00 con 0x7fd4381066d0 2024-04-05T23:26:18.059 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.468+0000 7fd43ec00700 1 -- 172.21.15.16:0/2814956765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 msgr2=0x7fd438106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:18.060 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.468+0000 7fd43ec00700 1 --2- 172.21.15.16:0/2814956765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd438106af0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4280098a0 tx=0x7fd428009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:18.060 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.468+0000 7fd43ec00700 1 -- 172.21.15.16:0/2814956765 shutdown_connections 2024-04-05T23:26:18.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.468+0000 7fd43ec00700 1 --2- 172.21.15.16:0/2814956765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd438106af0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:18.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.468+0000 7fd43ec00700 1 -- 172.21.15.16:0/2814956765 >> 172.21.15.16:0/2814956765 conn(0x7fd438101cd0 msgr2=0x7fd4381040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.468+0000 7fd43ec00700 1 -- 172.21.15.16:0/2814956765 shutdown_connections 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.468+0000 7fd43ec00700 1 -- 172.21.15.16:0/2814956765 wait complete. 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.469+0000 7fd43ec00700 1 Processor -- start 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.469+0000 7fd43ec00700 1 -- start start 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.470+0000 7fd43ec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd43819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.470+0000 7fd43ec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43819cb80 con 0x7fd4381066d0 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.470+0000 7fd437e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd43819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.470+0000 7fd437e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd43819c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38846/0 (socket says 172.21.15.16:38846) 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.470+0000 7fd437e00700 1 -- 172.21.15.16:0/1871967492 learned_addr learned my addr 172.21.15.16:0/1871967492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.470+0000 7fd437e00700 1 -- 172.21.15.16:0/1871967492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd428009550 con 0x7fd4381066d0 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.470+0000 7fd437e00700 1 --2- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd43819c640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd428009130 tx=0x7fd428011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.471+0000 7fd434c00700 1 -- 172.21.15.16:0/1871967492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd428003ab0 con 0x7fd4381066d0 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.471+0000 7fd434c00700 1 -- 172.21.15.16:0/1871967492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd428003c10 con 0x7fd4381066d0 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.471+0000 7fd434c00700 1 -- 172.21.15.16:0/1871967492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4280247d0 con 0x7fd4381066d0 2024-04-05T23:26:18.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.471+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43819cd80 con 0x7fd4381066d0 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.471+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43819d220 con 0x7fd4381066d0 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.472+0000 7fd434c00700 1 -- 172.21.15.16:0/1871967492 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45541+0+0 (secure 0 0 0) 0x7fd42801abb0 con 0x7fd4381066d0 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.472+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43810ab00 con 0x7fd4381066d0 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.472+0000 7fd434c00700 1 --2- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fd41c038750 0x7fd41c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.472+0000 7fd434c00700 1 -- 172.21.15.16:0/1871967492 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd42804c790 con 0x7fd4381066d0 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.473+0000 7fd437400700 1 --2- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fd41c038750 0x7fd41c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.473+0000 7fd437400700 1 --2- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fd41c038750 0x7fd41c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd424006fd0 tx=0x7fd424006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.478+0000 7fd434c00700 1 -- 172.21.15.16:0/1871967492 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd428018530 con 0x7fd4381066d0 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.693+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd438102f30 con 0x7fd41c038750 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.694+0000 7fd434c00700 1 -- 172.21.15.16:0/1871967492 <== mgr.14120 v2:172.21.15.16:6800/2429779560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fd438102f30 con 0x7fd41c038750 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fd41c038750 msgr2=0x7fd41c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 --2- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fd41c038750 0x7fd41c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd424006fd0 tx=0x7fd424006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 msgr2=0x7fd43819c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 --2- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd43819c640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd428009130 tx=0x7fd428011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 shutdown_connections 2024-04-05T23:26:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 --2- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fd41c038750 0x7fd41c03ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:18.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 --2- 172.21.15.16:0/1871967492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4381066d0 0x7fd43819c640 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:18.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 >> 172.21.15.16:0/1871967492 conn(0x7fd438101cd0 msgr2=0x7fd438102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:18.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 shutdown_connections 2024-04-05T23:26:18.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:17.696+0000 7fd43ec00700 1 -- 172.21.15.16:0/1871967492 wait complete. 2024-04-05T23:26:18.066 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-05T23:26:18.066 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-04-05T23:26:18.067 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:17 smithi016 ceph-mon[38248]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:18.067 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:17 smithi016 ceph-mon[38248]: Generating ssh key... 2024-04-05T23:26:18.067 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-04-05T23:26:19.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:18 smithi016 ceph-mon[38248]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:19.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:19 smithi016 ceph-mon[38248]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:20.987 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:20 smithi016 ceph-mon[38248]: Deploying cephadm binary to smithi016 2024-04-05T23:26:22.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-04-05T23:26:22.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.451+0000 7f45d2000700 1 Processor -- start 2024-04-05T23:26:22.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.452+0000 7f45d2000700 1 -- start start 2024-04-05T23:26:22.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.452+0000 7f45d2000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:22.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.452+0000 7f45d2000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc1070b0 con 0x7f45cc1066c0 2024-04-05T23:26:22.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.452+0000 7f45cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:22.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.452+0000 7f45cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38862/0 (socket says 172.21.15.16:38862) 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.452+0000 7f45cb400700 1 -- 172.21.15.16:0/1231301092 learned_addr learned my addr 172.21.15.16:0/1231301092 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.453+0000 7f45cb400700 1 -- 172.21.15.16:0/1231301092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc1078c0 con 0x7f45cc1066c0 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.453+0000 7f45cb400700 1 --2- 172.21.15.16:0/1231301092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc106ae0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0098a0 tx=0x7f45bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea05b0b68f29b7c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.453+0000 7f45caa00700 1 -- 172.21.15.16:0/1231301092 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45bc004160 con 0x7f45cc1066c0 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.453+0000 7f45caa00700 1 -- 172.21.15.16:0/1231301092 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45bc0042c0 con 0x7f45cc1066c0 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.453+0000 7f45caa00700 1 -- 172.21.15.16:0/1231301092 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45bc011a00 con 0x7f45cc1066c0 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.454+0000 7f45d2000700 1 -- 172.21.15.16:0/1231301092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 msgr2=0x7f45cc106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.454+0000 7f45d2000700 1 --2- 172.21.15.16:0/1231301092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc106ae0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0098a0 tx=0x7f45bc009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.454+0000 7f45d2000700 1 -- 172.21.15.16:0/1231301092 shutdown_connections 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.454+0000 7f45d2000700 1 --2- 172.21.15.16:0/1231301092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc106ae0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.454+0000 7f45d2000700 1 -- 172.21.15.16:0/1231301092 >> 172.21.15.16:0/1231301092 conn(0x7f45cc101c60 msgr2=0x7f45cc1040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.454+0000 7f45d2000700 1 -- 172.21.15.16:0/1231301092 shutdown_connections 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.454+0000 7f45d2000700 1 -- 172.21.15.16:0/1231301092 wait complete. 2024-04-05T23:26:22.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.455+0000 7f45d2000700 1 Processor -- start 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.455+0000 7f45d2000700 1 -- start start 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.455+0000 7f45d2000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.455+0000 7f45d2000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc19cba0 con 0x7f45cc1066c0 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.456+0000 7f45cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.456+0000 7f45cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc19c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38872/0 (socket says 172.21.15.16:38872) 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.456+0000 7f45cb400700 1 -- 172.21.15.16:0/2751899654 learned_addr learned my addr 172.21.15.16:0/2751899654 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.456+0000 7f45cb400700 1 -- 172.21.15.16:0/2751899654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc009550 con 0x7f45cc1066c0 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.456+0000 7f45cb400700 1 --2- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc19c660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009fb0 tx=0x7f45bc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.456+0000 7f45c8c00700 1 -- 172.21.15.16:0/2751899654 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45bc003a40 con 0x7f45cc1066c0 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.456+0000 7f45c8c00700 1 -- 172.21.15.16:0/2751899654 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45bc003ba0 con 0x7f45cc1066c0 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.457+0000 7f45c8c00700 1 -- 172.21.15.16:0/2751899654 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45bc01a750 con 0x7f45cc1066c0 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.457+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc19cda0 con 0x7f45cc1066c0 2024-04-05T23:26:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.457+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc19d240 con 0x7f45cc1066c0 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.458+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45cc10aaf0 con 0x7f45cc1066c0 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.458+0000 7f45c8c00700 1 -- 172.21.15.16:0/2751899654 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45541+0+0 (secure 0 0 0) 0x7f45bc024b60 con 0x7f45cc1066c0 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.458+0000 7f45c8c00700 1 --2- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f45b0038700 0x7f45b003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.458+0000 7f45c8c00700 1 -- 172.21.15.16:0/2751899654 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f45bc04c6c0 con 0x7f45cc1066c0 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.458+0000 7f45c3e00700 1 --2- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f45b0038700 0x7f45b003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.459+0000 7f45c3e00700 1 --2- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f45b0038700 0x7f45b003abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f45b4006fd0 tx=0x7f45b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.463+0000 7f45c8c00700 1 -- 172.21.15.16:0/2751899654 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f45bc024430 con 0x7f45cc1066c0 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:18.681+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}) v1 -- 0x7f45cc102f10 con 0x7f45b0038700 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.695+0000 7f45c8c00700 1 -- 172.21.15.16:0/2751899654 <== mgr.14120 v2:172.21.15.16:6800/2429779560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f45cc102f10 con 0x7f45b0038700 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.698+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f45b0038700 msgr2=0x7f45b003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.698+0000 7f45d2000700 1 --2- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f45b0038700 0x7f45b003abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f45b4006fd0 tx=0x7f45b4006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.698+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 msgr2=0x7f45cc19c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.698+0000 7f45d2000700 1 --2- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc19c660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009fb0 tx=0x7f45bc011ad0 comp rx=0 tx=0).stop 2024-04-05T23:26:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.698+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 shutdown_connections 2024-04-05T23:26:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.698+0000 7f45d2000700 1 --2- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f45b0038700 0x7f45b003abc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.698+0000 7f45d2000700 1 --2- 172.21.15.16:0/2751899654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45cc1066c0 0x7f45cc19c660 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.698+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 >> 172.21.15.16:0/2751899654 conn(0x7f45cc101c60 msgr2=0x7f45cc102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.698+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 shutdown_connections 2024-04-05T23:26:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:21.699+0000 7f45d2000700 1 -- 172.21.15.16:0/2751899654 wait complete. 2024-04-05T23:26:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mon service... 2024-04-05T23:26:22.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:22 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:22.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:22 smithi016 ceph-mon[38248]: Added host smithi016 2024-04-05T23:26:22.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:22 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:26:23.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-05T23:26:23.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.484+0000 7f4b8d200700 1 Processor -- start 2024-04-05T23:26:23.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.484+0000 7f4b8d200700 1 -- start start 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.484+0000 7f4b8d200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b880791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.484+0000 7f4b8d200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880797a0 con 0x7f4b8807ad70 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.485+0000 7f4b87400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b880791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.485+0000 7f4b87400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b880791d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48060/0 (socket says 172.21.15.16:48060) 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.485+0000 7f4b87400700 1 -- 172.21.15.16:0/1881143129 learned_addr learned my addr 172.21.15.16:0/1881143129 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.485+0000 7f4b87400700 1 -- 172.21.15.16:0/1881143129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b880798e0 con 0x7f4b8807ad70 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.485+0000 7f4b87400700 1 --2- 172.21.15.16:0/1881143129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b880791d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b780098a0 tx=0x7f4b78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2b4bb1b6406770 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.485+0000 7f4b86000700 1 -- 172.21.15.16:0/1881143129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b78004160 con 0x7f4b8807ad70 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.486+0000 7f4b86000700 1 -- 172.21.15.16:0/1881143129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b780042c0 con 0x7f4b8807ad70 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.486+0000 7f4b86000700 1 -- 172.21.15.16:0/1881143129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b78011a00 con 0x7f4b8807ad70 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.486+0000 7f4b8d200700 1 -- 172.21.15.16:0/1881143129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 msgr2=0x7f4b880791d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.486+0000 7f4b8d200700 1 --2- 172.21.15.16:0/1881143129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b880791d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b780098a0 tx=0x7f4b78009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:23.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.486+0000 7f4b8d200700 1 -- 172.21.15.16:0/1881143129 shutdown_connections 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.486+0000 7f4b8d200700 1 --2- 172.21.15.16:0/1881143129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b880791d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.486+0000 7f4b8d200700 1 -- 172.21.15.16:0/1881143129 >> 172.21.15.16:0/1881143129 conn(0x7f4b88101ad0 msgr2=0x7f4b88103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.486+0000 7f4b8d200700 1 -- 172.21.15.16:0/1881143129 shutdown_connections 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.486+0000 7f4b8d200700 1 -- 172.21.15.16:0/1881143129 wait complete. 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.487+0000 7f4b8d200700 1 Processor -- start 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.487+0000 7f4b8d200700 1 -- start start 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.488+0000 7f4b8d200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b8819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.488+0000 7f4b8d200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8819ca00 con 0x7f4b8807ad70 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.488+0000 7f4b87400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b8819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.488+0000 7f4b87400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b8819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48076/0 (socket says 172.21.15.16:48076) 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.488+0000 7f4b87400700 1 -- 172.21.15.16:0/1656481728 learned_addr learned my addr 172.21.15.16:0/1656481728 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.488+0000 7f4b87400700 1 -- 172.21.15.16:0/1656481728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b78009550 con 0x7f4b8807ad70 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.489+0000 7f4b87400700 1 --2- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b8819c4c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b78011a20 tx=0x7f4b78011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:23.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.489+0000 7f4b7fe00700 1 -- 172.21.15.16:0/1656481728 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b78003af0 con 0x7f4b8807ad70 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.489+0000 7f4b7fe00700 1 -- 172.21.15.16:0/1656481728 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b78003c50 con 0x7f4b8807ad70 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.489+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8819cc00 con 0x7f4b8807ad70 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.489+0000 7f4b7fe00700 1 -- 172.21.15.16:0/1656481728 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b78019760 con 0x7f4b8807ad70 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.489+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8819d0a0 con 0x7f4b8807ad70 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.490+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8810c8a0 con 0x7f4b8807ad70 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.490+0000 7f4b7fe00700 1 -- 172.21.15.16:0/1656481728 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45541+0+0 (secure 0 0 0) 0x7f4b7801a430 con 0x7f4b8807ad70 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.491+0000 7f4b7fe00700 1 --2- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f4b6c038750 0x7f4b6c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.491+0000 7f4b7fe00700 1 -- 172.21.15.16:0/1656481728 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4b7804c8f0 con 0x7f4b8807ad70 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.491+0000 7f4b86a00700 1 --2- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f4b6c038750 0x7f4b6c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.492+0000 7f4b86a00700 1 --2- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f4b6c038750 0x7f4b6c03ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4b74006fd0 tx=0x7f4b74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.496+0000 7f4b7fe00700 1 -- 172.21.15.16:0/1656481728 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4b7802d7c0 con 0x7f4b8807ad70 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.729+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4b88102e30 con 0x7f4b6c038750 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.734+0000 7f4b7fe00700 1 -- 172.21.15.16:0/1656481728 <== mgr.14120 v2:172.21.15.16:6800/2429779560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4b88102e30 con 0x7f4b6c038750 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.736+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f4b6c038750 msgr2=0x7f4b6c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:23.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.736+0000 7f4b8d200700 1 --2- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f4b6c038750 0x7f4b6c03ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4b74006fd0 tx=0x7f4b74006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:23.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.736+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 msgr2=0x7f4b8819c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:23.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.736+0000 7f4b8d200700 1 --2- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b8819c4c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b78011a20 tx=0x7f4b78011b00 comp rx=0 tx=0).stop 2024-04-05T23:26:23.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.736+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 shutdown_connections 2024-04-05T23:26:23.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.736+0000 7f4b8d200700 1 --2- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f4b6c038750 0x7f4b6c03ac10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:23.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.736+0000 7f4b8d200700 1 --2- 172.21.15.16:0/1656481728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b8807ad70 0x7f4b8819c4c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:23.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.737+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 >> 172.21.15.16:0/1656481728 conn(0x7f4b88101ad0 msgr2=0x7f4b881026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:23.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.737+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 shutdown_connections 2024-04-05T23:26:23.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:22.737+0000 7f4b8d200700 1 -- 172.21.15.16:0/1656481728 wait complete. 2024-04-05T23:26:23.166 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mgr service... 2024-04-05T23:26:24.001 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:23 smithi016 ceph-mon[38248]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:24.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:23 smithi016 ceph-mon[38248]: Saving service mon spec with placement count:5 2024-04-05T23:26:24.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:23 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:24.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:23 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:24.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-05T23:26:24.343 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.738+0000 7fc27c000700 1 Processor -- start 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.738+0000 7fc27c000700 1 -- start start 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.738+0000 7fc27c000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc274109580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.738+0000 7fc27c000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc274074680 con 0x7fc274107190 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.739+0000 7fc279600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc274109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.739+0000 7fc279600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc274109580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48084/0 (socket says 172.21.15.16:48084) 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.739+0000 7fc279600700 1 -- 172.21.15.16:0/3450212248 learned_addr learned my addr 172.21.15.16:0/3450212248 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.739+0000 7fc279600700 1 -- 172.21.15.16:0/3450212248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc274109ac0 con 0x7fc274107190 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.740+0000 7fc279600700 1 --2- 172.21.15.16:0/3450212248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc274109580 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc2680098a0 tx=0x7fc268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efda8207a5b734d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.740+0000 7fc273e00700 1 -- 172.21.15.16:0/3450212248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc268004160 con 0x7fc274107190 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.740+0000 7fc273e00700 1 -- 172.21.15.16:0/3450212248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2680042c0 con 0x7fc274107190 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.740+0000 7fc273e00700 1 -- 172.21.15.16:0/3450212248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc2680119b0 con 0x7fc274107190 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.741+0000 7fc27c000700 1 -- 172.21.15.16:0/3450212248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 msgr2=0x7fc274109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:24.344 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.741+0000 7fc27c000700 1 --2- 172.21.15.16:0/3450212248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc274109580 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc2680098a0 tx=0x7fc268009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.741+0000 7fc27c000700 1 -- 172.21.15.16:0/3450212248 shutdown_connections 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.741+0000 7fc27c000700 1 --2- 172.21.15.16:0/3450212248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc274109580 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.741+0000 7fc27c000700 1 -- 172.21.15.16:0/3450212248 >> 172.21.15.16:0/3450212248 conn(0x7fc274100b60 msgr2=0x7fc274102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.741+0000 7fc27c000700 1 -- 172.21.15.16:0/3450212248 shutdown_connections 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.741+0000 7fc27c000700 1 -- 172.21.15.16:0/3450212248 wait complete. 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.743+0000 7fc27c000700 1 Processor -- start 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.743+0000 7fc27c000700 1 -- start start 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.743+0000 7fc27c000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc2741069c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.743+0000 7fc27c000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc274106f00 con 0x7fc274107190 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.744+0000 7fc279600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc2741069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.744+0000 7fc279600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc2741069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48090/0 (socket says 172.21.15.16:48090) 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.744+0000 7fc279600700 1 -- 172.21.15.16:0/1408720598 learned_addr learned my addr 172.21.15.16:0/1408720598 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.744+0000 7fc279600700 1 -- 172.21.15.16:0/1408720598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc268009550 con 0x7fc274107190 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.744+0000 7fc279600700 1 --2- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc2741069c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc268011670 tx=0x7fc268011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.745+0000 7fc272000700 1 -- 172.21.15.16:0/1408720598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc268003b20 con 0x7fc274107190 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.745+0000 7fc272000700 1 -- 172.21.15.16:0/1408720598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc268003c80 con 0x7fc274107190 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.745+0000 7fc272000700 1 -- 172.21.15.16:0/1408720598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc268019770 con 0x7fc274107190 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.745+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc274105140 con 0x7fc274107190 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.745+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2741055e0 con 0x7fc274107190 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.746+0000 7fc272000700 1 -- 172.21.15.16:0/1408720598 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45541+0+0 (secure 0 0 0) 0x7fc26801a430 con 0x7fc274107190 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.746+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc274062510 con 0x7fc274107190 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.746+0000 7fc272000700 1 --2- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc25c0387a0 0x7fc25c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.746+0000 7fc272000700 1 -- 172.21.15.16:0/1408720598 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc26804c9e0 con 0x7fc274107190 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.747+0000 7fc278c00700 1 --2- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc25c0387a0 0x7fc25c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.747+0000 7fc278c00700 1 --2- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc25c0387a0 0x7fc25c03ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc260006fd0 tx=0x7fc260006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.752+0000 7fc272000700 1 -- 172.21.15.16:0/1408720598 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc26802d960 con 0x7fc274107190 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.978+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc274105820 con 0x7fc25c0387a0 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.984+0000 7fc272000700 1 -- 172.21.15.16:0/1408720598 <== mgr.14120 v2:172.21.15.16:6800/2429779560 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc274105820 con 0x7fc25c0387a0 2024-04-05T23:26:24.346 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc25c0387a0 msgr2=0x7fc25c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:24.347 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 --2- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc25c0387a0 0x7fc25c03ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc260006fd0 tx=0x7fc260006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:24.347 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 msgr2=0x7fc2741069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:24.347 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 --2- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc2741069c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc268011670 tx=0x7fc268011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:24.347 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 shutdown_connections 2024-04-05T23:26:24.347 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 --2- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc25c0387a0 0x7fc25c03ac60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:24.347 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 --2- 172.21.15.16:0/1408720598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274107190 0x7fc2741069c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:24.347 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 >> 172.21.15.16:0/1408720598 conn(0x7fc274100b60 msgr2=0x7fc274101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:24.347 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 shutdown_connections 2024-04-05T23:26:24.347 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:23.986+0000 7fc27c000700 1 -- 172.21.15.16:0/1408720598 wait complete. 2024-04-05T23:26:25.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:24 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:25.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:24 smithi016 ceph-mon[38248]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:25.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:24 smithi016 ceph-mon[38248]: Saving service mgr spec with placement count:2 2024-04-05T23:26:25.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:24 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:25.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.809+0000 7f0715000700 1 Processor -- start 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.809+0000 7f0715000700 1 -- start start 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.809+0000 7f0715000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f0710106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.809+0000 7f0715000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07101070b0 con 0x7f07101066c0 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.810+0000 7f070de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f0710106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.810+0000 7f070de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f0710106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48092/0 (socket says 172.21.15.16:48092) 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.810+0000 7f070de00700 1 -- 172.21.15.16:0/1820273126 learned_addr learned my addr 172.21.15.16:0/1820273126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.810+0000 7f070de00700 1 -- 172.21.15.16:0/1820273126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07101078c0 con 0x7f07101066c0 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.810+0000 7f070de00700 1 --2- 172.21.15.16:0/1820273126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f0710106ae0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f07000098a0 tx=0x7f0700009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e381c4312b02dab server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.811+0000 7f070ca00700 1 -- 172.21.15.16:0/1820273126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0700004160 con 0x7f07101066c0 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.811+0000 7f070ca00700 1 -- 172.21.15.16:0/1820273126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07000042c0 con 0x7f07101066c0 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.811+0000 7f070ca00700 1 -- 172.21.15.16:0/1820273126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07000119b0 con 0x7f07101066c0 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.811+0000 7f0715000700 1 -- 172.21.15.16:0/1820273126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 msgr2=0x7f0710106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:25.569 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.811+0000 7f0715000700 1 --2- 172.21.15.16:0/1820273126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f0710106ae0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f07000098a0 tx=0x7f0700009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.812+0000 7f0715000700 1 -- 172.21.15.16:0/1820273126 shutdown_connections 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.812+0000 7f0715000700 1 --2- 172.21.15.16:0/1820273126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f0710106ae0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.812+0000 7f0715000700 1 -- 172.21.15.16:0/1820273126 >> 172.21.15.16:0/1820273126 conn(0x7f0710101c60 msgr2=0x7f07101040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.812+0000 7f0715000700 1 -- 172.21.15.16:0/1820273126 shutdown_connections 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.812+0000 7f0715000700 1 -- 172.21.15.16:0/1820273126 wait complete. 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.813+0000 7f0715000700 1 Processor -- start 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.813+0000 7f0715000700 1 -- start start 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.813+0000 7f0715000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f071019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.813+0000 7f0715000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071019cc10 con 0x7f07101066c0 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.813+0000 7f070de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f071019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.814+0000 7f070de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f071019c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48098/0 (socket says 172.21.15.16:48098) 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.814+0000 7f070de00700 1 -- 172.21.15.16:0/736543234 learned_addr learned my addr 172.21.15.16:0/736543234 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.814+0000 7f070de00700 1 -- 172.21.15.16:0/736543234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700009550 con 0x7f07101066c0 2024-04-05T23:26:25.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.814+0000 7f070de00700 1 --2- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f071019c6d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0700003e60 tx=0x7f0700011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.814+0000 7f0706a00700 1 -- 172.21.15.16:0/736543234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0700003a60 con 0x7f07101066c0 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.815+0000 7f0706a00700 1 -- 172.21.15.16:0/736543234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0700003bc0 con 0x7f07101066c0 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.815+0000 7f0706a00700 1 -- 172.21.15.16:0/736543234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07000196c0 con 0x7f07101066c0 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.815+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071019ce10 con 0x7f07101066c0 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.815+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071019d2b0 con 0x7f07101066c0 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.816+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071010aaf0 con 0x7f07101066c0 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.816+0000 7f0706a00700 1 -- 172.21.15.16:0/736543234 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45541+0+0 (secure 0 0 0) 0x7f070001ab60 con 0x7f07101066c0 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.817+0000 7f0706a00700 1 --2- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f06f40386e0 0x7f06f403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.817+0000 7f0706a00700 1 -- 172.21.15.16:0/736543234 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f070004c9f0 con 0x7f07101066c0 2024-04-05T23:26:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.817+0000 7f070d400700 1 --2- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f06f40386e0 0x7f06f403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.818+0000 7f070d400700 1 --2- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f06f40386e0 0x7f06f403aba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f06fc006fd0 tx=0x7f06fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:24.822+0000 7f0706a00700 1 -- 172.21.15.16:0/736543234 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0700010790 con 0x7f07101066c0 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.043+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f071019fb40 con 0x7f07101066c0 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.046+0000 7f0706a00700 1 -- 172.21.15.16:0/736543234 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0700010380 con 0x7f07101066c0 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f06f40386e0 msgr2=0x7f06f403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 --2- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f06f40386e0 0x7f06f403aba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f06fc006fd0 tx=0x7f06fc006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 msgr2=0x7f071019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 --2- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f071019c6d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0700003e60 tx=0x7f0700011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 shutdown_connections 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 --2- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f06f40386e0 0x7f06f403aba0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 --2- 172.21.15.16:0/736543234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07101066c0 0x7f071019c6d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 >> 172.21.15.16:0/736543234 conn(0x7f0710101c60 msgr2=0x7f07101027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 shutdown_connections 2024-04-05T23:26:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:25.049+0000 7f0715000700 1 -- 172.21.15.16:0/736543234 wait complete. 2024-04-05T23:26:26.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:26 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/736543234' entity='client.admin' 2024-04-05T23:26:26.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:26 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.015+0000 7fc3dac00700 1 Processor -- start 2024-04-05T23:26:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.015+0000 7fc3dac00700 1 -- start start 2024-04-05T23:26:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.015+0000 7fc3dac00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d4079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.015+0000 7fc3dac00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d40797e0 con 0x7fc3d407adb0 2024-04-05T23:26:26.626 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.016+0000 7fc3d9800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d4079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.016+0000 7fc3d9800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d4079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48108/0 (socket says 172.21.15.16:48108) 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.016+0000 7fc3d9800700 1 -- 172.21.15.16:0/1332061532 learned_addr learned my addr 172.21.15.16:0/1332061532 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.016+0000 7fc3d9800700 1 -- 172.21.15.16:0/1332061532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d4079920 con 0x7fc3d407adb0 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.017+0000 7fc3d9800700 1 --2- 172.21.15.16:0/1332061532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d4079210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc3c80098a0 tx=0x7fc3c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943a862d4b96753e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.017+0000 7fc3d3e00700 1 -- 172.21.15.16:0/1332061532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3c8004160 con 0x7fc3d407adb0 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.017+0000 7fc3d3e00700 1 -- 172.21.15.16:0/1332061532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c80042c0 con 0x7fc3d407adb0 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.017+0000 7fc3d3e00700 1 -- 172.21.15.16:0/1332061532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3c80119b0 con 0x7fc3d407adb0 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.018+0000 7fc3dac00700 1 -- 172.21.15.16:0/1332061532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 msgr2=0x7fc3d4079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.018+0000 7fc3dac00700 1 --2- 172.21.15.16:0/1332061532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d4079210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc3c80098a0 tx=0x7fc3c8009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.018+0000 7fc3dac00700 1 -- 172.21.15.16:0/1332061532 shutdown_connections 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.018+0000 7fc3dac00700 1 --2- 172.21.15.16:0/1332061532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d4079210 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.018+0000 7fc3dac00700 1 -- 172.21.15.16:0/1332061532 >> 172.21.15.16:0/1332061532 conn(0x7fc3d4101ad0 msgr2=0x7fc3d4103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.018+0000 7fc3dac00700 1 -- 172.21.15.16:0/1332061532 shutdown_connections 2024-04-05T23:26:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.018+0000 7fc3dac00700 1 -- 172.21.15.16:0/1332061532 wait complete. 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.019+0000 7fc3dac00700 1 Processor -- start 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.020+0000 7fc3dac00700 1 -- start start 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.020+0000 7fc3dac00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.020+0000 7fc3dac00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d419ca70 con 0x7fc3d407adb0 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.020+0000 7fc3d9800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.020+0000 7fc3d9800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d419c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48112/0 (socket says 172.21.15.16:48112) 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.020+0000 7fc3d9800700 1 -- 172.21.15.16:0/1677683612 learned_addr learned my addr 172.21.15.16:0/1677683612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.021+0000 7fc3d9800700 1 -- 172.21.15.16:0/1677683612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8009550 con 0x7fc3d407adb0 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.021+0000 7fc3d9800700 1 --2- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d419c530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8011670 tx=0x7fc3c8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.021+0000 7fc3d2000700 1 -- 172.21.15.16:0/1677683612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3c8003af0 con 0x7fc3d407adb0 2024-04-05T23:26:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.021+0000 7fc3d2000700 1 -- 172.21.15.16:0/1677683612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c8003c50 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.021+0000 7fc3d2000700 1 -- 172.21.15.16:0/1677683612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3c8024820 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.021+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d419cc70 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.021+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d419d110 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.022+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d410c910 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.023+0000 7fc3d2000700 1 -- 172.21.15.16:0/1677683612 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45541+0+0 (secure 0 0 0) 0x7fc3c801ab60 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.023+0000 7fc3d2000700 1 --2- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc3bc038750 0x7fc3bc03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.023+0000 7fc3d2000700 1 -- 172.21.15.16:0/1677683612 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc3c804c7d0 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.024+0000 7fc3d8e00700 1 --2- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc3bc038750 0x7fc3bc03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.024+0000 7fc3d8e00700 1 --2- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc3bc038750 0x7fc3bc03ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0006fd0 tx=0x7fc3c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.028+0000 7fc3d2000700 1 -- 172.21.15.16:0/1677683612 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc3c8018760 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.252+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fc3d419f9a0 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.256+0000 7fc3d2000700 1 -- 172.21.15.16:0/1677683612 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fc3c800b3e0 con 0x7fc3d407adb0 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc3bc038750 msgr2=0x7fc3bc03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 --2- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc3bc038750 0x7fc3bc03ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0006fd0 tx=0x7fc3c0006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 msgr2=0x7fc3d419c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:26.630 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 --2- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d419c530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8011670 tx=0x7fc3c8011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:26.630 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 shutdown_connections 2024-04-05T23:26:26.630 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 --2- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7fc3bc038750 0x7fc3bc03ac10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:26.630 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 --2- 172.21.15.16:0/1677683612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d407adb0 0x7fc3d419c530 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:26.630 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 >> 172.21.15.16:0/1677683612 conn(0x7fc3d4101ad0 msgr2=0x7fc3d41027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:26.630 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 shutdown_connections 2024-04-05T23:26:26.630 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:26.259+0000 7fc3dac00700 1 -- 172.21.15.16:0/1677683612 wait complete. 2024-04-05T23:26:26.630 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-04-05T23:26:27.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:27 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1677683612' entity='client.admin' 2024-04-05T23:26:27.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:27 smithi016 ceph-mon[38248]: from='mgr.14120 172.21.15.16:0/686731546' entity='mgr.x' 2024-04-05T23:26:28.506 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:28 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1636901160' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-05T23:26:28.506 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: ignoring --setuser ceph since I am not root 2024-04-05T23:26:28.506 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: ignoring --setgroup ceph since I am not root 2024-04-05T23:26:28.638 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.046+0000 7f9260e00700 1 Processor -- start 2024-04-05T23:26:28.638 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.046+0000 7f9260e00700 1 -- start start 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.046+0000 7f9260e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c107350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.046+0000 7f9260e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c0746e0 con 0x7f925c104f20 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.047+0000 7f9259e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.047+0000 7f9259e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48118/0 (socket says 172.21.15.16:48118) 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.047+0000 7f9259e00700 1 -- 172.21.15.16:0/3991292339 learned_addr learned my addr 172.21.15.16:0/3991292339 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.047+0000 7f9259e00700 1 -- 172.21.15.16:0/3991292339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c107890 con 0x7f925c104f20 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.048+0000 7f9259e00700 1 --2- 172.21.15.16:0/3991292339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c107350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f924c0098a0 tx=0x7f924c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f377b544a3e91d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.048+0000 7f9258a00700 1 -- 172.21.15.16:0/3991292339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f924c004160 con 0x7f925c104f20 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.048+0000 7f9258a00700 1 -- 172.21.15.16:0/3991292339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f924c0042c0 con 0x7f925c104f20 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.048+0000 7f9258a00700 1 -- 172.21.15.16:0/3991292339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f924c0119b0 con 0x7f925c104f20 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.048+0000 7f9260e00700 1 -- 172.21.15.16:0/3991292339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 msgr2=0x7f925c107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.048+0000 7f9260e00700 1 --2- 172.21.15.16:0/3991292339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c107350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f924c0098a0 tx=0x7f924c009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:28.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.049+0000 7f9260e00700 1 -- 172.21.15.16:0/3991292339 shutdown_connections 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.049+0000 7f9260e00700 1 --2- 172.21.15.16:0/3991292339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c107350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.049+0000 7f9260e00700 1 -- 172.21.15.16:0/3991292339 >> 172.21.15.16:0/3991292339 conn(0x7f925c100b60 msgr2=0x7f925c102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.049+0000 7f9260e00700 1 -- 172.21.15.16:0/3991292339 shutdown_connections 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.049+0000 7f9260e00700 1 -- 172.21.15.16:0/3991292339 wait complete. 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.050+0000 7f9260e00700 1 Processor -- start 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.050+0000 7f9260e00700 1 -- start start 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.050+0000 7f9260e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c1a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.050+0000 7f9260e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c1a1280 con 0x7f925c104f20 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.050+0000 7f9259e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c1a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.050+0000 7f9259e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c1a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48120/0 (socket says 172.21.15.16:48120) 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.050+0000 7f9259e00700 1 -- 172.21.15.16:0/1636901160 learned_addr learned my addr 172.21.15.16:0/1636901160 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.051+0000 7f9259e00700 1 -- 172.21.15.16:0/1636901160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c009550 con 0x7f925c104f20 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.051+0000 7f9259e00700 1 --2- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c1a0d40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f924c011670 tx=0x7f924c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.051+0000 7f9252a00700 1 -- 172.21.15.16:0/1636901160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f924c003b20 con 0x7f925c104f20 2024-04-05T23:26:28.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.051+0000 7f9252a00700 1 -- 172.21.15.16:0/1636901160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f924c003c80 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.051+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c1a1480 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.051+0000 7f9252a00700 1 -- 172.21.15.16:0/1636901160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f924c019770 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.052+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c1a1920 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.053+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c067190 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.053+0000 7f9252a00700 1 -- 172.21.15.16:0/1636901160 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 45541+0+0 (secure 0 0 0) 0x7f924c01ab60 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.054+0000 7f9252a00700 1 --2- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9240038750 0x7f924003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.054+0000 7f9252a00700 1 -- 172.21.15.16:0/1636901160 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f924c04c940 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.054+0000 7f9259400700 1 --2- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9240038750 0x7f924003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.055+0000 7f9259400700 1 --2- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9240038750 0x7f924003ac10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9248006fd0 tx=0x7f9248006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.059+0000 7f9252a00700 1 -- 172.21.15.16:0/1636901160 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f924c01a430 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:27.360+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f925c108740 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.260+0000 7f9252a00700 1 -- 172.21.15.16:0/1636901160 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 9) v1 ==== 45554+0+0 (secure 0 0 0) 0x7f924c02d350 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.263+0000 7f9252a00700 1 -- 172.21.15.16:0/1636901160 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f924c017720 con 0x7f925c104f20 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.266+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9240038750 msgr2=0x7f924003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:28.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.267+0000 7f9260e00700 1 --2- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9240038750 0x7f924003ac10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9248006fd0 tx=0x7f9248006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:28.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.267+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 msgr2=0x7f925c1a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:28.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.267+0000 7f9260e00700 1 --2- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c1a0d40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f924c011670 tx=0x7f924c011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:28.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.267+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 shutdown_connections 2024-04-05T23:26:28.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.267+0000 7f9260e00700 1 --2- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9240038750 0x7f924003ac10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:28.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.267+0000 7f9260e00700 1 --2- 172.21.15.16:0/1636901160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f925c104f20 0x7f925c1a0d40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:28.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.267+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 >> 172.21.15.16:0/1636901160 conn(0x7f925c100b60 msgr2=0x7f925c102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:28.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.267+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 shutdown_connections 2024-04-05T23:26:28.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:28.267+0000 7f9260e00700 1 -- 172.21.15.16:0/1636901160 wait complete. 2024-04-05T23:26:29.034 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:28.915+0000 7fdda4ae2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-05T23:26:29.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:29 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1636901160' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-05T23:26:29.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:29 smithi016 ceph-mon[38248]: mgrmap e9: x(active, since 16s) 2024-04-05T23:26:29.333 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:29.040+0000 7fdda4ae2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-05T23:26:29.333 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:29.162+0000 7fdda4ae2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-05T23:26:29.334 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:29.276+0000 7fdda4ae2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-05T23:26:29.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.084+0000 7ff4ae400700 1 Processor -- start 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.084+0000 7ff4ae400700 1 -- start start 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.084+0000 7ff4ae400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a8106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.084+0000 7ff4ae400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a81070b0 con 0x7ff4a81066c0 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.085+0000 7ff4a7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a8106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.085+0000 7ff4a7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a8106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48156/0 (socket says 172.21.15.16:48156) 2024-04-05T23:26:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.085+0000 7ff4a7400700 1 -- 172.21.15.16:0/3829548905 learned_addr learned my addr 172.21.15.16:0/3829548905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.085+0000 7ff4a7400700 1 -- 172.21.15.16:0/3829548905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a81078c0 con 0x7ff4a81066c0 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.085+0000 7ff4a7400700 1 --2- 172.21.15.16:0/3829548905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a8106ae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4980098a0 tx=0x7ff498009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b04de69cd62f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.086+0000 7ff4a6000700 1 -- 172.21.15.16:0/3829548905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff498004160 con 0x7ff4a81066c0 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.086+0000 7ff4a6000700 1 -- 172.21.15.16:0/3829548905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4980042c0 con 0x7ff4a81066c0 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.086+0000 7ff4a6000700 1 -- 172.21.15.16:0/3829548905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4980119b0 con 0x7ff4a81066c0 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.086+0000 7ff4ae400700 1 -- 172.21.15.16:0/3829548905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 msgr2=0x7ff4a8106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.086+0000 7ff4ae400700 1 --2- 172.21.15.16:0/3829548905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a8106ae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4980098a0 tx=0x7ff498009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.086+0000 7ff4ae400700 1 -- 172.21.15.16:0/3829548905 shutdown_connections 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.086+0000 7ff4ae400700 1 --2- 172.21.15.16:0/3829548905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a8106ae0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.086+0000 7ff4ae400700 1 -- 172.21.15.16:0/3829548905 >> 172.21.15.16:0/3829548905 conn(0x7ff4a8101c60 msgr2=0x7ff4a81040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.087+0000 7ff4ae400700 1 -- 172.21.15.16:0/3829548905 shutdown_connections 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.087+0000 7ff4ae400700 1 -- 172.21.15.16:0/3829548905 wait complete. 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.087+0000 7ff4ae400700 1 Processor -- start 2024-04-05T23:26:29.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.088+0000 7ff4ae400700 1 -- start start 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.088+0000 7ff4ae400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a807aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.088+0000 7ff4ae400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a807af70 con 0x7ff4a81066c0 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.088+0000 7ff4a7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a807aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.088+0000 7ff4a7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a807aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48160/0 (socket says 172.21.15.16:48160) 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.088+0000 7ff4a7400700 1 -- 172.21.15.16:0/336935527 learned_addr learned my addr 172.21.15.16:0/336935527 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.089+0000 7ff4a7400700 1 -- 172.21.15.16:0/336935527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff498009550 con 0x7ff4a81066c0 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.089+0000 7ff4a7400700 1 --2- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a807aa30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff498003e60 tx=0x7ff498011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.089+0000 7ff49fe00700 1 -- 172.21.15.16:0/336935527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff498003a60 con 0x7ff4a81066c0 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.089+0000 7ff49fe00700 1 -- 172.21.15.16:0/336935527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff498003bc0 con 0x7ff4a81066c0 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.089+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a80791b0 con 0x7ff4a81066c0 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.089+0000 7ff49fe00700 1 -- 172.21.15.16:0/336935527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4980196c0 con 0x7ff4a81066c0 2024-04-05T23:26:29.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.089+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a8079650 con 0x7ff4a81066c0 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.090+0000 7ff49fe00700 1 -- 172.21.15.16:0/336935527 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 45554+0+0 (secure 0 0 0) 0x7ff49801a430 con 0x7ff4a81066c0 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.091+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a810aaf0 con 0x7ff4a81066c0 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.091+0000 7ff49fe00700 1 --2- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7ff48c038730 0x7ff48c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.091+0000 7ff49fe00700 1 -- 172.21.15.16:0/336935527 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff49804ca40 con 0x7ff4a81066c0 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.091+0000 7ff4a6a00700 1 -- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7ff48c038730 msgr2=0x7ff48c03abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2429779560 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.091+0000 7ff4a6a00700 1 --2- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7ff48c038730 0x7ff48c03abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.096+0000 7ff49fe00700 1 -- 172.21.15.16:0/336935527 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff498010560 con 0x7ff4a81066c0 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.292+0000 7ff4a6a00700 1 -- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7ff48c038730 msgr2=0x7ff48c03abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2429779560 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.292+0000 7ff4a6a00700 1 --2- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7ff48c038730 0x7ff48c03abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.367+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff4a8079d90 con 0x7ff4a81066c0 2024-04-05T23:26:29.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.367+0000 7ff49fe00700 1 -- 172.21.15.16:0/336935527 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7ff498010380 con 0x7ff4a81066c0 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7ff48c038730 msgr2=0x7ff48c03abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 --2- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7ff48c038730 0x7ff48c03abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 msgr2=0x7ff4a807aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 --2- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a807aa30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff498003e60 tx=0x7ff498011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 shutdown_connections 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 --2- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7ff48c038730 0x7ff48c03abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 --2- 172.21.15.16:0/336935527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4a81066c0 0x7ff4a807aa30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 >> 172.21.15.16:0/336935527 conn(0x7ff4a8101c60 msgr2=0x7ff4a81028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 shutdown_connections 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:29.370+0000 7ff4ae400700 1 -- 172.21.15.16:0/336935527 wait complete. 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-04-05T23:26:29.731 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-04-05T23:26:30.485 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:30 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/336935527' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-05T23:26:30.485 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:30.163+0000 7fdda4ae2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:26:30.743 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:30.491+0000 7fdda4ae2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-05T23:26:30.743 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:30.605+0000 7fdda4ae2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-05T23:26:30.743 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:30.749+0000 7fdda4ae2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-05T23:26:31.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:30.951+0000 7fdda4ae2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-05T23:26:31.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:31 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:31.125+0000 7fdda4ae2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-05T23:26:31.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:31 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:31.239+0000 7fdda4ae2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-05T23:26:32.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:31 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:31.719+0000 7fdda4ae2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-05T23:26:32.473 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:32.220+0000 7fdda4ae2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-05T23:26:32.834 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:32.479+0000 7fdda4ae2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-05T23:26:32.834 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:32.605+0000 7fdda4ae2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-05T23:26:33.914 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:33.591+0000 7fdda4ae2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-05T23:26:33.914 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:33.720+0000 7fdda4ae2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-05T23:26:33.914 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:33.920+0000 7fdda4ae2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-05T23:26:35.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:34.752+0000 7fdda4ae2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-05T23:26:35.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:35 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:34.999+0000 7fdda4ae2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-05T23:26:35.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:35 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:35.263+0000 7fdda4ae2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-05T23:26:36.830 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:36 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:36.448+0000 7fdda4ae2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-05T23:26:36.830 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:36 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:36.836+0000 7fdda4ae2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:26:37.083 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:36 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:36.959+0000 7fdda4ae2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-05T23:26:37.583 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:26:37.224+0000 7fdda4ae2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-05T23:26:37.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: Active manager daemon x restarted 2024-04-05T23:26:37.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: Activating manager daemon x 2024-04-05T23:26:37.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: osdmap e3: 0 total, 0 up, 0 in 2024-04-05T23:26:37.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: mgrmap e10: x(active, starting, since 0.00422822s) 2024-04-05T23:26:37.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:26:37.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-05T23:26:37.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T23:26:37.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T23:26:37.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T23:26:37.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: Manager daemon x is now available 2024-04-05T23:26:37.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:26:38.518 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-05T23:26:38.518 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-05T23:26:38.518 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:38 smithi016 ceph-mon[38248]: mgrmap e11: x(active, since 1.00802s) 2024-04-05T23:26:38.665 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-05T23:26:38.665 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-05T23:26:38.665 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-05T23:26:38.665 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-05T23:26:38.665 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.200+0000 7f97b4800700 1 Processor -- start 2024-04-05T23:26:38.665 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.201+0000 7f97b4800700 1 -- start start 2024-04-05T23:26:38.665 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.201+0000 7f97b4800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:38.666 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.201+0000 7f97b4800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac0797e0 con 0x7f97ac07adb0 2024-04-05T23:26:38.666 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.201+0000 7f97b1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:38.666 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.201+0000 7f97b1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48162/0 (socket says 172.21.15.16:48162) 2024-04-05T23:26:38.666 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.201+0000 7f97b1e00700 1 -- 172.21.15.16:0/2150255714 learned_addr learned my addr 172.21.15.16:0/2150255714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:38.666 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.202+0000 7f97b1e00700 1 -- 172.21.15.16:0/2150255714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac079920 con 0x7f97ac07adb0 2024-04-05T23:26:38.666 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.202+0000 7f97b1e00700 1 --2- 172.21.15.16:0/2150255714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac079210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f97a00098a0 tx=0x7f97a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09a5956e858cf06 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:38.666 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.202+0000 7f97b0a00700 1 -- 172.21.15.16:0/2150255714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97a0004160 con 0x7f97ac07adb0 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.202+0000 7f97b0a00700 1 -- 172.21.15.16:0/2150255714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97a00042c0 con 0x7f97ac07adb0 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.202+0000 7f97b0a00700 1 -- 172.21.15.16:0/2150255714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97a00119b0 con 0x7f97ac07adb0 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.203+0000 7f97b4800700 1 -- 172.21.15.16:0/2150255714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 msgr2=0x7f97ac079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.203+0000 7f97b4800700 1 --2- 172.21.15.16:0/2150255714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac079210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f97a00098a0 tx=0x7f97a0009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.203+0000 7f97b4800700 1 -- 172.21.15.16:0/2150255714 shutdown_connections 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.203+0000 7f97b4800700 1 --2- 172.21.15.16:0/2150255714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac079210 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.203+0000 7f97b4800700 1 -- 172.21.15.16:0/2150255714 >> 172.21.15.16:0/2150255714 conn(0x7f97ac101c60 msgr2=0x7f97ac1040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.203+0000 7f97b4800700 1 -- 172.21.15.16:0/2150255714 shutdown_connections 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.203+0000 7f97b4800700 1 -- 172.21.15.16:0/2150255714 wait complete. 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.204+0000 7f97b4800700 1 Processor -- start 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.204+0000 7f97b4800700 1 -- start start 2024-04-05T23:26:38.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.205+0000 7f97b4800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.205+0000 7f97b4800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac19cbf0 con 0x7f97ac07adb0 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.205+0000 7f97b1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.205+0000 7f97b1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac19c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48164/0 (socket says 172.21.15.16:48164) 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.205+0000 7f97b1e00700 1 -- 172.21.15.16:0/2769059830 learned_addr learned my addr 172.21.15.16:0/2769059830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.205+0000 7f97b1e00700 1 -- 172.21.15.16:0/2769059830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a0009550 con 0x7f97ac07adb0 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.205+0000 7f97b1e00700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac19c6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97a0003e60 tx=0x7f97a0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.206+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97a0003a60 con 0x7f97ac07adb0 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.206+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97a0003bc0 con 0x7f97ac07adb0 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.206+0000 7f97b4800700 1 -- 172.21.15.16:0/2769059830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ac19cdf0 con 0x7f97ac07adb0 2024-04-05T23:26:38.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.206+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97a00196c0 con 0x7f97ac07adb0 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.206+0000 7f97b4800700 1 -- 172.21.15.16:0/2769059830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ac19d290 con 0x7f97ac07adb0 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.207+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 45554+0+0 (secure 0 0 0) 0x7f97a0019930 con 0x7f97ac07adb0 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.207+0000 7f97a6a00700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 0x7f979403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.207+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 --> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f979403b350 con 0x7f9794038780 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.207+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f97a004ca80 con 0x7f97ac07adb0 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.207+0000 7f97b1400700 1 -- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 msgr2=0x7f979403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2429779560 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.208+0000 7f97b1400700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 0x7f979403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.409+0000 7f97b1400700 1 -- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 msgr2=0x7f979403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2429779560 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.409+0000 7f97b1400700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 0x7f979403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T23:26:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.810+0000 7f97b1400700 1 -- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 msgr2=0x7f979403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2429779560 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:30.810+0000 7f97b1400700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 0x7f979403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:31.611+0000 7f97b1400700 1 -- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 msgr2=0x7f979403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2429779560 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:31.611+0000 7f97b1400700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 0x7f979403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:33.212+0000 7f97b1400700 1 -- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 msgr2=0x7f979403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2429779560 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:33.212+0000 7f97b1400700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 0x7f979403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:36.415+0000 7f97b1400700 1 -- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 msgr2=0x7f979403ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2429779560 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:36.415+0000 7f97b1400700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 0x7f979403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:37.229+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 10) v1 ==== 45321+0+0 (secure 0 0 0) 0x7f97a002ca00 con 0x7f97ac07adb0 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:37.229+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 msgr2=0x7f979403ac40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:37.229+0000 7f97a6a00700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 0x7f979403ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.231+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 11) v1 ==== 45448+0+0 (secure 0 0 0) 0x7f97a004e5c0 con 0x7f97ac07adb0 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.231+0000 7f97a6a00700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f979403bff0 0x7f979403e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:38.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.232+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f979403b350 con 0x7f979403bff0 2024-04-05T23:26:38.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.232+0000 7f97b1400700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f979403bff0 0x7f979403e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:38.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.233+0000 7f97b1400700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f979403bff0 0x7f979403e3e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f979c003a10 tx=0x7f979c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:38.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.234+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f979403b350 con 0x7f979403bff0 2024-04-05T23:26:38.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.241+0000 7f97b4800700 1 -- 172.21.15.16:0/2769059830 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f97ac10cae0 con 0x7f979403bff0 2024-04-05T23:26:38.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.241+0000 7f97a6a00700 1 -- 172.21.15.16:0/2769059830 <== mgr.14152 v2:172.21.15.16:6800/1565223355 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f97ac10cae0 con 0x7f979403bff0 2024-04-05T23:26:38.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.241+0000 7f97b4800700 1 -- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f979403bff0 msgr2=0x7f979403e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:38.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.241+0000 7f97b4800700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f979403bff0 0x7f979403e3e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f979c003a10 tx=0x7f979c009270 comp rx=0 tx=0).stop 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.241+0000 7f97b4800700 1 -- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 msgr2=0x7f97ac19c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.241+0000 7f97b4800700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac19c6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97a0003e60 tx=0x7f97a0011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.242+0000 7f97b4800700 1 -- 172.21.15.16:0/2769059830 shutdown_connections 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.242+0000 7f97b4800700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97ac07adb0 0x7f97ac19c6b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.242+0000 7f97b4800700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f979403bff0 0x7f979403e3e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.242+0000 7f97b4800700 1 --2- 172.21.15.16:0/2769059830 >> [v2:172.21.15.16:6800/2429779560,v1:172.21.15.16:6801/2429779560] conn(0x7f9794038780 0x7f979403ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.242+0000 7f97b4800700 1 -- 172.21.15.16:0/2769059830 >> 172.21.15.16:0/2769059830 conn(0x7f97ac101c60 msgr2=0x7f97ac1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.242+0000 7f97b4800700 1 -- 172.21.15.16:0/2769059830 shutdown_connections 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:38.242+0000 7f97b4800700 1 -- 172.21.15.16:0/2769059830 wait complete. 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-04-05T23:26:38.672 INFO:teuthology.orchestra.run.smithi016.stdout:Generating a dashboard self-signed certificate... 2024-04-05T23:26:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:39 smithi016 ceph-mon[38248]: [05/Apr/2024:23:26:37] ENGINE Bus STARTING 2024-04-05T23:26:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:39 smithi016 ceph-mon[38248]: [05/Apr/2024:23:26:38] ENGINE Serving on http://172.21.15.16:8765 2024-04-05T23:26:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:39 smithi016 ceph-mon[38248]: [05/Apr/2024:23:26:38] ENGINE Serving on https://172.21.15.16:7150 2024-04-05T23:26:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:39 smithi016 ceph-mon[38248]: [05/Apr/2024:23:26:38] ENGINE Bus STARTED 2024-04-05T23:26:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:39 smithi016 ceph-mon[38248]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-05T23:26:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:39 smithi016 ceph-mon[38248]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-05T23:26:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:39 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:39.951 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-05T23:26:39.951 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.155+0000 7fd957200700 1 Processor -- start 2024-04-05T23:26:39.951 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.155+0000 7fd957200700 1 -- start start 2024-04-05T23:26:39.951 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.156+0000 7fd957200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd958108b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:39.951 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.156+0000 7fd957200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd958109100 con 0x7fd958108710 2024-04-05T23:26:39.951 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.156+0000 7fd955e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd958108b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:39.951 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.156+0000 7fd955e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd958108b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50948/0 (socket says 172.21.15.16:50948) 2024-04-05T23:26:39.951 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.156+0000 7fd955e00700 1 -- 172.21.15.16:0/3929977115 learned_addr learned my addr 172.21.15.16:0/3929977115 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.156+0000 7fd955e00700 1 -- 172.21.15.16:0/3929977115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd958109910 con 0x7fd958108710 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.157+0000 7fd955e00700 1 --2- 172.21.15.16:0/3929977115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd958108b30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd9480098a0 tx=0x7fd948009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4789163ed403b98a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.157+0000 7fd954a00700 1 -- 172.21.15.16:0/3929977115 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd948004160 con 0x7fd958108710 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.157+0000 7fd954a00700 1 -- 172.21.15.16:0/3929977115 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9480042c0 con 0x7fd958108710 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.157+0000 7fd954a00700 1 -- 172.21.15.16:0/3929977115 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9480119b0 con 0x7fd958108710 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.158+0000 7fd957200700 1 -- 172.21.15.16:0/3929977115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 msgr2=0x7fd958108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.158+0000 7fd957200700 1 --2- 172.21.15.16:0/3929977115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd958108b30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd9480098a0 tx=0x7fd948009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.158+0000 7fd957200700 1 -- 172.21.15.16:0/3929977115 shutdown_connections 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.158+0000 7fd957200700 1 --2- 172.21.15.16:0/3929977115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd958108b30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.158+0000 7fd957200700 1 -- 172.21.15.16:0/3929977115 >> 172.21.15.16:0/3929977115 conn(0x7fd95807bc00 msgr2=0x7fd9581064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.158+0000 7fd957200700 1 -- 172.21.15.16:0/3929977115 shutdown_connections 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.158+0000 7fd957200700 1 -- 172.21.15.16:0/3929977115 wait complete. 2024-04-05T23:26:39.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.159+0000 7fd957200700 1 Processor -- start 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.159+0000 7fd957200700 1 -- start start 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.159+0000 7fd957200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd95819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.159+0000 7fd957200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95819ca30 con 0x7fd958108710 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.159+0000 7fd955e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd95819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.159+0000 7fd955e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd95819c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50950/0 (socket says 172.21.15.16:50950) 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.159+0000 7fd955e00700 1 -- 172.21.15.16:0/2702371983 learned_addr learned my addr 172.21.15.16:0/2702371983 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.160+0000 7fd955e00700 1 -- 172.21.15.16:0/2702371983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd948009550 con 0x7fd958108710 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.160+0000 7fd955e00700 1 --2- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd95819c4f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd948011670 tx=0x7fd948011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.160+0000 7fd94ea00700 1 -- 172.21.15.16:0/2702371983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd948003af0 con 0x7fd958108710 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.160+0000 7fd94ea00700 1 -- 172.21.15.16:0/2702371983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd948003c50 con 0x7fd958108710 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.160+0000 7fd94ea00700 1 -- 172.21.15.16:0/2702371983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd948024820 con 0x7fd958108710 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.160+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95819cc30 con 0x7fd958108710 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.161+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95819d0d0 con 0x7fd958108710 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.162+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95810cb40 con 0x7fd958108710 2024-04-05T23:26:39.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.162+0000 7fd94ea00700 1 -- 172.21.15.16:0/2702371983 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 11) v1 ==== 45448+0+0 (secure 0 0 0) 0x7fd94801ab60 con 0x7fd958108710 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.162+0000 7fd94ea00700 1 --2- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd93c038670 0x7fd93c03ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.162+0000 7fd94ea00700 1 -- 172.21.15.16:0/2702371983 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd94804c6b0 con 0x7fd958108710 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.162+0000 7fd955400700 1 --2- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd93c038670 0x7fd93c03ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.163+0000 7fd955400700 1 --2- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd93c038670 0x7fd93c03ab30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd944006fd0 tx=0x7fd944006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.167+0000 7fd94ea00700 1 -- 172.21.15.16:0/2702371983 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd948018760 con 0x7fd958108710 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.410+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd95819f960 con 0x7fd93c038670 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.589+0000 7fd94ea00700 1 -- 172.21.15.16:0/2702371983 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd95819f960 con 0x7fd93c038670 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd93c038670 msgr2=0x7fd93c03ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 --2- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd93c038670 0x7fd93c03ab30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd944006fd0 tx=0x7fd944006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 msgr2=0x7fd95819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 --2- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd95819c4f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd948011670 tx=0x7fd948011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 shutdown_connections 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 --2- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd93c038670 0x7fd93c03ab30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 --2- 172.21.15.16:0/2702371983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd958108710 0x7fd95819c4f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:39.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 >> 172.21.15.16:0/2702371983 conn(0x7fd95807bc00 msgr2=0x7fd958105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:39.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 shutdown_connections 2024-04-05T23:26:39.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:39.592+0000 7fd957200700 1 -- 172.21.15.16:0/2702371983 wait complete. 2024-04-05T23:26:39.955 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-04-05T23:26:40.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:40 smithi016 ceph-mon[38248]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:40.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:40.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:40.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:40 smithi016 ceph-mon[38248]: mgrmap e12: x(active, since 2s) 2024-04-05T23:26:40.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SjpSykCfV3F6H7XzAmWqUOJcC6lU5N8PbYvB4YTJv0c1q8aB5nqVK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712359600, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-05T23:26:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.427+0000 7f1cf8600700 1 Processor -- start 2024-04-05T23:26:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.428+0000 7f1cf8600700 1 -- start start 2024-04-05T23:26:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.428+0000 7f1cf8600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf0106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.428+0000 7f1cf8600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf01070b0 con 0x7f1cf01066c0 2024-04-05T23:26:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.428+0000 7f1cf5c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf0106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.429+0000 7f1cf5c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf0106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50960/0 (socket says 172.21.15.16:50960) 2024-04-05T23:26:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.429+0000 7f1cf5c00700 1 -- 172.21.15.16:0/2060621791 learned_addr learned my addr 172.21.15.16:0/2060621791 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.429+0000 7f1cf5c00700 1 -- 172.21.15.16:0/2060621791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf01078c0 con 0x7f1cf01066c0 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.429+0000 7f1cf5c00700 1 --2- 172.21.15.16:0/2060621791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf0106ae0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40098a0 tx=0x7f1ce4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4c653071ded736 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.429+0000 7f1cefe00700 1 -- 172.21.15.16:0/2060621791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ce4004160 con 0x7f1cf01066c0 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.429+0000 7f1cefe00700 1 -- 172.21.15.16:0/2060621791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ce40042c0 con 0x7f1cf01066c0 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.430+0000 7f1cefe00700 1 -- 172.21.15.16:0/2060621791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ce40119b0 con 0x7f1cf01066c0 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.430+0000 7f1cf8600700 1 -- 172.21.15.16:0/2060621791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 msgr2=0x7f1cf0106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.430+0000 7f1cf8600700 1 --2- 172.21.15.16:0/2060621791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf0106ae0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40098a0 tx=0x7f1ce4009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.431+0000 7f1cf8600700 1 -- 172.21.15.16:0/2060621791 shutdown_connections 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.431+0000 7f1cf8600700 1 --2- 172.21.15.16:0/2060621791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf0106ae0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.431+0000 7f1cf8600700 1 -- 172.21.15.16:0/2060621791 >> 172.21.15.16:0/2060621791 conn(0x7f1cf0101c60 msgr2=0x7f1cf01040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.431+0000 7f1cf8600700 1 -- 172.21.15.16:0/2060621791 shutdown_connections 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.431+0000 7f1cf8600700 1 -- 172.21.15.16:0/2060621791 wait complete. 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.432+0000 7f1cf8600700 1 Processor -- start 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.432+0000 7f1cf8600700 1 -- start start 2024-04-05T23:26:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.432+0000 7f1cf8600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf007aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.432+0000 7f1cf8600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf007af70 con 0x7f1cf01066c0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.433+0000 7f1cf5c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf007aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.433+0000 7f1cf5c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf007aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50968/0 (socket says 172.21.15.16:50968) 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.433+0000 7f1cf5c00700 1 -- 172.21.15.16:0/3767992420 learned_addr learned my addr 172.21.15.16:0/3767992420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.433+0000 7f1cf5c00700 1 -- 172.21.15.16:0/3767992420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce4009550 con 0x7f1cf01066c0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.433+0000 7f1cf5c00700 1 --2- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf007aa30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4003e60 tx=0x7f1ce4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.434+0000 7f1cee000700 1 -- 172.21.15.16:0/3767992420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ce4003a60 con 0x7f1cf01066c0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.434+0000 7f1cee000700 1 -- 172.21.15.16:0/3767992420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ce4003bc0 con 0x7f1cf01066c0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.434+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf00791b0 con 0x7f1cf01066c0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.434+0000 7f1cee000700 1 -- 172.21.15.16:0/3767992420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ce40196c0 con 0x7f1cf01066c0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.434+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf0079650 con 0x7f1cf01066c0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.435+0000 7f1cee000700 1 -- 172.21.15.16:0/3767992420 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 45554+0+0 (secure 0 0 0) 0x7f1ce4019930 con 0x7f1cf01066c0 2024-04-05T23:26:41.548 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.435+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf010aaf0 con 0x7f1cf01066c0 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.435+0000 7f1cee000700 1 --2- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1cd8038780 0x7f1cd803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.435+0000 7f1cee000700 1 -- 172.21.15.16:0/3767992420 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1ce404cc60 con 0x7f1cf01066c0 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.436+0000 7f1cf5200700 1 --2- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1cd8038780 0x7f1cd803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.436+0000 7f1cf5200700 1 --2- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1cd8038780 0x7f1cd803ac40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc006fd0 tx=0x7f1cdc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.441+0000 7f1cee000700 1 -- 172.21.15.16:0/3767992420 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1ce4010560 con 0x7f1cf01066c0 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.685+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1cf0062330 con 0x7f1cd8038780 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.932+0000 7f1cee000700 1 -- 172.21.15.16:0/3767992420 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1cf0062330 con 0x7f1cd8038780 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1cd8038780 msgr2=0x7f1cd803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 --2- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1cd8038780 0x7f1cd803ac40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc006fd0 tx=0x7f1cdc006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 msgr2=0x7f1cf007aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 --2- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf007aa30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4003e60 tx=0x7f1ce4011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 shutdown_connections 2024-04-05T23:26:41.549 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 --2- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1cd8038780 0x7f1cd803ac40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:41.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 --2- 172.21.15.16:0/3767992420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cf01066c0 0x7f1cf007aa30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:41.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 >> 172.21.15.16:0/3767992420 conn(0x7f1cf0101c60 msgr2=0x7f1cf0102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:41.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 shutdown_connections 2024-04-05T23:26:41.550 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:40.935+0000 7f1cf8600700 1 -- 172.21.15.16:0/3767992420 wait complete. 2024-04-05T23:26:41.550 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-04-05T23:26:42.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:41 smithi016 ceph-mon[38248]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:42.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:41 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:42.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-04-05T23:26:42.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.958+0000 7f0bac600700 1 Processor -- start 2024-04-05T23:26:42.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.959+0000 7f0bac600700 1 -- start start 2024-04-05T23:26:42.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.959+0000 7f0bac600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba4108d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:42.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.959+0000 7f0bac600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba4109310 con 0x7f0ba4108920 2024-04-05T23:26:42.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.959+0000 7f0ba9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba4108d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:42.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.959+0000 7f0ba9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba4108d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50978/0 (socket says 172.21.15.16:50978) 2024-04-05T23:26:42.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.959+0000 7f0ba9c00700 1 -- 172.21.15.16:0/475079974 learned_addr learned my addr 172.21.15.16:0/475079974 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:42.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.960+0000 7f0ba9c00700 1 -- 172.21.15.16:0/475079974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba4109ad0 con 0x7f0ba4108920 2024-04-05T23:26:42.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.960+0000 7f0ba9c00700 1 --2- 172.21.15.16:0/475079974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba4108d40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0b980098a0 tx=0x7f0b98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531a8aa59aa0202e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:42.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.960+0000 7f0ba3e00700 1 -- 172.21.15.16:0/475079974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98004160 con 0x7f0ba4108920 2024-04-05T23:26:42.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.960+0000 7f0ba3e00700 1 -- 172.21.15.16:0/475079974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b980042c0 con 0x7f0ba4108920 2024-04-05T23:26:42.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.960+0000 7f0ba3e00700 1 -- 172.21.15.16:0/475079974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b980119b0 con 0x7f0ba4108920 2024-04-05T23:26:42.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.961+0000 7f0bac600700 1 -- 172.21.15.16:0/475079974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 msgr2=0x7f0ba4108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:42.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.961+0000 7f0bac600700 1 --2- 172.21.15.16:0/475079974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba4108d40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0b980098a0 tx=0x7f0b98009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:42.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.961+0000 7f0bac600700 1 -- 172.21.15.16:0/475079974 shutdown_connections 2024-04-05T23:26:42.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.961+0000 7f0bac600700 1 --2- 172.21.15.16:0/475079974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba4108d40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:42.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.961+0000 7f0bac600700 1 -- 172.21.15.16:0/475079974 >> 172.21.15.16:0/475079974 conn(0x7f0ba407bdd0 msgr2=0x7f0ba41064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:42.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.961+0000 7f0bac600700 1 -- 172.21.15.16:0/475079974 shutdown_connections 2024-04-05T23:26:42.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.961+0000 7f0bac600700 1 -- 172.21.15.16:0/475079974 wait complete. 2024-04-05T23:26:42.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.962+0000 7f0bac600700 1 Processor -- start 2024-04-05T23:26:42.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.962+0000 7f0bac600700 1 -- start start 2024-04-05T23:26:42.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.962+0000 7f0bac600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:42.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.962+0000 7f0bac600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba419cbf0 con 0x7f0ba4108920 2024-04-05T23:26:42.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.963+0000 7f0ba9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:42.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.963+0000 7f0ba9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50986/0 (socket says 172.21.15.16:50986) 2024-04-05T23:26:42.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.963+0000 7f0ba9c00700 1 -- 172.21.15.16:0/498840505 learned_addr learned my addr 172.21.15.16:0/498840505 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:42.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.963+0000 7f0ba9c00700 1 -- 172.21.15.16:0/498840505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98009550 con 0x7f0ba4108920 2024-04-05T23:26:42.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.963+0000 7f0ba9c00700 1 --2- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba419c6b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b98011670 tx=0x7f0b98011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:42.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.963+0000 7f0ba2000700 1 -- 172.21.15.16:0/498840505 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98003af0 con 0x7f0ba4108920 2024-04-05T23:26:42.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.964+0000 7f0ba2000700 1 -- 172.21.15.16:0/498840505 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b98003c50 con 0x7f0ba4108920 2024-04-05T23:26:42.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.964+0000 7f0bac600700 1 -- 172.21.15.16:0/498840505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba419cdf0 con 0x7f0ba4108920 2024-04-05T23:26:42.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.964+0000 7f0ba2000700 1 -- 172.21.15.16:0/498840505 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98024820 con 0x7f0ba4108920 2024-04-05T23:26:42.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.964+0000 7f0bac600700 1 -- 172.21.15.16:0/498840505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba419d290 con 0x7f0ba4108920 2024-04-05T23:26:42.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.965+0000 7f0ba2000700 1 -- 172.21.15.16:0/498840505 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 45554+0+0 (secure 0 0 0) 0x7f0b9801a430 con 0x7f0ba4108920 2024-04-05T23:26:42.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.965+0000 7f0b97400700 1 -- 172.21.15.16:0/498840505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b800052f0 con 0x7f0ba4108920 2024-04-05T23:26:42.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.966+0000 7f0ba2000700 1 --2- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b8c038750 0x7f0b8c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:42.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.966+0000 7f0ba2000700 1 -- 172.21.15.16:0/498840505 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0b9804c8a0 con 0x7f0ba4108920 2024-04-05T23:26:42.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.966+0000 7f0ba9200700 1 --2- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b8c038750 0x7f0b8c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:42.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.967+0000 7f0ba9200700 1 --2- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b8c038750 0x7f0b8c03ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0b90006fd0 tx=0x7f0b90006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:42.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:41.971+0000 7f0ba2000700 1 -- 172.21.15.16:0/498840505 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0b98018920 con 0x7f0ba4108920 2024-04-05T23:26:42.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.183+0000 7f0b97400700 1 -- 172.21.15.16:0/498840505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f0b80005b20 con 0x7f0ba4108920 2024-04-05T23:26:42.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.183+0000 7f0ba2000700 1 -- 172.21.15.16:0/498840505 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0b98018350 con 0x7f0ba4108920 2024-04-05T23:26:42.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.185+0000 7f0b97400700 1 -- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b8c038750 msgr2=0x7f0b8c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:42.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.185+0000 7f0b97400700 1 --2- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b8c038750 0x7f0b8c03ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0b90006fd0 tx=0x7f0b90006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:42.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.185+0000 7f0b97400700 1 -- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 msgr2=0x7f0ba419c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:42.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.185+0000 7f0b97400700 1 --2- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba419c6b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b98011670 tx=0x7f0b98011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:42.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.186+0000 7f0b97400700 1 -- 172.21.15.16:0/498840505 shutdown_connections 2024-04-05T23:26:42.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.186+0000 7f0b97400700 1 --2- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b8c038750 0x7f0b8c03ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:42.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.186+0000 7f0b97400700 1 --2- 172.21.15.16:0/498840505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba4108920 0x7f0ba419c6b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:42.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.186+0000 7f0b97400700 1 -- 172.21.15.16:0/498840505 >> 172.21.15.16:0/498840505 conn(0x7f0ba407bdd0 msgr2=0x7f0ba4105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:42.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.186+0000 7f0b97400700 1 -- 172.21.15.16:0/498840505 shutdown_connections 2024-04-05T23:26:42.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:42.186+0000 7f0b97400700 1 -- 172.21.15.16:0/498840505 wait complete. 2024-04-05T23:26:42.580 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T23:26:42.580 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-05T23:26:42.596 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T23:26:42.596 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-05T23:26:42.596 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-05T23:26:42.596 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-05T23:26:42.598 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-04-05T23:26:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016.front.sepia.ceph.com:8443/ 2024-04-05T23:26:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-04-05T23:26:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: Password: scrwywwteq 2024-04-05T23:26:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:42.599 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config directory 2024-04-05T23:26:42.599 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling autotune for osd_memory_target 2024-04-05T23:26:43.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:42 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/498840505' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-05T23:26:43.639 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.059+0000 7f2e25c00700 1 Processor -- start 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.060+0000 7f2e25c00700 1 -- start start 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.060+0000 7f2e25c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e20106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.060+0000 7f2e25c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e201070b0 con 0x7f2e201066c0 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.061+0000 7f2e1ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e20106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.061+0000 7f2e1ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e20106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50046/0 (socket says 172.21.15.16:50046) 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.061+0000 7f2e1ea00700 1 -- 172.21.15.16:0/225733891 learned_addr learned my addr 172.21.15.16:0/225733891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.061+0000 7f2e1ea00700 1 -- 172.21.15.16:0/225733891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e201078c0 con 0x7f2e201066c0 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.061+0000 7f2e1ea00700 1 --2- 172.21.15.16:0/225733891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e20106ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e100098a0 tx=0x7f2e10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecefd2cf365f682 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.061+0000 7f2e1d600700 1 -- 172.21.15.16:0/225733891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e10004160 con 0x7f2e201066c0 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.061+0000 7f2e1d600700 1 -- 172.21.15.16:0/225733891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e100042c0 con 0x7f2e201066c0 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.061+0000 7f2e1d600700 1 -- 172.21.15.16:0/225733891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e100119b0 con 0x7f2e201066c0 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.062+0000 7f2e25c00700 1 -- 172.21.15.16:0/225733891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 msgr2=0x7f2e20106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.062+0000 7f2e25c00700 1 --2- 172.21.15.16:0/225733891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e20106ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e100098a0 tx=0x7f2e10009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:43.640 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.062+0000 7f2e25c00700 1 -- 172.21.15.16:0/225733891 shutdown_connections 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.062+0000 7f2e25c00700 1 --2- 172.21.15.16:0/225733891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e20106ae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.062+0000 7f2e25c00700 1 -- 172.21.15.16:0/225733891 >> 172.21.15.16:0/225733891 conn(0x7f2e20101c60 msgr2=0x7f2e201040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.062+0000 7f2e25c00700 1 -- 172.21.15.16:0/225733891 shutdown_connections 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.062+0000 7f2e25c00700 1 -- 172.21.15.16:0/225733891 wait complete. 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.063+0000 7f2e25c00700 1 Processor -- start 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.063+0000 7f2e25c00700 1 -- start start 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.064+0000 7f2e25c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e2019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.064+0000 7f2e25c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2019cbc0 con 0x7f2e201066c0 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.064+0000 7f2e1ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e2019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.064+0000 7f2e1ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e2019c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50050/0 (socket says 172.21.15.16:50050) 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.064+0000 7f2e1ea00700 1 -- 172.21.15.16:0/4092591827 learned_addr learned my addr 172.21.15.16:0/4092591827 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.064+0000 7f2e1ea00700 1 -- 172.21.15.16:0/4092591827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e10009550 con 0x7f2e201066c0 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.064+0000 7f2e1ea00700 1 --2- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e2019c680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e10003e60 tx=0x7f2e10011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.065+0000 7f2e17400700 1 -- 172.21.15.16:0/4092591827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e10003a60 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.065+0000 7f2e17400700 1 -- 172.21.15.16:0/4092591827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e10003bc0 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.065+0000 7f2e17400700 1 -- 172.21.15.16:0/4092591827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e100196c0 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.065+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2019cdc0 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.065+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2019d260 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.066+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2010aaf0 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.066+0000 7f2e17400700 1 -- 172.21.15.16:0/4092591827 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 45554+0+0 (secure 0 0 0) 0x7f2e1001ab60 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.066+0000 7f2e17400700 1 --2- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2e04038780 0x7f2e0403ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.066+0000 7f2e17400700 1 -- 172.21.15.16:0/4092591827 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2e1004caf0 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.067+0000 7f2e1e000700 1 --2- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2e04038780 0x7f2e0403ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.067+0000 7f2e1e000700 1 --2- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2e04038780 0x7f2e0403ac40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c006fd0 tx=0x7f2e0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.071+0000 7f2e17400700 1 -- 172.21.15.16:0/4092591827 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2e10018b00 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.284+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f2e2010ad30 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.284+0000 7f2e17400700 1 -- 172.21.15.16:0/4092591827 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2e10018b00 con 0x7f2e201066c0 2024-04-05T23:26:43.642 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2e04038780 msgr2=0x7f2e0403ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:43.643 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 --2- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2e04038780 0x7f2e0403ac40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c006fd0 tx=0x7f2e0c006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:43.643 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 msgr2=0x7f2e2019c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:43.643 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 --2- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e2019c680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e10003e60 tx=0x7f2e10011a30 comp rx=0 tx=0).stop 2024-04-05T23:26:43.643 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 shutdown_connections 2024-04-05T23:26:43.643 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 --2- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2e04038780 0x7f2e0403ac40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:43.643 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 --2- 172.21.15.16:0/4092591827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e201066c0 0x7f2e2019c680 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:43.643 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 >> 172.21.15.16:0/4092591827 conn(0x7f2e20101c60 msgr2=0x7f2e20102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:43.643 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 shutdown_connections 2024-04-05T23:26:43.643 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:43.287+0000 7f2e25c00700 1 -- 172.21.15.16:0/4092591827 wait complete. 2024-04-05T23:26:44.771 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.073+0000 7f2647e00700 1 Processor -- start 2024-04-05T23:26:44.771 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.073+0000 7f2647e00700 1 -- start start 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.073+0000 7f2647e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f26480791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.073+0000 7f2647e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26480797c0 con 0x7f264807ad90 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.074+0000 7f2646a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f26480791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.074+0000 7f2646a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f26480791f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50062/0 (socket says 172.21.15.16:50062) 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.074+0000 7f2646a00700 1 -- 172.21.15.16:0/3819558625 learned_addr learned my addr 172.21.15.16:0/3819558625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.074+0000 7f2646a00700 1 -- 172.21.15.16:0/3819558625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2648079900 con 0x7f264807ad90 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.074+0000 7f2646a00700 1 --2- 172.21.15.16:0/3819558625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f26480791f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f26380098a0 tx=0x7f2638009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c553a242d6388b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.074+0000 7f2645600700 1 -- 172.21.15.16:0/3819558625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2638004160 con 0x7f264807ad90 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.074+0000 7f2645600700 1 -- 172.21.15.16:0/3819558625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26380042c0 con 0x7f264807ad90 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.074+0000 7f2645600700 1 -- 172.21.15.16:0/3819558625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26380119b0 con 0x7f264807ad90 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.075+0000 7f2647e00700 1 -- 172.21.15.16:0/3819558625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 msgr2=0x7f26480791f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.075+0000 7f2647e00700 1 --2- 172.21.15.16:0/3819558625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f26480791f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f26380098a0 tx=0x7f2638009bb0 comp rx=0 tx=0).stop 2024-04-05T23:26:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.075+0000 7f2647e00700 1 -- 172.21.15.16:0/3819558625 shutdown_connections 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.075+0000 7f2647e00700 1 --2- 172.21.15.16:0/3819558625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f26480791f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.075+0000 7f2647e00700 1 -- 172.21.15.16:0/3819558625 >> 172.21.15.16:0/3819558625 conn(0x7f2648101ad0 msgr2=0x7f2648103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.075+0000 7f2647e00700 1 -- 172.21.15.16:0/3819558625 shutdown_connections 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.075+0000 7f2647e00700 1 -- 172.21.15.16:0/3819558625 wait complete. 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.077+0000 7f2647e00700 1 Processor -- start 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.077+0000 7f2647e00700 1 -- start start 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.077+0000 7f2647e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f264819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.077+0000 7f2647e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264819ca30 con 0x7f264807ad90 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.077+0000 7f2646a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f264819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.078+0000 7f2646a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f264819c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50066/0 (socket says 172.21.15.16:50066) 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.078+0000 7f2646a00700 1 -- 172.21.15.16:0/4013950757 learned_addr learned my addr 172.21.15.16:0/4013950757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.078+0000 7f2646a00700 1 -- 172.21.15.16:0/4013950757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2638009550 con 0x7f264807ad90 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.078+0000 7f2646a00700 1 --2- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f264819c4f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2638011980 tx=0x7f2638011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.078+0000 7f263f400700 1 -- 172.21.15.16:0/4013950757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2638003af0 con 0x7f264807ad90 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.078+0000 7f263f400700 1 -- 172.21.15.16:0/4013950757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2638003c50 con 0x7f264807ad90 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.078+0000 7f263f400700 1 -- 172.21.15.16:0/4013950757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2638019760 con 0x7f264807ad90 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.078+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264819cc30 con 0x7f264807ad90 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.079+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264819d0d0 con 0x7f264807ad90 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.079+0000 7f263f400700 1 -- 172.21.15.16:0/4013950757 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7f263801ab10 con 0x7f264807ad90 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.080+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264810c930 con 0x7f264807ad90 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.080+0000 7f263f400700 1 --2- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f262c038890 0x7f262c03ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.080+0000 7f263f400700 1 -- 172.21.15.16:0/4013950757 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f263804cbf0 con 0x7f264807ad90 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.080+0000 7f2646000700 1 --2- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f262c038890 0x7f262c03ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:44.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.081+0000 7f2646000700 1 --2- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f262c038890 0x7f262c03ad50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2634006fd0 tx=0x7f2634006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:44.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.085+0000 7f263f400700 1 -- 172.21.15.16:0/4013950757 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f263801a430 con 0x7f264807ad90 2024-04-05T23:26:44.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.415+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f264819f9c0 con 0x7f264807ad90 2024-04-05T23:26:44.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.418+0000 7f263f400700 1 -- 172.21.15.16:0/4013950757 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f263802db00 con 0x7f264807ad90 2024-04-05T23:26:44.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f262c038890 msgr2=0x7f262c03ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:44.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 --2- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f262c038890 0x7f262c03ad50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2634006fd0 tx=0x7f2634006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:44.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 msgr2=0x7f264819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 --2- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f264819c4f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2638011980 tx=0x7f2638011a60 comp rx=0 tx=0).stop 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 shutdown_connections 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 --2- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f262c038890 0x7f262c03ad50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 --2- 172.21.15.16:0/4013950757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f264807ad90 0x7f264819c4f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 >> 172.21.15.16:0/4013950757 conn(0x7f2648101ad0 msgr2=0x7f26481026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 shutdown_connections 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-05T23:26:44.420+0000 7f2647e00700 1 -- 172.21.15.16:0/4013950757 wait complete. 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:44.776 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-04-05T23:26:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-05T23:26:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:44.777 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-05T23:26:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-04-05T23:26:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:44.777 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-04-05T23:26:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-05T23:26:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:44.778 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-04-05T23:26:44.826 INFO:tasks.cephadm:Fetching config... 2024-04-05T23:26:44.827 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:26:44.827 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-05T23:26:44.869 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-05T23:26:44.869 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:26:44.869 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-05T23:26:44.953 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-05T23:26:44.953 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:26:44.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-04-05T23:26:45.039 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:44 smithi016 ceph-mon[38248]: mgrmap e13: x(active, since 6s) 2024-04-05T23:26:45.039 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:44 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4013950757' entity='client.admin' 2024-04-05T23:26:45.070 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-05T23:26:45.070 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:26:45.070 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-05T23:26:45.155 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-05T23:26:45.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDBD+3pbLOJbhKPHoBxGja0yX3uIAk9Q0MvND3y4qdMzRrdb3aRvbQfisjNiN9kwZxA858fNYNXW9r9+ZiqbsT7WO4cdRNI97q1SVS75ukiGX4COU35tWQbKDm7r2NqXSWWRc32mWUw803JgzAzx/0baqdhjdkz9W96Hb7Wl9wA+oA7O/KrqyZ8Yo5MdgniKzuPp/l45hEs0OMiobY7hAzNRI9CEV4038Bra+WdGhZ2avTscSZDyPze47KGxy1zCguepU6AjwBNcZ5Q6jmXkrwBFQOuROFuFaXPTShp6SgstrhGxhb+VuYLyLL5Jp+T+7XeD8OCiRMbL3XmWE4BrDxOaAZHiOyXSWJkXyYUF36D9KJFbqXGV5lzqSnD22JmFaEwGKDXuFppOTe7eyImdRCZ6/Mp7/AdqUQ2UjA7N+IWAeSpy6I+Cvou7xNtrwhqZzNgfzQ2e/Mq2/FZEkjFaNVLfzJ38L/zkq1VByOawEkSJMkv2EVelU5LWQa/uCw0eM= ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-05T23:26:45.290 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDBD+3pbLOJbhKPHoBxGja0yX3uIAk9Q0MvND3y4qdMzRrdb3aRvbQfisjNiN9kwZxA858fNYNXW9r9+ZiqbsT7WO4cdRNI97q1SVS75ukiGX4COU35tWQbKDm7r2NqXSWWRc32mWUw803JgzAzx/0baqdhjdkz9W96Hb7Wl9wA+oA7O/KrqyZ8Yo5MdgniKzuPp/l45hEs0OMiobY7hAzNRI9CEV4038Bra+WdGhZ2avTscSZDyPze47KGxy1zCguepU6AjwBNcZ5Q6jmXkrwBFQOuROFuFaXPTShp6SgstrhGxhb+VuYLyLL5Jp+T+7XeD8OCiRMbL3XmWE4BrDxOaAZHiOyXSWJkXyYUF36D9KJFbqXGV5lzqSnD22JmFaEwGKDXuFppOTe7eyImdRCZ6/Mp7/AdqUQ2UjA7N+IWAeSpy6I+Cvou7xNtrwhqZzNgfzQ2e/Mq2/FZEkjFaNVLfzJ38L/zkq1VByOawEkSJMkv2EVelU5LWQa/uCw0eM= ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:26:45.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDBD+3pbLOJbhKPHoBxGja0yX3uIAk9Q0MvND3y4qdMzRrdb3aRvbQfisjNiN9kwZxA858fNYNXW9r9+ZiqbsT7WO4cdRNI97q1SVS75ukiGX4COU35tWQbKDm7r2NqXSWWRc32mWUw803JgzAzx/0baqdhjdkz9W96Hb7Wl9wA+oA7O/KrqyZ8Yo5MdgniKzuPp/l45hEs0OMiobY7hAzNRI9CEV4038Bra+WdGhZ2avTscSZDyPze47KGxy1zCguepU6AjwBNcZ5Q6jmXkrwBFQOuROFuFaXPTShp6SgstrhGxhb+VuYLyLL5Jp+T+7XeD8OCiRMbL3XmWE4BrDxOaAZHiOyXSWJkXyYUF36D9KJFbqXGV5lzqSnD22JmFaEwGKDXuFppOTe7eyImdRCZ6/Mp7/AdqUQ2UjA7N+IWAeSpy6I+Cvou7xNtrwhqZzNgfzQ2e/Mq2/FZEkjFaNVLfzJ38L/zkq1VByOawEkSJMkv2EVelU5LWQa/uCw0eM= ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-05T23:26:45.367 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDBD+3pbLOJbhKPHoBxGja0yX3uIAk9Q0MvND3y4qdMzRrdb3aRvbQfisjNiN9kwZxA858fNYNXW9r9+ZiqbsT7WO4cdRNI97q1SVS75ukiGX4COU35tWQbKDm7r2NqXSWWRc32mWUw803JgzAzx/0baqdhjdkz9W96Hb7Wl9wA+oA7O/KrqyZ8Yo5MdgniKzuPp/l45hEs0OMiobY7hAzNRI9CEV4038Bra+WdGhZ2avTscSZDyPze47KGxy1zCguepU6AjwBNcZ5Q6jmXkrwBFQOuROFuFaXPTShp6SgstrhGxhb+VuYLyLL5Jp+T+7XeD8OCiRMbL3XmWE4BrDxOaAZHiOyXSWJkXyYUF36D9KJFbqXGV5lzqSnD22JmFaEwGKDXuFppOTe7eyImdRCZ6/Mp7/AdqUQ2UjA7N+IWAeSpy6I+Cvou7xNtrwhqZzNgfzQ2e/Mq2/FZEkjFaNVLfzJ38L/zkq1VByOawEkSJMkv2EVelU5LWQa/uCw0eM= ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:26:45.381 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDBD+3pbLOJbhKPHoBxGja0yX3uIAk9Q0MvND3y4qdMzRrdb3aRvbQfisjNiN9kwZxA858fNYNXW9r9+ZiqbsT7WO4cdRNI97q1SVS75ukiGX4COU35tWQbKDm7r2NqXSWWRc32mWUw803JgzAzx/0baqdhjdkz9W96Hb7Wl9wA+oA7O/KrqyZ8Yo5MdgniKzuPp/l45hEs0OMiobY7hAzNRI9CEV4038Bra+WdGhZ2avTscSZDyPze47KGxy1zCguepU6AjwBNcZ5Q6jmXkrwBFQOuROFuFaXPTShp6SgstrhGxhb+VuYLyLL5Jp+T+7XeD8OCiRMbL3XmWE4BrDxOaAZHiOyXSWJkXyYUF36D9KJFbqXGV5lzqSnD22JmFaEwGKDXuFppOTe7eyImdRCZ6/Mp7/AdqUQ2UjA7N+IWAeSpy6I+Cvou7xNtrwhqZzNgfzQ2e/Mq2/FZEkjFaNVLfzJ38L/zkq1VByOawEkSJMkv2EVelU5LWQa/uCw0eM= ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-05T23:26:45.427 INFO:teuthology.orchestra.run.smithi130.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDBD+3pbLOJbhKPHoBxGja0yX3uIAk9Q0MvND3y4qdMzRrdb3aRvbQfisjNiN9kwZxA858fNYNXW9r9+ZiqbsT7WO4cdRNI97q1SVS75ukiGX4COU35tWQbKDm7r2NqXSWWRc32mWUw803JgzAzx/0baqdhjdkz9W96Hb7Wl9wA+oA7O/KrqyZ8Yo5MdgniKzuPp/l45hEs0OMiobY7hAzNRI9CEV4038Bra+WdGhZ2avTscSZDyPze47KGxy1zCguepU6AjwBNcZ5Q6jmXkrwBFQOuROFuFaXPTShp6SgstrhGxhb+VuYLyLL5Jp+T+7XeD8OCiRMbL3XmWE4BrDxOaAZHiOyXSWJkXyYUF36D9KJFbqXGV5lzqSnD22JmFaEwGKDXuFppOTe7eyImdRCZ6/Mp7/AdqUQ2UjA7N+IWAeSpy6I+Cvou7xNtrwhqZzNgfzQ2e/Mq2/FZEkjFaNVLfzJ38L/zkq1VByOawEkSJMkv2EVelU5LWQa/uCw0eM= ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:26:45.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-05T23:26:45.819 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:26:46.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.970+0000 7efda1600700 1 -- 172.21.15.16:0/977486629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 msgr2=0x7efd9c100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:46.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.970+0000 7efda1600700 1 --2- 172.21.15.16:0/977486629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 0x7efd9c100c90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009910 tx=0x7efd8c009c20 comp rx=0 tx=0).stop 2024-04-05T23:26:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.970+0000 7efda1600700 1 -- 172.21.15.16:0/977486629 shutdown_connections 2024-04-05T23:26:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.970+0000 7efda1600700 1 --2- 172.21.15.16:0/977486629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 0x7efd9c100c90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.970+0000 7efda1600700 1 -- 172.21.15.16:0/977486629 >> 172.21.15.16:0/977486629 conn(0x7efd9c0fbe10 msgr2=0x7efd9c0fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.970+0000 7efda1600700 1 -- 172.21.15.16:0/977486629 shutdown_connections 2024-04-05T23:26:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.971+0000 7efda1600700 1 -- 172.21.15.16:0/977486629 wait complete. 2024-04-05T23:26:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.972+0000 7efda1600700 1 Processor -- start 2024-04-05T23:26:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.972+0000 7efda1600700 1 -- start start 2024-04-05T23:26:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.972+0000 7efda1600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 0x7efd9c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.972+0000 7efda1600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c198460 con 0x7efd9c100870 2024-04-05T23:26:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.972+0000 7efd9aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 0x7efd9c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.972+0000 7efd9aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 0x7efd9c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50070/0 (socket says 172.21.15.16:50070) 2024-04-05T23:26:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.972+0000 7efd9aa00700 1 -- 172.21.15.16:0/4107711183 learned_addr learned my addr 172.21.15.16:0/4107711183 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.973+0000 7efd9aa00700 1 -- 172.21.15.16:0/4107711183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c0095f0 con 0x7efd9c100870 2024-04-05T23:26:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.973+0000 7efd9aa00700 1 --2- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 0x7efd9c197f20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efd8c005950 tx=0x7efd8c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.973+0000 7efd93400700 1 -- 172.21.15.16:0/4107711183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd8c01d070 con 0x7efd9c100870 2024-04-05T23:26:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.973+0000 7efd93400700 1 -- 172.21.15.16:0/4107711183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd8c00bc10 con 0x7efd9c100870 2024-04-05T23:26:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.973+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9c198660 con 0x7efd9c100870 2024-04-05T23:26:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.973+0000 7efd93400700 1 -- 172.21.15.16:0/4107711183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd8c003a10 con 0x7efd9c100870 2024-04-05T23:26:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.974+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c198b00 con 0x7efd9c100870 2024-04-05T23:26:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.975+0000 7efd93400700 1 -- 172.21.15.16:0/4107711183 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7efd8c004390 con 0x7efd9c100870 2024-04-05T23:26:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.975+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9c191990 con 0x7efd9c100870 2024-04-05T23:26:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.975+0000 7efd93400700 1 --2- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efd800387f0 0x7efd8003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.975+0000 7efd93400700 1 -- 172.21.15.16:0/4107711183 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efd8c04c4e0 con 0x7efd9c100870 2024-04-05T23:26:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.976+0000 7efd9a000700 1 --2- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efd800387f0 0x7efd8003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.976+0000 7efd9a000700 1 --2- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efd800387f0 0x7efd8003acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efd88006fd0 tx=0x7efd88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:46.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:46.980+0000 7efd93400700 1 -- 172.21.15.16:0/4107711183 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efd8c02abf0 con 0x7efd9c100870 2024-04-05T23:26:47.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.197+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7efd9c062510 con 0x7efd9c100870 2024-04-05T23:26:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.200+0000 7efd93400700 1 -- 172.21.15.16:0/4107711183 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7efd9c062510 con 0x7efd9c100870 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efd800387f0 msgr2=0x7efd8003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 --2- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efd800387f0 0x7efd8003acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efd88006fd0 tx=0x7efd88006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 msgr2=0x7efd9c197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 --2- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 0x7efd9c197f20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efd8c005950 tx=0x7efd8c0057c0 comp rx=0 tx=0).stop 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 shutdown_connections 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 --2- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efd800387f0 0x7efd8003acb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 --2- 172.21.15.16:0/4107711183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9c100870 0x7efd9c197f20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 >> 172.21.15.16:0/4107711183 conn(0x7efd9c0fbe10 msgr2=0x7efd9c0fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 shutdown_connections 2024-04-05T23:26:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:47.203+0000 7efda1600700 1 -- 172.21.15.16:0/4107711183 wait complete. 2024-04-05T23:26:47.729 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-05T23:26:47.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-05T23:26:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:26:48.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:48 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4107711183' entity='client.admin' 2024-04-05T23:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.371+0000 7faaf6c00700 1 -- 172.21.15.16:0/2298665509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 msgr2=0x7faaf0073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.371+0000 7faaf6c00700 1 --2- 172.21.15.16:0/2298665509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 0x7faaf0073370 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faae0009910 tx=0x7faae0009c20 comp rx=0 tx=0).stop 2024-04-05T23:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.371+0000 7faaf6c00700 1 -- 172.21.15.16:0/2298665509 shutdown_connections 2024-04-05T23:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.371+0000 7faaf6c00700 1 --2- 172.21.15.16:0/2298665509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 0x7faaf0073370 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.371+0000 7faaf6c00700 1 -- 172.21.15.16:0/2298665509 >> 172.21.15.16:0/2298665509 conn(0x7faaf00fbe10 msgr2=0x7faaf00fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.371+0000 7faaf6c00700 1 -- 172.21.15.16:0/2298665509 shutdown_connections 2024-04-05T23:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.371+0000 7faaf6c00700 1 -- 172.21.15.16:0/2298665509 wait complete. 2024-04-05T23:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.372+0000 7faaf6c00700 1 Processor -- start 2024-04-05T23:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.372+0000 7faaf6c00700 1 -- start start 2024-04-05T23:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.372+0000 7faaf6c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 0x7faaf0197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.372+0000 7faaf6c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf0198400 con 0x7faaf0074f10 2024-04-05T23:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.373+0000 7faaefe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 0x7faaf0197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.373+0000 7faaefe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 0x7faaf0197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50092/0 (socket says 172.21.15.16:50092) 2024-04-05T23:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.373+0000 7faaefe00700 1 -- 172.21.15.16:0/552585760 learned_addr learned my addr 172.21.15.16:0/552585760 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.373+0000 7faaefe00700 1 -- 172.21.15.16:0/552585760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae00095f0 con 0x7faaf0074f10 2024-04-05T23:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.373+0000 7faaefe00700 1 --2- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 0x7faaf0197ec0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faae0005950 tx=0x7faae00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.373+0000 7faaecc00700 1 -- 172.21.15.16:0/552585760 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faae001d070 con 0x7faaf0074f10 2024-04-05T23:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.374+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf0198600 con 0x7faaf0074f10 2024-04-05T23:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.374+0000 7faaecc00700 1 -- 172.21.15.16:0/552585760 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faae000bc10 con 0x7faaf0074f10 2024-04-05T23:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.374+0000 7faaecc00700 1 -- 172.21.15.16:0/552585760 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faae0003a10 con 0x7faaf0074f10 2024-04-05T23:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.374+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf0198aa0 con 0x7faaf0074f10 2024-04-05T23:26:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.375+0000 7faaecc00700 1 -- 172.21.15.16:0/552585760 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7faae0004390 con 0x7faaf0074f10 2024-04-05T23:26:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.375+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf01917a0 con 0x7faaf0074f10 2024-04-05T23:26:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.375+0000 7faaecc00700 1 --2- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faad40387f0 0x7faad403acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.375+0000 7faaecc00700 1 -- 172.21.15.16:0/552585760 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faae004c4e0 con 0x7faaf0074f10 2024-04-05T23:26:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.375+0000 7faaef400700 1 --2- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faad40387f0 0x7faad403acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.376+0000 7faaef400700 1 --2- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faad40387f0 0x7faad403acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faadc006fd0 tx=0x7faadc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.381+0000 7faaecc00700 1 -- 172.21.15.16:0/552585760 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faae002abf0 con 0x7faaf0074f10 2024-04-05T23:26:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.602+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7faaf0061320 con 0x7faad40387f0 2024-04-05T23:26:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.605+0000 7faaecc00700 1 -- 172.21.15.16:0/552585760 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faaf0061320 con 0x7faad40387f0 2024-04-05T23:26:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.607+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faad40387f0 msgr2=0x7faad403acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.607+0000 7faaf6c00700 1 --2- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faad40387f0 0x7faad403acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faadc006fd0 tx=0x7faadc006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.608+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 msgr2=0x7faaf0197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.608+0000 7faaf6c00700 1 --2- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 0x7faaf0197ec0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faae0005950 tx=0x7faae00057c0 comp rx=0 tx=0).stop 2024-04-05T23:26:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.608+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 shutdown_connections 2024-04-05T23:26:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.608+0000 7faaf6c00700 1 --2- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faad40387f0 0x7faad403acb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.608+0000 7faaf6c00700 1 --2- 172.21.15.16:0/552585760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaf0074f10 0x7faaf0197ec0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.608+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 >> 172.21.15.16:0/552585760 conn(0x7faaf00fbe10 msgr2=0x7faaf00fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.608+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 shutdown_connections 2024-04-05T23:26:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:49.608+0000 7faaf6c00700 1 -- 172.21.15.16:0/552585760 wait complete. 2024-04-05T23:26:50.008 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi027 2024-04-05T23:26:50.009 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:26:50.009 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/etc/ceph/ceph.conf 2024-04-05T23:26:50.030 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:26:50.030 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:26:50.092 INFO:tasks.cephadm:Adding host smithi027 to orchestrator... 2024-04-05T23:26:50.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi027 2024-04-05T23:26:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:26:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:26:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:50.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:50.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:26:50.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:26:50.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:26:50.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-05T23:26:50.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:50 smithi016 ceph-mon[38248]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.722+0000 7f37f9c00700 1 -- 172.21.15.16:0/1572077202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 msgr2=0x7f37f4073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.722+0000 7f37f9c00700 1 --2- 172.21.15.16:0/1572077202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 0x7f37f4073660 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009910 tx=0x7f37e4009c20 comp rx=0 tx=0).stop 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.723+0000 7f37f9c00700 1 -- 172.21.15.16:0/1572077202 shutdown_connections 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.723+0000 7f37f9c00700 1 --2- 172.21.15.16:0/1572077202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 0x7f37f4073660 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.723+0000 7f37f9c00700 1 -- 172.21.15.16:0/1572077202 >> 172.21.15.16:0/1572077202 conn(0x7f37f40fc2e0 msgr2=0x7f37f40fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.723+0000 7f37f9c00700 1 -- 172.21.15.16:0/1572077202 shutdown_connections 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.723+0000 7f37f9c00700 1 -- 172.21.15.16:0/1572077202 wait complete. 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.724+0000 7f37f9c00700 1 Processor -- start 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.724+0000 7f37f9c00700 1 -- start start 2024-04-05T23:26:51.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.724+0000 7f37f9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 0x7f37f419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.724+0000 7f37f9c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f419cbc0 con 0x7f37f4073280 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.725+0000 7f37f2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 0x7f37f419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.725+0000 7f37f2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 0x7f37f419c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50114/0 (socket says 172.21.15.16:50114) 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.725+0000 7f37f2a00700 1 -- 172.21.15.16:0/2623077600 learned_addr learned my addr 172.21.15.16:0/2623077600 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.725+0000 7f37f2a00700 1 -- 172.21.15.16:0/2623077600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40095f0 con 0x7f37f4073280 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.725+0000 7f37f2a00700 1 --2- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 0x7f37f419c680 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37e4006010 tx=0x7f37e4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.726+0000 7f37eb400700 1 -- 172.21.15.16:0/2623077600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37e401d070 con 0x7f37f4073280 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.726+0000 7f37eb400700 1 -- 172.21.15.16:0/2623077600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37e400bbb0 con 0x7f37f4073280 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.726+0000 7f37eb400700 1 -- 172.21.15.16:0/2623077600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37e4003880 con 0x7f37f4073280 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.726+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f419ce20 con 0x7f37f4073280 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.726+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f419d2a0 con 0x7f37f4073280 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.727+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f40749c0 con 0x7f37f4073280 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.727+0000 7f37eb400700 1 -- 172.21.15.16:0/2623077600 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7f37e400bd20 con 0x7f37f4073280 2024-04-05T23:26:51.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.728+0000 7f37eb400700 1 --2- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f37d80387f0 0x7f37d803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:51.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.728+0000 7f37eb400700 1 -- 172.21.15.16:0/2623077600 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f37e404c480 con 0x7f37f4073280 2024-04-05T23:26:51.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.728+0000 7f37f2000700 1 --2- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f37d80387f0 0x7f37d803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:51.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.729+0000 7f37f2000700 1 --2- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f37d80387f0 0x7f37d803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f37e0006fd0 tx=0x7f37e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:51.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.733+0000 7f37eb400700 1 -- 172.21.15.16:0/2623077600 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f37e402aba0 con 0x7f37f4073280 2024-04-05T23:26:51.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:51 smithi016 ceph-mon[38248]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:26:51.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:51 smithi016 ceph-mon[38248]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:26:51.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:51.954+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi027", "target": ["mon-mgr", ""]}) v1 -- 0x7f37f41110e0 con 0x7f37d80387f0 2024-04-05T23:26:53.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:53.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:53.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:53.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:52 smithi016 ceph-mon[38248]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:26:54.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:53 smithi016 ceph-mon[38248]: Deploying cephadm binary to smithi027 2024-04-05T23:26:54.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.305+0000 7f37eb400700 1 -- 172.21.15.16:0/2623077600 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f37f41110e0 con 0x7f37d80387f0 2024-04-05T23:26:54.298 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi027' with addr '172.21.15.27' 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f37d80387f0 msgr2=0x7f37d803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 --2- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f37d80387f0 0x7f37d803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f37e0006fd0 tx=0x7f37e0006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 msgr2=0x7f37f419c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 --2- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 0x7f37f419c680 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37e4006010 tx=0x7f37e4004ff0 comp rx=0 tx=0).stop 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 shutdown_connections 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 --2- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f37d80387f0 0x7f37d803acb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 --2- 172.21.15.16:0/2623077600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37f4073280 0x7f37f419c680 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 >> 172.21.15.16:0/2623077600 conn(0x7f37f40fc2e0 msgr2=0x7f37f40fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 shutdown_connections 2024-04-05T23:26:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:54.308+0000 7f37f9c00700 1 -- 172.21.15.16:0/2623077600 wait complete. 2024-04-05T23:26:54.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-05T23:26:55.031 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:26:55.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:55 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:55.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:55 smithi016 ceph-mon[38248]: Added host smithi027 2024-04-05T23:26:55.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:55 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:26:55.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:55 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:55.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:55 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.224+0000 7f6f86800700 1 -- 172.21.15.16:0/742489770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 msgr2=0x7f6f80106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.224+0000 7f6f86800700 1 --2- 172.21.15.16:0/742489770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 0x7f6f80106be0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009910 tx=0x7f6f70009c20 comp rx=0 tx=0).stop 2024-04-05T23:26:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.224+0000 7f6f86800700 1 -- 172.21.15.16:0/742489770 shutdown_connections 2024-04-05T23:26:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.224+0000 7f6f86800700 1 --2- 172.21.15.16:0/742489770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 0x7f6f80106be0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.224+0000 7f6f86800700 1 -- 172.21.15.16:0/742489770 >> 172.21.15.16:0/742489770 conn(0x7f6f80075290 msgr2=0x7f6f800756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.224+0000 7f6f86800700 1 -- 172.21.15.16:0/742489770 shutdown_connections 2024-04-05T23:26:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.224+0000 7f6f86800700 1 -- 172.21.15.16:0/742489770 wait complete. 2024-04-05T23:26:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.225+0000 7f6f86800700 1 Processor -- start 2024-04-05T23:26:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.226+0000 7f6f86800700 1 -- start start 2024-04-05T23:26:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.226+0000 7f6f86800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 0x7f6f801982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.226+0000 7f6f86800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f80198830 con 0x7f6f80106800 2024-04-05T23:26:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.226+0000 7f6f7f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 0x7f6f801982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.226+0000 7f6f7f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 0x7f6f801982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44976/0 (socket says 172.21.15.16:44976) 2024-04-05T23:26:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.226+0000 7f6f7f400700 1 -- 172.21.15.16:0/2285989750 learned_addr learned my addr 172.21.15.16:0/2285989750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.226+0000 7f6f7f400700 1 -- 172.21.15.16:0/2285989750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f700095f0 con 0x7f6f80106800 2024-04-05T23:26:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.227+0000 7f6f7f400700 1 --2- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 0x7f6f801982f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6f70006010 tx=0x7f6f70004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.227+0000 7f6f77e00700 1 -- 172.21.15.16:0/2285989750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f7001d070 con 0x7f6f80106800 2024-04-05T23:26:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.227+0000 7f6f77e00700 1 -- 172.21.15.16:0/2285989750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f7000bbb0 con 0x7f6f80106800 2024-04-05T23:26:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.227+0000 7f6f77e00700 1 -- 172.21.15.16:0/2285989750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f70003810 con 0x7f6f80106800 2024-04-05T23:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.227+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f80198a90 con 0x7f6f80106800 2024-04-05T23:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.227+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f80198f10 con 0x7f6f80106800 2024-04-05T23:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.228+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f80101f40 con 0x7f6f80106800 2024-04-05T23:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.228+0000 7f6f77e00700 1 -- 172.21.15.16:0/2285989750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7f6f7000bd20 con 0x7f6f80106800 2024-04-05T23:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.229+0000 7f6f77e00700 1 --2- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6f640387a0 0x7f6f6403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.229+0000 7f6f77e00700 1 -- 172.21.15.16:0/2285989750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6f7004c450 con 0x7f6f80106800 2024-04-05T23:26:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.229+0000 7f6f7ea00700 1 --2- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6f640387a0 0x7f6f6403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.230+0000 7f6f7ea00700 1 --2- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6f640387a0 0x7f6f6403ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c006fd0 tx=0x7f6f6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.234+0000 7f6f77e00700 1 -- 172.21.15.16:0/2285989750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6f7002ab60 con 0x7f6f80106800 2024-04-05T23:26:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.452+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6f8010aaf0 con 0x7f6f640387a0 2024-04-05T23:26:56.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.453+0000 7f6f77e00700 1 -- 172.21.15.16:0/2285989750 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f6f8010aaf0 con 0x7f6f640387a0 2024-04-05T23:26:56.445 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:26:56.446 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}] 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6f640387a0 msgr2=0x7f6f6403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 --2- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6f640387a0 0x7f6f6403ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c006fd0 tx=0x7f6f6c006e40 comp rx=0 tx=0).stop 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 msgr2=0x7f6f801982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 --2- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 0x7f6f801982f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6f70006010 tx=0x7f6f70004ff0 comp rx=0 tx=0).stop 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 shutdown_connections 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 --2- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6f640387a0 0x7f6f6403ac60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 --2- 172.21.15.16:0/2285989750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80106800 0x7f6f801982f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 >> 172.21.15.16:0/2285989750 conn(0x7f6f80075290 msgr2=0x7f6f800fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 shutdown_connections 2024-04-05T23:26:56.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:56.455+0000 7f6f86800700 1 -- 172.21.15.16:0/2285989750 wait complete. 2024-04-05T23:26:56.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:56.826 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi130 2024-04-05T23:26:56.826 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:26:56.826 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.conf 2024-04-05T23:26:56.847 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:26:56.847 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:26:56.907 INFO:tasks.cephadm:Adding host smithi130 to orchestrator... 2024-04-05T23:26:56.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi130 2024-04-05T23:26:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:26:58.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.434+0000 7f0d45e00700 1 -- 172.21.15.16:0/3845909919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 msgr2=0x7f0d40106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:26:58.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.434+0000 7f0d45e00700 1 --2- 172.21.15.16:0/3845909919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 0x7f0d40106bd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009910 tx=0x7f0d30009c20 comp rx=0 tx=0).stop 2024-04-05T23:26:58.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.434+0000 7f0d45e00700 1 -- 172.21.15.16:0/3845909919 shutdown_connections 2024-04-05T23:26:58.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.434+0000 7f0d45e00700 1 --2- 172.21.15.16:0/3845909919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 0x7f0d40106bd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:26:58.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.434+0000 7f0d45e00700 1 -- 172.21.15.16:0/3845909919 >> 172.21.15.16:0/3845909919 conn(0x7f0d400fc2e0 msgr2=0x7f0d400fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:26:58.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.434+0000 7f0d45e00700 1 -- 172.21.15.16:0/3845909919 shutdown_connections 2024-04-05T23:26:58.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.434+0000 7f0d45e00700 1 -- 172.21.15.16:0/3845909919 wait complete. 2024-04-05T23:26:58.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.436+0000 7f0d45e00700 1 Processor -- start 2024-04-05T23:26:58.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.436+0000 7f0d45e00700 1 -- start start 2024-04-05T23:26:58.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.436+0000 7f0d45e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 0x7f0d40198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:58.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.436+0000 7f0d45e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d401987d0 con 0x7f0d401067f0 2024-04-05T23:26:58.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.437+0000 7f0d3ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 0x7f0d40198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:58.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.437+0000 7f0d3ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 0x7f0d40198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45000/0 (socket says 172.21.15.16:45000) 2024-04-05T23:26:58.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.437+0000 7f0d3ea00700 1 -- 172.21.15.16:0/3079978093 learned_addr learned my addr 172.21.15.16:0/3079978093 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:26:58.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.437+0000 7f0d3ea00700 1 -- 172.21.15.16:0/3079978093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d300095f0 con 0x7f0d401067f0 2024-04-05T23:26:58.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.437+0000 7f0d3ea00700 1 --2- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 0x7f0d40198290 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0d30006010 tx=0x7f0d30004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:58.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.437+0000 7f0d44a00700 1 -- 172.21.15.16:0/3079978093 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d3001d070 con 0x7f0d401067f0 2024-04-05T23:26:58.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.437+0000 7f0d44a00700 1 -- 172.21.15.16:0/3079978093 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d3000bbb0 con 0x7f0d401067f0 2024-04-05T23:26:58.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.437+0000 7f0d44a00700 1 -- 172.21.15.16:0/3079978093 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d30003810 con 0x7f0d401067f0 2024-04-05T23:26:58.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.437+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d40198a30 con 0x7f0d401067f0 2024-04-05T23:26:58.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.438+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d40198eb0 con 0x7f0d401067f0 2024-04-05T23:26:58.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.439+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d40101f30 con 0x7f0d401067f0 2024-04-05T23:26:58.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.439+0000 7f0d44a00700 1 -- 172.21.15.16:0/3079978093 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7f0d3000bd20 con 0x7f0d401067f0 2024-04-05T23:26:58.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.439+0000 7f0d44a00700 1 --2- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0d240387f0 0x7f0d2403acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:26:58.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.439+0000 7f0d44a00700 1 -- 172.21.15.16:0/3079978093 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0d3004c430 con 0x7f0d401067f0 2024-04-05T23:26:58.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.440+0000 7f0d3e000700 1 --2- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0d240387f0 0x7f0d2403acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:26:58.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.440+0000 7f0d3e000700 1 --2- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0d240387f0 0x7f0d2403acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c006fd0 tx=0x7f0d2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:26:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.444+0000 7f0d44a00700 1 -- 172.21.15.16:0/3079978093 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0d3002aba0 con 0x7f0d401067f0 2024-04-05T23:26:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:58 smithi016 ceph-mon[38248]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:26:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:58 smithi016 ceph-mon[38248]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:26:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:58.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:58.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:26:58.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:26:58.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:26:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:26:58.666+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d4010aaf0 con 0x7f0d240387f0 2024-04-05T23:26:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:59 smithi016 ceph-mon[38248]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-05T23:26:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:59 smithi016 ceph-mon[38248]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:26:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:59 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:59 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:26:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:26:59 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:00 smithi016 ceph-mon[38248]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:27:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:00 smithi016 ceph-mon[38248]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:00 smithi016 ceph-mon[38248]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:27:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:00 smithi016 ceph-mon[38248]: Deploying cephadm binary to smithi130 2024-04-05T23:27:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:00 smithi016 ceph-mon[38248]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:00.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.916+0000 7f0d44a00700 1 -- 172.21.15.16:0/3079978093 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f0d4010aaf0 con 0x7f0d240387f0 2024-04-05T23:27:00.909 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi130' with addr '172.21.15.130' 2024-04-05T23:27:00.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.918+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0d240387f0 msgr2=0x7f0d2403acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:00.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.918+0000 7f0d45e00700 1 --2- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0d240387f0 0x7f0d2403acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c006fd0 tx=0x7f0d2c006e40 comp rx=0 tx=0).stop 2024-04-05T23:27:00.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.918+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 msgr2=0x7f0d40198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:00.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.918+0000 7f0d45e00700 1 --2- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 0x7f0d40198290 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0d30006010 tx=0x7f0d30004ff0 comp rx=0 tx=0).stop 2024-04-05T23:27:00.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.919+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 shutdown_connections 2024-04-05T23:27:00.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.919+0000 7f0d45e00700 1 --2- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0d240387f0 0x7f0d2403acb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:00.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.919+0000 7f0d45e00700 1 --2- 172.21.15.16:0/3079978093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d401067f0 0x7f0d40198290 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:00.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.919+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 >> 172.21.15.16:0/3079978093 conn(0x7f0d400fc2e0 msgr2=0x7f0d40108370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:00.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.919+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 shutdown_connections 2024-04-05T23:27:00.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:00.919+0000 7f0d45e00700 1 -- 172.21.15.16:0/3079978093 wait complete. 2024-04-05T23:27:01.272 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-05T23:27:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:27:02.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:01 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:02.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:01 smithi016 ceph-mon[38248]: Added host smithi130 2024-04-05T23:27:02.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:01 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:02.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:01 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:02.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:01 smithi016 ceph-mon[38248]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:02.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:01 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:02.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.765+0000 7fb14ce00700 1 -- 172.21.15.16:0/3900124196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 msgr2=0x7fb148106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:02.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.765+0000 7fb14ce00700 1 --2- 172.21.15.16:0/3900124196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 0x7fb148106bd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb138009910 tx=0x7fb138009c20 comp rx=0 tx=0).stop 2024-04-05T23:27:02.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.765+0000 7fb14ce00700 1 -- 172.21.15.16:0/3900124196 shutdown_connections 2024-04-05T23:27:02.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.765+0000 7fb14ce00700 1 --2- 172.21.15.16:0/3900124196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 0x7fb148106bd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:02.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.765+0000 7fb14ce00700 1 -- 172.21.15.16:0/3900124196 >> 172.21.15.16:0/3900124196 conn(0x7fb1480fc2e0 msgr2=0x7fb1480fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:02.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.765+0000 7fb14ce00700 1 -- 172.21.15.16:0/3900124196 shutdown_connections 2024-04-05T23:27:02.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.765+0000 7fb14ce00700 1 -- 172.21.15.16:0/3900124196 wait complete. 2024-04-05T23:27:02.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.766+0000 7fb14ce00700 1 Processor -- start 2024-04-05T23:27:02.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.767+0000 7fb14ce00700 1 -- start start 2024-04-05T23:27:02.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.767+0000 7fb14ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 0x7fb148196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:02.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.767+0000 7fb14ce00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1481965d0 con 0x7fb1481067f0 2024-04-05T23:27:02.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.767+0000 7fb145e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 0x7fb148196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:02.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.768+0000 7fb145e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 0x7fb148196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59888/0 (socket says 172.21.15.16:59888) 2024-04-05T23:27:02.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.768+0000 7fb145e00700 1 -- 172.21.15.16:0/2869486240 learned_addr learned my addr 172.21.15.16:0/2869486240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:27:02.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.768+0000 7fb145e00700 1 -- 172.21.15.16:0/2869486240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1380095f0 con 0x7fb1481067f0 2024-04-05T23:27:02.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.768+0000 7fb145e00700 1 --2- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 0x7fb148196090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb138006010 tx=0x7fb138004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:02.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.768+0000 7fb13ea00700 1 -- 172.21.15.16:0/2869486240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb13801d070 con 0x7fb1481067f0 2024-04-05T23:27:02.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.768+0000 7fb13ea00700 1 -- 172.21.15.16:0/2869486240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb13800bbb0 con 0x7fb1481067f0 2024-04-05T23:27:02.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.768+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb148196830 con 0x7fb1481067f0 2024-04-05T23:27:02.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.768+0000 7fb13ea00700 1 -- 172.21.15.16:0/2869486240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb138003810 con 0x7fb1481067f0 2024-04-05T23:27:02.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.769+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb148196cb0 con 0x7fb1481067f0 2024-04-05T23:27:02.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.769+0000 7fb13ea00700 1 -- 172.21.15.16:0/2869486240 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7fb13800bd20 con 0x7fb1481067f0 2024-04-05T23:27:02.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.770+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb148101f30 con 0x7fb1481067f0 2024-04-05T23:27:02.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.770+0000 7fb13ea00700 1 --2- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb12c0387a0 0x7fb12c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:02.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.770+0000 7fb13ea00700 1 -- 172.21.15.16:0/2869486240 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb13804c470 con 0x7fb1481067f0 2024-04-05T23:27:02.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.770+0000 7fb145400700 1 --2- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb12c0387a0 0x7fb12c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:02.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.771+0000 7fb145400700 1 --2- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb12c0387a0 0x7fb12c03ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb134006fd0 tx=0x7fb134006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.776+0000 7fb13ea00700 1 -- 172.21.15.16:0/2869486240 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb138029360 con 0x7fb1481067f0 2024-04-05T23:27:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.992+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb14810aaf0 con 0x7fb12c0387a0 2024-04-05T23:27:02.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.994+0000 7fb13ea00700 1 -- 172.21.15.16:0/2869486240 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7fb14810aaf0 con 0x7fb12c0387a0 2024-04-05T23:27:02.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:27:02.986 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}] 2024-04-05T23:27:02.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb12c0387a0 msgr2=0x7fb12c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:02.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 --2- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb12c0387a0 0x7fb12c03ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb134006fd0 tx=0x7fb134006e40 comp rx=0 tx=0).stop 2024-04-05T23:27:02.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 msgr2=0x7fb148196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:02.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 --2- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 0x7fb148196090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb138006010 tx=0x7fb138004ff0 comp rx=0 tx=0).stop 2024-04-05T23:27:02.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 shutdown_connections 2024-04-05T23:27:02.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 --2- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb12c0387a0 0x7fb12c03ac60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:02.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 --2- 172.21.15.16:0/2869486240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1481067f0 0x7fb148196090 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:02.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 >> 172.21.15.16:0/2869486240 conn(0x7fb1480fc2e0 msgr2=0x7fb148108420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:02.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 shutdown_connections 2024-04-05T23:27:02.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:02.996+0000 7fb14ce00700 1 -- 172.21.15.16:0/2869486240 wait complete. 2024-04-05T23:27:03.389 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-05T23:27:03.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-05T23:27:03.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:03.741 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:27:04.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:04 smithi016 ceph-mon[38248]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:27:04.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:04 smithi016 ceph-mon[38248]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.907+0000 7f8091600700 1 -- 172.21.15.16:0/3926662589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 msgr2=0x7f808c073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.907+0000 7f8091600700 1 --2- 172.21.15.16:0/3926662589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 0x7f808c073660 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f807c009910 tx=0x7f807c009c20 comp rx=0 tx=0).stop 2024-04-05T23:27:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.907+0000 7f8091600700 1 -- 172.21.15.16:0/3926662589 shutdown_connections 2024-04-05T23:27:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.907+0000 7f8091600700 1 --2- 172.21.15.16:0/3926662589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 0x7f808c073660 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.907+0000 7f8091600700 1 -- 172.21.15.16:0/3926662589 >> 172.21.15.16:0/3926662589 conn(0x7f808c0fc2e0 msgr2=0x7f808c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.907+0000 7f8091600700 1 -- 172.21.15.16:0/3926662589 shutdown_connections 2024-04-05T23:27:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.907+0000 7f8091600700 1 -- 172.21.15.16:0/3926662589 wait complete. 2024-04-05T23:27:04.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.908+0000 7f8091600700 1 Processor -- start 2024-04-05T23:27:04.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.908+0000 7f8091600700 1 -- start start 2024-04-05T23:27:04.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.909+0000 7f8091600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 0x7f808c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:04.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.909+0000 7f8091600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808c19cbc0 con 0x7f808c073280 2024-04-05T23:27:04.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.909+0000 7f808aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 0x7f808c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:04.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.909+0000 7f808aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 0x7f808c19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59902/0 (socket says 172.21.15.16:59902) 2024-04-05T23:27:04.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.909+0000 7f808aa00700 1 -- 172.21.15.16:0/2960195618 learned_addr learned my addr 172.21.15.16:0/2960195618 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:27:04.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.909+0000 7f808aa00700 1 -- 172.21.15.16:0/2960195618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c0095f0 con 0x7f808c073280 2024-04-05T23:27:04.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.910+0000 7f808aa00700 1 --2- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 0x7f808c19c680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f807c006010 tx=0x7f807c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:04.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.910+0000 7f8083400700 1 -- 172.21.15.16:0/2960195618 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f807c01d070 con 0x7f808c073280 2024-04-05T23:27:04.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.910+0000 7f8083400700 1 -- 172.21.15.16:0/2960195618 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f807c00bbb0 con 0x7f808c073280 2024-04-05T23:27:04.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.910+0000 7f8083400700 1 -- 172.21.15.16:0/2960195618 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f807c003880 con 0x7f808c073280 2024-04-05T23:27:04.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.910+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f808c19ce20 con 0x7f808c073280 2024-04-05T23:27:04.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.910+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f808c19d2a0 con 0x7f808c073280 2024-04-05T23:27:04.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.911+0000 7f8083400700 1 -- 172.21.15.16:0/2960195618 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7f807c00bd20 con 0x7f808c073280 2024-04-05T23:27:04.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.911+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808c0749c0 con 0x7f808c073280 2024-04-05T23:27:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.912+0000 7f8083400700 1 --2- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f80700387f0 0x7f807003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.912+0000 7f8083400700 1 -- 172.21.15.16:0/2960195618 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f807c04c4c0 con 0x7f808c073280 2024-04-05T23:27:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.912+0000 7f808a000700 1 --2- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f80700387f0 0x7f807003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.913+0000 7f808a000700 1 --2- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f80700387f0 0x7f807003acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8078006fd0 tx=0x7f8078006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:04.917+0000 7f8083400700 1 -- 172.21.15.16:0/2960195618 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f807c02aba0 con 0x7f808c073280 2024-04-05T23:27:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.138+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f808c062510 con 0x7f808c073280 2024-04-05T23:27:05.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.550+0000 7f8083400700 1 -- 172.21.15.16:0/2960195618 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f807c026020 con 0x7f808c073280 2024-04-05T23:27:05.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f80700387f0 msgr2=0x7f807003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:05.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 --2- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f80700387f0 0x7f807003acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8078006fd0 tx=0x7f8078006e40 comp rx=0 tx=0).stop 2024-04-05T23:27:05.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 msgr2=0x7f808c19c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:05.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 --2- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 0x7f808c19c680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f807c006010 tx=0x7f807c004ff0 comp rx=0 tx=0).stop 2024-04-05T23:27:05.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 shutdown_connections 2024-04-05T23:27:05.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 --2- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f80700387f0 0x7f807003acb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:05.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 --2- 172.21.15.16:0/2960195618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f808c073280 0x7f808c19c680 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:05.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 >> 172.21.15.16:0/2960195618 conn(0x7f808c0fc2e0 msgr2=0x7f808c0fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:05.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 shutdown_connections 2024-04-05T23:27:05.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:05.553+0000 7f8091600700 1 -- 172.21.15.16:0/2960195618 wait complete. 2024-04-05T23:27:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-04-05T23:27:05.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:05.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:05.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:05.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:05.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:27:05.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2960195618' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:05.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:05.905 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-04-05T23:27:05.905 INFO:tasks.cephadm:Adding mon.b on smithi027 2024-04-05T23:27:05.905 INFO:tasks.cephadm:Adding mon.c on smithi130 2024-04-05T23:27:05.905 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '3;smithi016:172.21.15.16=a;smithi027:172.21.15.27=b;smithi130:172.21.15.130=c' 2024-04-05T23:27:06.147 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:06.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:06 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2960195618' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-05T23:27:06.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:06 smithi016 ceph-mon[38248]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T23:27:07.110 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.115+0000 7f3413a54700 1 -- 172.21.15.130:0/932186035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 msgr2=0x7f340c106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:07.110 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.115+0000 7f3413a54700 1 --2- 172.21.15.130:0/932186035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 0x7f340c106be0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3408009910 tx=0x7f3408009c20 comp rx=0 tx=0).stop 2024-04-05T23:27:07.110 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.116+0000 7f3413a54700 1 -- 172.21.15.130:0/932186035 shutdown_connections 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.116+0000 7f3413a54700 1 --2- 172.21.15.130:0/932186035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 0x7f340c106be0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.116+0000 7f3413a54700 1 -- 172.21.15.130:0/932186035 >> 172.21.15.130:0/932186035 conn(0x7f340c075290 msgr2=0x7f340c0756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.116+0000 7f3413a54700 1 -- 172.21.15.130:0/932186035 shutdown_connections 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.116+0000 7f3413a54700 1 -- 172.21.15.130:0/932186035 wait complete. 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.116+0000 7f3413a54700 1 Processor -- start 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.117+0000 7f3413a54700 1 -- start start 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.117+0000 7f3413a54700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 0x7f340c196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.117+0000 7f3413a54700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1965b0 con 0x7f340c106800 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.117+0000 7f34117f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 0x7f340c196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.117+0000 7f34117f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 0x7f340c196070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.130:32956/0 (socket says 172.21.15.130:32956) 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.117+0000 7f34117f0700 1 -- 172.21.15.130:0/1740911046 learned_addr learned my addr 172.21.15.130:0/1740911046 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:27:07.111 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.117+0000 7f34117f0700 1 -- 172.21.15.130:0/1740911046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34080095f0 con 0x7f340c106800 2024-04-05T23:27:07.112 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.118+0000 7f34117f0700 1 --2- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 0x7f340c196070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3408004f40 tx=0x7f3408005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:07.112 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.118+0000 7f34027fc700 1 -- 172.21.15.130:0/1740911046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f340801d070 con 0x7f340c106800 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.118+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340c196810 con 0x7f340c106800 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.118+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f340c196c90 con 0x7f340c106800 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.118+0000 7f34027fc700 1 -- 172.21.15.130:0/1740911046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f340800bbb0 con 0x7f340c106800 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.118+0000 7f34027fc700 1 -- 172.21.15.130:0/1740911046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3408003810 con 0x7f340c106800 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.119+0000 7f34027fc700 1 -- 172.21.15.130:0/1740911046 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7f3408003a80 con 0x7f340c106800 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.119+0000 7f34027fc700 1 --2- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f33f80387a0 0x7f33f803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.119+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340c101f40 con 0x7f340c106800 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.119+0000 7f34027fc700 1 -- 172.21.15.130:0/1740911046 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f340804c540 con 0x7f340c106800 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.119+0000 7f3410fef700 1 --2- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f33f80387a0 0x7f33f803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:07.113 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.120+0000 7f3410fef700 1 --2- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f33f80387a0 0x7f33f803ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f33fc006fd0 tx=0x7f33fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:07.117 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.125+0000 7f34027fc700 1 -- 172.21.15.130:0/1740911046 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3408015740 con 0x7f340c106800 2024-04-05T23:27:07.318 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.324+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi027:172.21.15.27=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f340c10ac20 con 0x7f33f80387a0 2024-04-05T23:27:07.325 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.332+0000 7f34027fc700 1 -- 172.21.15.130:0/1740911046 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f340c10ac20 con 0x7f33f80387a0 2024-04-05T23:27:07.325 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mon update... 2024-04-05T23:27:07.327 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f33f80387a0 msgr2=0x7f33f803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:07.328 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 --2- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f33f80387a0 0x7f33f803ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f33fc006fd0 tx=0x7f33fc006e40 comp rx=0 tx=0).stop 2024-04-05T23:27:07.328 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 msgr2=0x7f340c196070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:07.328 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 --2- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 0x7f340c196070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3408004f40 tx=0x7f3408005020 comp rx=0 tx=0).stop 2024-04-05T23:27:07.328 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 shutdown_connections 2024-04-05T23:27:07.328 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 --2- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f33f80387a0 0x7f33f803ac60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:07.328 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 --2- 172.21.15.130:0/1740911046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f340c106800 0x7f340c196070 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:07.328 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 >> 172.21.15.130:0/1740911046 conn(0x7f340c075290 msgr2=0x7f340c0fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:07.328 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 shutdown_connections 2024-04-05T23:27:07.328 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:07.334+0000 7f3413a54700 1 -- 172.21.15.130:0/1740911046 wait complete. 2024-04-05T23:27:07.698 DEBUG:teuthology.orchestra.run.smithi027:mon.b> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.b.service 2024-04-05T23:27:07.700 DEBUG:teuthology.orchestra.run.smithi130:mon.c> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.c.service 2024-04-05T23:27:07.703 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-05T23:27:07.703 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T23:27:07.733 INFO:journalctl@ceph.mon.c.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:27:07.738 INFO:journalctl@ceph.mon.b.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:27:08.031 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi027:172.21.15.27=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi027:172.21.15.27=b;smithi130:172.21.15.130=c;count:3 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:08.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:08 smithi016 ceph-mon[38248]: Deploying daemon mon.c on smithi130 2024-04-05T23:27:09.431 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.437+0000 7fc244165700 1 -- 172.21.15.130:0/689315979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 msgr2=0x7fc23c101e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:09.431 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.437+0000 7fc244165700 1 --2- 172.21.15.130:0/689315979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 0x7fc23c101e10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009910 tx=0x7fc22c009c20 comp rx=0 tx=0).stop 2024-04-05T23:27:09.431 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.437+0000 7fc244165700 1 -- 172.21.15.130:0/689315979 shutdown_connections 2024-04-05T23:27:09.431 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.437+0000 7fc244165700 1 --2- 172.21.15.130:0/689315979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 0x7fc23c101e10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:09.431 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.437+0000 7fc244165700 1 -- 172.21.15.130:0/689315979 >> 172.21.15.130:0/689315979 conn(0x7fc23c0fb220 msgr2=0x7fc23c0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.437+0000 7fc244165700 1 -- 172.21.15.130:0/689315979 shutdown_connections 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.437+0000 7fc244165700 1 -- 172.21.15.130:0/689315979 wait complete. 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.437+0000 7fc244165700 1 Processor -- start 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.437+0000 7fc244165700 1 -- start start 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.438+0000 7fc244165700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 0x7fc23c0ff690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.438+0000 7fc244165700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c101550 con 0x7fc23c101a30 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.438+0000 7fc241f01700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 0x7fc23c0ff690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.438+0000 7fc241f01700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 0x7fc23c0ff690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.130:32966/0 (socket says 172.21.15.130:32966) 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.438+0000 7fc241f01700 1 -- 172.21.15.130:0/3497076510 learned_addr learned my addr 172.21.15.130:0/3497076510 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:27:09.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.438+0000 7fc241f01700 1 -- 172.21.15.130:0/3497076510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0095f0 con 0x7fc23c101a30 2024-04-05T23:27:09.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.439+0000 7fc241f01700 1 --2- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 0x7fc23c0ff690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc22c004f40 tx=0x7fc22c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:09.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.439+0000 7fc232ffd700 1 -- 172.21.15.130:0/3497076510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc22c01d070 con 0x7fc23c101a30 2024-04-05T23:27:09.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.439+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c0ffbd0 con 0x7fc23c101a30 2024-04-05T23:27:09.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.439+0000 7fc232ffd700 1 -- 172.21.15.130:0/3497076510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc22c004370 con 0x7fc23c101a30 2024-04-05T23:27:09.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.439+0000 7fc232ffd700 1 -- 172.21.15.130:0/3497076510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc22c003970 con 0x7fc23c101a30 2024-04-05T23:27:09.434 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.439+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c0ffff0 con 0x7fc23c101a30 2024-04-05T23:27:09.434 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.440+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c069140 con 0x7fc23c101a30 2024-04-05T23:27:09.434 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.440+0000 7fc232ffd700 1 -- 172.21.15.130:0/3497076510 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7fc22c003c30 con 0x7fc23c101a30 2024-04-05T23:27:09.434 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.440+0000 7fc232ffd700 1 --2- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2280387a0 0x7fc22803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:09.434 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.440+0000 7fc232ffd700 1 -- 172.21.15.130:0/3497076510 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc22c04c4e0 con 0x7fc23c101a30 2024-04-05T23:27:09.434 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.440+0000 7fc241700700 1 --2- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2280387a0 0x7fc22803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:09.434 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.441+0000 7fc241700700 1 --2- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2280387a0 0x7fc22803ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc238006fd0 tx=0x7fc238006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:09.438 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.446+0000 7fc232ffd700 1 -- 172.21.15.130:0/3497076510 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc22c0044e0 con 0x7fc23c101a30 2024-04-05T23:27:09.731 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.737+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc23c100c40 con 0x7fc23c101a30 2024-04-05T23:27:09.731 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.738+0000 7fc232ffd700 1 -- 172.21.15.130:0/3497076510 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7fc22c026030 con 0x7fc23c101a30 2024-04-05T23:27:09.732 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:27:09.732 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":1,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","modified":"2024-04-05T23:25:32.262742Z","created":"2024-04-05T23:25:32.262742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T23:27:09.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2280387a0 msgr2=0x7fc22803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:09.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 --2- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2280387a0 0x7fc22803ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc238006fd0 tx=0x7fc238006e40 comp rx=0 tx=0).stop 2024-04-05T23:27:09.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 msgr2=0x7fc23c0ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:09.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 --2- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 0x7fc23c0ff690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc22c004f40 tx=0x7fc22c005020 comp rx=0 tx=0).stop 2024-04-05T23:27:09.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 shutdown_connections 2024-04-05T23:27:09.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 --2- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2280387a0 0x7fc22803ac60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:09.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 --2- 172.21.15.130:0/3497076510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc23c101a30 0x7fc23c0ff690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:09.735 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 >> 172.21.15.130:0/3497076510 conn(0x7fc23c0fb220 msgr2=0x7fc23c1047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:09.735 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 shutdown_connections 2024-04-05T23:27:09.735 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:09.741+0000 7fc244165700 1 -- 172.21.15.130:0/3497076510 wait complete. 2024-04-05T23:27:09.736 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 1 2024-04-05T23:27:10.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:10 smithi016 ceph-mon[38248]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:10.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:10 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/3497076510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T23:27:11.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:10 smithi130 bash[79380]: 96e30fb978aa0f2369b325298735e46a2212d83ab35d2e28cdf15f0b5c57267d 2024-04-05T23:27:11.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:10 smithi130 systemd[1]: Started Ceph mon.c for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:27:11.135 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-05T23:27:11.136 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T23:27:11.353 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:27:15.866 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.872+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/3191209428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9c0057d0 msgr2=0x7fbb9c007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:15.866 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.872+0000 7fbbb4ff0700 1 --2- 172.21.15.130:0/3191209428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9c0057d0 0x7fbb9c007c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00ff770 tx=0x7fbb98014ec0 comp rx=0 tx=0).stop 2024-04-05T23:27:15.866 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.872+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/3191209428 shutdown_connections 2024-04-05T23:27:15.867 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.872+0000 7fbbb4ff0700 1 --2- 172.21.15.130:0/3191209428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9c0057d0 0x7fbb9c007c60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:15.867 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.872+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/3191209428 >> 172.21.15.130:0/3191209428 conn(0x7fbbb00fb390 msgr2=0x7fbbb00fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:15.867 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.872+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/3191209428 shutdown_connections 2024-04-05T23:27:15.867 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.872+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/3191209428 wait complete. 2024-04-05T23:27:15.867 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.873+0000 7fbbb4ff0700 1 Processor -- start 2024-04-05T23:27:15.867 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.873+0000 7fbbb4ff0700 1 -- start start 2024-04-05T23:27:15.869 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbb4ff0700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 0x7fbbb01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:15.869 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbb4ff0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbb0198910 0x7fbbb019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:15.869 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbb4ff0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb0198ea0 con 0x7fbbb0198910 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbb4ff0700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb0199010 con 0x7fbb9c0057d0 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbae59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 0x7fbbb01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbae59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 0x7fbbb01983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:35920/0 (socket says 172.21.15.130:35920) 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbae59c700 1 -- 172.21.15.130:0/1030178976 learned_addr learned my addr 172.21.15.130:0/1030178976 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbae59c700 1 -- 172.21.15.130:0/1030178976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 msgr2=0x7fbbb01983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbae59c700 1 -- 172.21.15.130:0/1030178976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 msgr2=0x7fbbb01983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbae59c700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 0x7fbbb01983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbae59c700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 0x7fbbb01983d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbadd9b700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbb0198910 0x7fbbb019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbadd9b700 1 -- 172.21.15.130:0/1030178976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 msgr2=0x7fbbb01983d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbadd9b700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 0x7fbbb01983d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.874+0000 7fbbadd9b700 1 -- 172.21.15.130:0/1030178976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb98014910 con 0x7fbbb0198910 2024-04-05T23:27:15.870 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.875+0000 7fbbadd9b700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbb0198910 0x7fbbb019ccc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbba000b4d0 tx=0x7fbba000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.875+0000 7fbba77fe700 1 -- 172.21.15.130:0/1030178976 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fbba0011630 con 0x7fbbb0198910 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.875+0000 7fbba77fe700 1 -- 172.21.15.130:0/1030178976 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbba0011c50 con 0x7fbbb0198910 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.875+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb019d260 con 0x7fbbb0198910 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.875+0000 7fbba77fe700 1 -- 172.21.15.130:0/1030178976 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fbba0010d90 con 0x7fbbb0198910 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.875+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb019d7b0 con 0x7fbbb0198910 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.876+0000 7fbba77fe700 1 -- 172.21.15.130:0/1030178976 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7fbba001a400 con 0x7fbbb0198910 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.876+0000 7fbba77fe700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb90038630 0x7fbb9003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.876+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb0192230 con 0x7fbbb0198910 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.876+0000 7fbba77fe700 1 -- 172.21.15.130:0/1030178976 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbba004aec0 con 0x7fbbb0198910 2024-04-05T23:27:15.871 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.877+0000 7fbbae59c700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb90038630 0x7fbb9003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:15.872 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.877+0000 7fbbae59c700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb90038630 0x7fbb9003aaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005b40 tx=0x7fbb98005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:15.875 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:15.882+0000 7fbba77fe700 1 -- 172.21.15.130:0/1030178976 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbba00193b0 con 0x7fbbb0198910 2024-04-05T23:27:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:27:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: mon.a calling monitor election 2024-04-05T23:27:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: mon.c calling monitor election 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: monmap e2: 2 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: fsmap 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: mgrmap e13: x(active, since 38s) 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: overall HEALTH_OK 2024-04-05T23:27:16.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:16.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:16.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:16.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:16.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:16.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:27:16.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: mon.a calling monitor election 2024-04-05T23:27:16.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:16.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: mon.c calling monitor election 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: monmap e2: 2 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: fsmap 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: mgrmap e13: x(active, since 38s) 2024-04-05T23:27:16.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: overall HEALTH_OK 2024-04-05T23:27:16.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:16.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:16.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:16.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:16.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:16.146 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.153+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbb0062510 con 0x7fbbb0198910 2024-04-05T23:27:16.147 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.154+0000 7fbba77fe700 1 -- 172.21.15.130:0/1030178976 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7fbba003f090 con 0x7fbbb0198910 2024-04-05T23:27:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:27:16.147 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":2,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","modified":"2024-04-05T23:27:10.729756Z","created":"2024-04-05T23:25:32.262742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-05T23:27:16.149 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb90038630 msgr2=0x7fbb9003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:16.149 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb90038630 0x7fbb9003aaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005b40 tx=0x7fbb98005a90 comp rx=0 tx=0).stop 2024-04-05T23:27:16.149 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbb0198910 msgr2=0x7fbbb019ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:16.149 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbb0198910 0x7fbbb019ccc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbba000b4d0 tx=0x7fbba000b890 comp rx=0 tx=0).stop 2024-04-05T23:27:16.149 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 shutdown_connections 2024-04-05T23:27:16.149 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb90038630 0x7fbb9003aaf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:16.149 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb9c0057d0 0x7fbbb01983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:16.150 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 --2- 172.21.15.130:0/1030178976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbb0198910 0x7fbbb019ccc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:16.150 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 >> 172.21.15.130:0/1030178976 conn(0x7fbbb00fb390 msgr2=0x7fbbb00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:16.150 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 shutdown_connections 2024-04-05T23:27:16.150 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:16.156+0000 7fbbb4ff0700 1 -- 172.21.15.130:0/1030178976 wait complete. 2024-04-05T23:27:16.150 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 2 2024-04-05T23:27:17.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:16 smithi016 ceph-mon[38248]: Deploying daemon mon.b on smithi027 2024-04-05T23:27:17.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:16 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/1030178976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T23:27:17.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:16 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:17.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:16 smithi130 ceph-mon[79417]: Deploying daemon mon.b on smithi027 2024-04-05T23:27:17.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:16 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/1030178976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T23:27:17.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:16 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:17.516 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-05T23:27:17.516 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T23:27:17.744 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:27:18.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:17 smithi016 ceph-mon[38248]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:18.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:17 smithi130 ceph-mon[79417]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:18.637 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.642+0000 7f16f3af2700 1 -- 172.21.15.130:0/568928450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16ec106910 msgr2=0x7f16ec106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.642+0000 7f16f3af2700 1 --2- 172.21.15.130:0/568928450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16ec106910 0x7f16ec106cf0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009910 tx=0x7f16e0009c20 comp rx=0 tx=0).stop 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.642+0000 7f16f3af2700 1 -- 172.21.15.130:0/568928450 shutdown_connections 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.642+0000 7f16f3af2700 1 --2- 172.21.15.130:0/568928450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16ec106910 0x7f16ec106cf0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.642+0000 7f16f3af2700 1 -- 172.21.15.130:0/568928450 >> 172.21.15.130:0/568928450 conn(0x7f16ec0fc440 msgr2=0x7f16ec0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.642+0000 7f16f3af2700 1 -- 172.21.15.130:0/568928450 shutdown_connections 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.642+0000 7f16f3af2700 1 -- 172.21.15.130:0/568928450 wait complete. 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f3af2700 1 Processor -- start 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f3af2700 1 -- start start 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f3af2700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 0x7f16ec0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f3af2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16ec073720 0x7f16ec073ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f3af2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec0750a0 con 0x7f16ec073720 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f3af2700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec1a2470 con 0x7f16ec106910 2024-04-05T23:27:18.638 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f188e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 0x7f16ec0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:18.639 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f188e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 0x7f16ec0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:35930/0 (socket says 172.21.15.130:35930) 2024-04-05T23:27:18.639 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f188e700 1 -- 172.21.15.130:0/4177503516 learned_addr learned my addr 172.21.15.130:0/4177503516 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:27:18.639 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f188e700 1 -- 172.21.15.130:0/4177503516 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 msgr2=0x7f16ec0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-05T23:27:18.639 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f188e700 1 -- 172.21.15.130:0/4177503516 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 msgr2=0x7f16ec0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T23:27:18.639 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.643+0000 7f16f188e700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 0x7f16ec0731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-05T23:27:18.639 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16f188e700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 0x7f16ec0731e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T23:27:18.639 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16f108d700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16ec073720 0x7f16ec073ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:18.640 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16f108d700 1 -- 172.21.15.130:0/4177503516 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 msgr2=0x7f16ec0731e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16f108d700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 0x7f16ec0731e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16f108d700 1 -- 172.21.15.130:0/4177503516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e00095f0 con 0x7f16ec073720 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16f108d700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16ec073720 0x7f16ec073ba0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f16e800b4d0 tx=0x7f16e800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16deffd700 1 -- 172.21.15.130:0/4177503516 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f16e8011630 con 0x7f16ec073720 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec1a2610 con 0x7f16ec073720 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16deffd700 1 -- 172.21.15.130:0/4177503516 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16e8011c50 con 0x7f16ec073720 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16deffd700 1 -- 172.21.15.130:0/4177503516 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f16e8010d90 con 0x7f16ec073720 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.644+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec1a2ae0 con 0x7f16ec073720 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.645+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec102050 con 0x7f16ec073720 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.645+0000 7f16deffd700 1 -- 172.21.15.130:0/4177503516 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7f16e8010410 con 0x7f16ec073720 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.646+0000 7f16deffd700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16d80385e0 0x7f16d803aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.646+0000 7f16deffd700 1 -- 172.21.15.130:0/4177503516 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f16e804ad20 con 0x7f16ec073720 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.646+0000 7f16f188e700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16d80385e0 0x7f16d803aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:18.641 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.646+0000 7f16f188e700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16d80385e0 0x7f16d803aaa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f16e0000c00 tx=0x7f16e000b580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:18.645 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.652+0000 7f16deffd700 1 -- 172.21.15.130:0/4177503516 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f16e8017910 con 0x7f16ec073720 2024-04-05T23:27:18.851 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.857+0000 7f16deffd700 1 -- 172.21.15.130:0/4177503516 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8022e80 con 0x7f16ec073720 2024-04-05T23:27:18.922 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:18.927+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16ec062510 con 0x7f16ec073720 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 systemd[1]: Started Ceph mon.b for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: RocksDB version: 7.9.2 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Git sha 0 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Compile date 2024-04-05 12:53:35 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: DB SUMMARY 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: DB Session ID: ORFSL2OCK88FYK7P254Z 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: CURRENT file: CURRENT 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: IDENTITY file: IDENTITY 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-05T23:27:19.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.error_if_exists: 0 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.create_if_missing: 0 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.paranoid_checks: 1 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.env: 0x55deab5f17a0 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.fs: PosixFileSystem 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.info_log: 0x55deadb35500 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_file_opening_threads: 16 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.statistics: (nil) 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.use_fsync: 0 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_log_file_size: 0 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.keep_log_file_num: 1000 2024-04-05T23:27:19.220 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.recycle_log_file_num: 0 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.allow_fallocate: 1 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.allow_mmap_reads: 0 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.allow_mmap_writes: 0 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.use_direct_reads: 0 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.create_missing_column_families: 0 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.db_log_dir: 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.wal_dir: 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.advise_random_on_open: 1 2024-04-05T23:27:19.221 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.db_write_buffer_size: 0 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.write_buffer_manager: 0x55deacdca320 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.rate_limiter: (nil) 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.wal_recovery_mode: 2 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.enable_thread_tracking: 0 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.enable_pipelined_write: 0 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.unordered_write: 0 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-05T23:27:19.222 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.row_cache: None 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.wal_filter: None 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.allow_ingest_behind: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.two_write_queues: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.manual_wal_flush: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.wal_compression: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.atomic_flush: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.log_readahead_size: 0 2024-04-05T23:27:19.223 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.best_efforts_recovery: 0 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.allow_data_in_errors: 0 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.db_host_id: __hostname__ 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_background_jobs: 2 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_background_compactions: -1 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_subcompactions: 1 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_total_wal_size: 0 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-05T23:27:19.224 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_open_files: -1 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bytes_per_sync: 0 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_readahead_size: 0 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_background_flushes: -1 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Compression algorithms supported: 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: kZSTD supported: 0 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: kXpressCompression supported: 0 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: kLZ4HCCompression supported: 1 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: kZlibCompression supported: 1 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: kSnappyCompression supported: 1 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: kLZ4Compression supported: 1 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: kBZip2Compression supported: 0 2024-04-05T23:27:19.225 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-05T23:27:19.226 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-05T23:27:19.226 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-05T23:27:19.226 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-05T23:27:19.226 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-05T23:27:19.226 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.merge_operator: 2024-04-05T23:27:19.226 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_filter: None 2024-04-05T23:27:19.226 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_filter_factory: None 2024-04-05T23:27:19.226 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.sst_partitioner_factory: None 2024-04-05T23:27:19.226 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55deadb35600) 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: index_type: 0 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: data_block_index_type: 0 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: index_shortening: 1 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: checksum: 4 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: no_block_cache: 0 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: block_cache: 0x55deace4b1f0 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-04-05T23:27:19.227 INFO:journalctl@ceph.mon.b.smithi027.stdout: block_cache_options: 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: capacity : 536870912 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: num_shard_bits : 4 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: strict_capacity_limit : 0 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: block_cache_compressed: (nil) 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: persistent_cache: (nil) 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: block_size: 4096 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: block_size_deviation: 10 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: block_restart_interval: 16 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: index_block_restart_interval: 1 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: metadata_block_size: 4096 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: partition_filters: 0 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: use_delta_encoding: 1 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: filter_policy: bloomfilter 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: whole_key_filtering: 1 2024-04-05T23:27:19.228 INFO:journalctl@ceph.mon.b.smithi027.stdout: verify_compression: 0 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout: format_version: 5 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout: enable_index_compression: 1 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout: block_align: 0 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout: max_auto_readahead_size: 262144 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout: prepopulate_block_cache: 0 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout: initial_auto_readahead_size: 8192 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.write_buffer_size: 33554432 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_write_buffer_number: 2 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression: NoCompression 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression: Disabled 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.prefix_extractor: nullptr 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.num_levels: 7 2024-04-05T23:27:19.229 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-05T23:27:19.230 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression_opts.level: 32767 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression_opts.strategy: 0 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression_opts.enabled: false 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.target_file_size_base: 67108864 2024-04-05T23:27:19.231 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-05T23:27:19.232 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.arena_block_size: 1048576 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.disable_auto_compactions: 0 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.table_properties_collectors: 2024-04-05T23:27:19.233 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.inplace_update_support: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.bloom_locality: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.max_successive_merges: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.paranoid_file_checks: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.force_consistency_checks: 1 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.report_bg_io_stats: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.ttl: 2592000 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-05T23:27:19.234 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.enable_blob_files: false 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.min_blob_size: 0 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.blob_file_size: 268435456 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.blob_file_starting_level: 0 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 70265d95-833c-4279-a5e7-390d2ba4b9ac 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712359638827097, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-05T23:27:19.235 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712359638827651, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712359638, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "70265d95-833c-4279-a5e7-390d2ba4b9ac", "db_session_id": "ORFSL2OCK88FYK7P254Z", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712359638827748, "job": 1, "event": "recovery_finished"} 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55deaceea000 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: DB pointer 0x55deaced8000 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout: ** DB Stats ** 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T23:27:19.236 INFO:journalctl@ceph.mon.b.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T23:27:19.237 INFO:journalctl@ceph.mon.b.smithi027.stdout: 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: Interval compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: Block cache BinnedLRUCache@0x55deace4b1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-05T23:27:19.238 INFO:journalctl@ceph.mon.b.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout: 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: using public_addr v2:172.21.15.27:0/0 -> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: starting mon.b rank -1 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(???) e0 preinit fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).mds e1 new map 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout: e1 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout: legacy client fscid: -1 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout: 2024-04-05T23:27:19.239 INFO:journalctl@ceph.mon.b.smithi027.stdout: No filesystems configured 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Deploying cephadm binary to smithi130 2024-04-05T23:27:19.240 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Added host smithi130 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:19.241 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2960195618' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2960195618' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.242 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi027:172.21.15.27=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi027:172.21.15.27=b;smithi130:172.21.15.130=c;count:3 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Deploying daemon mon.c on smithi130 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/3497076510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.a calling monitor election 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.243 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.c calling monitor election 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: monmap e2: 2 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: fsmap 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mgrmap e13: x(active, since 38s) 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: overall HEALTH_OK 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.244 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.245 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:19.245 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:19.245 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:19.245 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: Deploying daemon mon.b on smithi027 2024-04-05T23:27:19.245 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/1030178976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T23:27:19.245 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:19.245 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:19.245 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:18 smithi027 ceph-mon[78597]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-05T23:27:23.877 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.884+0000 7f16deffd700 1 -- 172.21.15.130:0/4177503516 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1272 (secure 0 0 0) 0x7f16e80172c0 con 0x7f16ec073720 2024-04-05T23:27:23.878 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-05T23:27:23.878 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":3,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","modified":"2024-04-05T23:27:18.857600Z","created":"2024-04-05T23:25:32.262742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16d80385e0 msgr2=0x7f16d803aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16d80385e0 0x7f16d803aaa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f16e0000c00 tx=0x7f16e000b580 comp rx=0 tx=0).stop 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16ec073720 msgr2=0x7f16ec073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16ec073720 0x7f16ec073ba0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f16e800b4d0 tx=0x7f16e800b890 comp rx=0 tx=0).stop 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 shutdown_connections 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16d80385e0 0x7f16d803aaa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ec106910 0x7f16ec0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 --2- 172.21.15.130:0/4177503516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16ec073720 0x7f16ec073ba0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 >> 172.21.15.130:0/4177503516 conn(0x7f16ec0fc440 msgr2=0x7f16ec102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.887+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 shutdown_connections 2024-04-05T23:27:23.881 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:23.888+0000 7f16f3af2700 1 -- 172.21.15.130:0/4177503516 wait complete. 2024-04-05T23:27:23.882 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 3 2024-04-05T23:27:24.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: mon.a calling monitor election 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: mon.c calling monitor election 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/4177503516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: mon.b calling monitor election 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: fsmap 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: mgrmap e13: x(active, since 46s) 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: overall HEALTH_OK 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:23 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:27:24.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: mon.a calling monitor election 2024-04-05T23:27:24.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: mon.c calling monitor election 2024-04-05T23:27:24.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:24.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/4177503516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T23:27:24.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:24.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: mon.b calling monitor election 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: fsmap 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: mgrmap e13: x(active, since 46s) 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: overall HEALTH_OK 2024-04-05T23:27:24.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:23 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.235 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-05T23:27:24.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-05T23:27:24.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:27:24.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: mon.a calling monitor election 2024-04-05T23:27:24.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: mon.c calling monitor election 2024-04-05T23:27:24.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:27:24.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/4177503516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T23:27:24.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:24.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: mon.b calling monitor election 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: fsmap 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: mgrmap e13: x(active, since 46s) 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: overall HEALTH_OK 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.300 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:23 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:24.590 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:27:25.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:25.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:25.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:25.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:25.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:25.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:25.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:25.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:25.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:25.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:25.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:25.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:27:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.775+0000 7f90a0c00700 1 -- 172.21.15.16:0/2411887309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909c073280 msgr2=0x7f909c073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.775+0000 7f90a0c00700 1 --2- 172.21.15.16:0/2411887309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909c073280 0x7f909c073660 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f908c009910 tx=0x7f908c009c20 comp rx=0 tx=0).stop 2024-04-05T23:27:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.775+0000 7f90a0c00700 1 -- 172.21.15.16:0/2411887309 shutdown_connections 2024-04-05T23:27:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.775+0000 7f90a0c00700 1 --2- 172.21.15.16:0/2411887309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909c073280 0x7f909c073660 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.775+0000 7f90a0c00700 1 -- 172.21.15.16:0/2411887309 >> 172.21.15.16:0/2411887309 conn(0x7f909c0fc2e0 msgr2=0x7f909c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.775+0000 7f90a0c00700 1 -- 172.21.15.16:0/2411887309 shutdown_connections 2024-04-05T23:27:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.775+0000 7f90a0c00700 1 -- 172.21.15.16:0/2411887309 wait complete. 2024-04-05T23:27:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.776+0000 7f90a0c00700 1 Processor -- start 2024-04-05T23:27:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.777+0000 7f90a0c00700 1 -- start start 2024-04-05T23:27:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.777+0000 7f90a0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c073280 0x7f909c1a2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.777+0000 7f90a0c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909c1a2990 0x7f909c19c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.777+0000 7f90a0c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f909c19cb30 0x7f909c19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.777+0000 7f90a0c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1a3050 con 0x7f909c1a2990 2024-04-05T23:27:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.777+0000 7f90a0c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1a9320 con 0x7f909c073280 2024-04-05T23:27:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.777+0000 7f90a0c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1a9460 con 0x7f909c19cb30 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.777+0000 7f9099e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c073280 0x7f909c1a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.778+0000 7f909a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f909c19cb30 0x7f909c19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.777+0000 7f9099400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909c1a2990 0x7f909c19c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.778+0000 7f9099e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c073280 0x7f909c1a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:40290/0 (socket says 172.21.15.16:40290) 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.778+0000 7f9099e00700 1 -- 172.21.15.16:0/1463348471 learned_addr learned my addr 172.21.15.16:0/1463348471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.778+0000 7f9099e00700 1 -- 172.21.15.16:0/1463348471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f909c19cb30 msgr2=0x7f909c19cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.778+0000 7f9099e00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f909c19cb30 0x7f909c19cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.778+0000 7f9099e00700 1 -- 172.21.15.16:0/1463348471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909c1a2990 msgr2=0x7f909c19c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.778+0000 7f9099e00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909c1a2990 0x7f909c19c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.778+0000 7f9099e00700 1 -- 172.21.15.16:0/1463348471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0095f0 con 0x7f909c073280 2024-04-05T23:27:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.778+0000 7f9099e00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c073280 0x7f909c1a2450 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f908c004c40 tx=0x7f908c004c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.779+0000 7f9092a00700 1 -- 172.21.15.16:0/1463348471 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c01d070 con 0x7f909c073280 2024-04-05T23:27:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.779+0000 7f9092a00700 1 -- 172.21.15.16:0/1463348471 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f908c005000 con 0x7f909c073280 2024-04-05T23:27:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.779+0000 7f9092a00700 1 -- 172.21.15.16:0/1463348471 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c022ce0 con 0x7f909c073280 2024-04-05T23:27:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.779+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c1a9740 con 0x7f909c073280 2024-04-05T23:27:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.779+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c1a9bf0 con 0x7f909c073280 2024-04-05T23:27:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.780+0000 7f9092a00700 1 -- 172.21.15.16:0/1463348471 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7f908c022470 con 0x7f909c073280 2024-04-05T23:27:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.780+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909c0749c0 con 0x7f909c073280 2024-04-05T23:27:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.780+0000 7f9092a00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f9080038c70 0x7f908003b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.780+0000 7f9092a00700 1 -- 172.21.15.16:0/1463348471 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f908c022e40 con 0x7f909c073280 2024-04-05T23:27:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.781+0000 7f9099400700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f9080038c70 0x7f908003b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:25.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.781+0000 7f9099400700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f9080038c70 0x7f908003b130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9088009ad0 tx=0x7f9088009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:25.786+0000 7f9092a00700 1 -- 172.21.15.16:0/1463348471 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f908c00ebd0 con 0x7f909c073280 2024-04-05T23:27:25.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.004+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f909c062510 con 0x7f909c073280 2024-04-05T23:27:25.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.005+0000 7f9092a00700 1 -- 172.21.15.16:0/1463348471 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7f908c00ebd0 con 0x7f909c073280 2024-04-05T23:27:25.998 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:27:25.999 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-04-05T23:27:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:27:25.999 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] 2024-04-05T23:27:26.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f9080038c70 msgr2=0x7f908003b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:26.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f9080038c70 0x7f908003b130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9088009ad0 tx=0x7f9088009210 comp rx=0 tx=0).stop 2024-04-05T23:27:26.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c073280 msgr2=0x7f909c1a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:26.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c073280 0x7f909c1a2450 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f908c004c40 tx=0x7f908c004c70 comp rx=0 tx=0).stop 2024-04-05T23:27:26.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 shutdown_connections 2024-04-05T23:27:26.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f9080038c70 0x7f908003b130 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:26.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f909c19cb30 0x7f909c19cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:26.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c073280 0x7f909c1a2450 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:26.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 --2- 172.21.15.16:0/1463348471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909c1a2990 0x7f909c19c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:26.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.007+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 >> 172.21.15.16:0/1463348471 conn(0x7f909c0fc2e0 msgr2=0x7f909c0fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:26.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.008+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 shutdown_connections 2024-04-05T23:27:26.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:26.008+0000 7f90a0c00700 1 -- 172.21.15.16:0/1463348471 wait complete. 2024-04-05T23:27:26.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:26.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:26.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-05T23:27:26.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-05T23:27:26.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-05T23:27:26.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:26.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:26.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:26.193 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:26.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:26.374 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-05T23:27:26.374 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:27:26.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-04-05T23:27:26.442 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:27:26.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:27:26.550 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:27:26.550 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-04-05T23:27:26.588 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:27:26.589 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:27:26.663 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:27:26.663 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.conf 2024-04-05T23:27:26.699 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:27:26.699 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:27:26.770 INFO:tasks.cephadm:Adding mgr.x on smithi016 2024-04-05T23:27:26.770 INFO:tasks.cephadm:Adding mgr.y on smithi027 2024-04-05T23:27:26.770 INFO:tasks.cephadm:Adding mgr.z on smithi130 2024-04-05T23:27:26.770 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch apply mgr '3;smithi016=x;smithi027=y;smithi130=z' 2024-04-05T23:27:27.026 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:27:27.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:27.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1463348471' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:27.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: Reconfiguring mon.a (unknown last config time)... 2024-04-05T23:27:27.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:27.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T23:27:27.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:27.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:26 smithi016 ceph-mon[38248]: Reconfiguring daemon mon.a on smithi016 2024-04-05T23:27:27.320 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:27.320 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.320 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1463348471' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: Reconfiguring mon.a (unknown last config time)... 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:27.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:26 smithi130 ceph-mon[79417]: Reconfiguring daemon mon.a on smithi016 2024-04-05T23:27:27.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:27:27.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1463348471' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: Reconfiguring mon.a (unknown last config time)... 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:27.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:26 smithi027 ceph-mon[78597]: Reconfiguring daemon mon.a on smithi016 2024-04-05T23:27:28.038 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.043+0000 7f00798be700 1 -- 172.21.15.130:0/3708315586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00741009a0 msgr2=0x7f0074100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:28.038 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.043+0000 7f00798be700 1 --2- 172.21.15.130:0/3708315586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00741009a0 0x7f0074100dc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0068009910 tx=0x7f0068009c20 comp rx=0 tx=0).stop 2024-04-05T23:27:28.038 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.043+0000 7f00798be700 1 -- 172.21.15.130:0/3708315586 shutdown_connections 2024-04-05T23:27:28.038 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.043+0000 7f00798be700 1 --2- 172.21.15.130:0/3708315586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00741009a0 0x7f0074100dc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:28.038 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.043+0000 7f00798be700 1 -- 172.21.15.130:0/3708315586 >> 172.21.15.130:0/3708315586 conn(0x7f00740fbf60 msgr2=0x7f00740fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:28.038 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.043+0000 7f00798be700 1 -- 172.21.15.130:0/3708315586 shutdown_connections 2024-04-05T23:27:28.038 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.043+0000 7f00798be700 1 -- 172.21.15.130:0/3708315586 wait complete. 2024-04-05T23:27:28.038 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.043+0000 7f00798be700 1 Processor -- start 2024-04-05T23:27:28.038 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00798be700 1 -- start start 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00798be700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00741009a0 0x7f00741980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00798be700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0074198600 0x7f007419d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00798be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0074198a80 0x7f007419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00798be700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00741a0350 con 0x7f0074198a80 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00798be700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00741a04c0 con 0x7f00741009a0 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00798be700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00741a0680 con 0x7f0074198600 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00727fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0074198600 0x7f007419d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00727fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0074198600 0x7f007419d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:44726/0 (socket says 172.21.15.130:44726) 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00727fc700 1 -- 172.21.15.130:0/151844906 learned_addr learned my addr 172.21.15.130:0/151844906 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00737fe700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0074198a80 0x7f007419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f0072ffd700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00741009a0 0x7f00741980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00727fc700 1 -- 172.21.15.130:0/151844906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00741009a0 msgr2=0x7f00741980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00727fc700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00741009a0 0x7f00741980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:28.039 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00727fc700 1 -- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0074198a80 msgr2=0x7f007419fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:28.040 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00727fc700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0074198a80 0x7f007419fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:28.040 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00727fc700 1 -- 172.21.15.130:0/151844906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00680095f0 con 0x7f0074198600 2024-04-05T23:27:28.040 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.044+0000 7f00727fc700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0074198600 0x7f007419d620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f006400b780 tx=0x7f006400bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:28.040 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.045+0000 7f00788bc700 1 -- 172.21.15.130:0/151844906 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064004230 con 0x7f0074198600 2024-04-05T23:27:28.040 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.045+0000 7f00788bc700 1 -- 172.21.15.130:0/151844906 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0064004390 con 0x7f0074198600 2024-04-05T23:27:28.040 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.045+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00741a0960 con 0x7f0074198600 2024-04-05T23:27:28.040 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.045+0000 7f00788bc700 1 -- 172.21.15.130:0/151844906 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064011b30 con 0x7f0074198600 2024-04-05T23:27:28.040 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.045+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00741a0ee0 con 0x7f0074198600 2024-04-05T23:27:28.041 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.045+0000 7f00788bc700 1 -- 172.21.15.130:0/151844906 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7f0064011cd0 con 0x7f0074198600 2024-04-05T23:27:28.041 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.046+0000 7f00788bc700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f005c038c70 0x7f005c03b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:28.041 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.046+0000 7f00788bc700 1 -- 172.21.15.130:0/151844906 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f006404add0 con 0x7f0074198600 2024-04-05T23:27:28.041 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.046+0000 7f0072ffd700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f005c038c70 0x7f005c03b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:28.041 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.046+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0074191ac0 con 0x7f0074198600 2024-04-05T23:27:28.041 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.047+0000 7f0072ffd700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f005c038c70 0x7f005c03b130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f00680098e0 tx=0x7f0068005200 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:28.044 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.051+0000 7f00788bc700 1 -- 172.21.15.130:0/151844906 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f00640193a0 con 0x7f0074198600 2024-04-05T23:27:28.245 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.252+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi027=y;smithi130=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f0074061320 con 0x7f005c038c70 2024-04-05T23:27:28.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.259+0000 7f00788bc700 1 -- 172.21.15.130:0/151844906 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0074061320 con 0x7f005c038c70 2024-04-05T23:27:28.252 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mgr update... 2024-04-05T23:27:28.254 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f005c038c70 msgr2=0x7f005c03b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f005c038c70 0x7f005c03b130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f00680098e0 tx=0x7f0068005200 comp rx=0 tx=0).stop 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0074198600 msgr2=0x7f007419d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0074198600 0x7f007419d620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f006400b780 tx=0x7f006400bb40 comp rx=0 tx=0).stop 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 shutdown_connections 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f005c038c70 0x7f005c03b130 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0074198a80 0x7f007419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00741009a0 0x7f00741980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 --2- 172.21.15.130:0/151844906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0074198600 0x7f007419d620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 >> 172.21.15.130:0/151844906 conn(0x7f00740fbf60 msgr2=0x7f00740fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 shutdown_connections 2024-04-05T23:27:28.255 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:27:28.261+0000 7f00798be700 1 -- 172.21.15.130:0/151844906 wait complete. 2024-04-05T23:27:28.459 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:28 smithi027 ceph-mon[78597]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:28.459 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.459 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.459 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.459 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:28.459 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T23:27:28.459 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:28.459 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:28 smithi016 ceph-mon[38248]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T23:27:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:28 smithi130 ceph-mon[79417]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:28.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:28.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T23:27:28.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:28.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:28.632 DEBUG:teuthology.orchestra.run.smithi027:mgr.y> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.y.service 2024-04-05T23:27:28.635 DEBUG:teuthology.orchestra.run.smithi130:mgr.z> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.z.service 2024-04-05T23:27:28.637 INFO:tasks.cephadm:Deploying OSDs... 2024-04-05T23:27:28.637 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:27:28.637 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-04-05T23:27:28.664 INFO:journalctl@ceph.mgr.y.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:27:28.666 INFO:journalctl@ceph.mgr.z.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:27:28.684 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-05T23:27:28.684 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-04-05T23:27:28.770 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-05T23:27:28.770 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:28.770 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 452 Links: 1 2024-04-05T23:27:28.770 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:28.770 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:28.770 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-05 23:26:48.360241649 +0000 2024-04-05T23:27:28.770 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-05 23:19:53.269769971 +0000 2024-04-05T23:27:28.770 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-05 23:19:53.269769971 +0000 2024-04-05T23:27:28.770 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-05 23:19:53.249770280 +0000 2024-04-05T23:27:28.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-05T23:27:28.876 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-05T23:27:28.876 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-05T23:27:28.876 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000184651 s, 2.8 MB/s 2024-04-05T23:27:28.878 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-05T23:27:28.923 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-04-05T23:27:29.005 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-05T23:27:29.006 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:29.006 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 458 Links: 1 2024-04-05T23:27:29.006 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:29.006 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:29.006 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-05 23:26:48.447240302 +0000 2024-04-05T23:27:29.006 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-05 23:19:53.754762479 +0000 2024-04-05T23:27:29.006 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-05 23:19:53.754762479 +0000 2024-04-05T23:27:29.006 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-05 23:19:53.733762804 +0000 2024-04-05T23:27:29.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-05T23:27:29.108 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-05T23:27:29.108 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-05T23:27:29.108 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000181543 s, 2.8 MB/s 2024-04-05T23:27:29.111 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-05T23:27:29.154 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-04-05T23:27:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-05T23:27:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:29.236 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 464 Links: 1 2024-04-05T23:27:29.236 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:29.236 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:29.236 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-05 23:26:48.530239018 +0000 2024-04-05T23:27:29.237 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-05 23:19:54.233755079 +0000 2024-04-05T23:27:29.237 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-05 23:19:54.233755079 +0000 2024-04-05T23:27:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-05 23:19:54.212755404 +0000 2024-04-05T23:27:29.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-05T23:27:29.341 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-05T23:27:29.341 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-05T23:27:29.341 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000158724 s, 3.2 MB/s 2024-04-05T23:27:29.343 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-05T23:27:29.385 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-04-05T23:27:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-05T23:27:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:29.468 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 470 Links: 1 2024-04-05T23:27:29.469 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:29.469 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:29.469 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-05 23:26:48.612237749 +0000 2024-04-05T23:27:29.469 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-05 23:19:54.711747695 +0000 2024-04-05T23:27:29.469 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-05 23:19:54.711747695 +0000 2024-04-05T23:27:29.469 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-05 23:19:54.687748066 +0000 2024-04-05T23:27:29.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-05T23:27:29.504 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:29 smithi130 ceph-mon[79417]: Reconfiguring mon.b (monmap changed)... 2024-04-05T23:27:29.504 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:29 smithi130 ceph-mon[79417]: Reconfiguring daemon mon.b on smithi027 2024-04-05T23:27:29.504 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:29 smithi130 ceph-mon[79417]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi027=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:29.504 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:29 smithi130 ceph-mon[79417]: Saving service mgr spec with placement smithi016=x;smithi027=y;smithi130=z;count:3 2024-04-05T23:27:29.504 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:29.504 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:29.504 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:29.504 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T23:27:29.504 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:29.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:29 smithi016 ceph-mon[38248]: Reconfiguring mon.b (monmap changed)... 2024-04-05T23:27:29.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:29 smithi016 ceph-mon[38248]: Reconfiguring daemon mon.b on smithi027 2024-04-05T23:27:29.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:29 smithi016 ceph-mon[38248]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi027=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:29.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:29 smithi016 ceph-mon[38248]: Saving service mgr spec with placement smithi016=x;smithi027=y;smithi130=z;count:3 2024-04-05T23:27:29.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:29.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:29.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:29.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T23:27:29.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-05T23:27:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-05T23:27:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000162561 s, 3.1 MB/s 2024-04-05T23:27:29.577 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-05T23:27:29.620 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-05T23:27:29.620 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-04-05T23:27:29.642 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-05T23:27:29.642 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-04-05T23:27:29.705 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-05T23:27:29.705 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:29.705 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 210332 Links: 1 2024-04-05T23:27:29.706 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:29.706 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:29.706 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-05 23:27:18.178491042 +0000 2024-04-05T23:27:29.706 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-05 23:18:36.100630426 +0000 2024-04-05T23:27:29.706 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-05 23:18:36.100630426 +0000 2024-04-05T23:27:29.706 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-04-05T23:27:29.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-05T23:27:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:29 smithi027 ceph-mon[78597]: Reconfiguring mon.b (monmap changed)... 2024-04-05T23:27:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:29 smithi027 ceph-mon[78597]: Reconfiguring daemon mon.b on smithi027 2024-04-05T23:27:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:29 smithi027 ceph-mon[78597]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi027=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:29 smithi027 ceph-mon[78597]: Saving service mgr spec with placement smithi016=x;smithi027=y;smithi130=z;count:3 2024-04-05T23:27:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T23:27:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T23:27:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:29.748 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-05T23:27:29.749 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-05T23:27:29.749 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000218035 s, 2.3 MB/s 2024-04-05T23:27:29.750 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-05T23:27:29.811 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-04-05T23:27:29.871 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-05T23:27:29.871 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:29.871 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 212188 Links: 1 2024-04-05T23:27:29.872 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:29.872 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:29.872 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-05 23:27:18.178491042 +0000 2024-04-05T23:27:29.872 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-05 23:18:36.460621181 +0000 2024-04-05T23:27:29.872 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-05 23:18:36.460621181 +0000 2024-04-05T23:27:29.872 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-04-05T23:27:29.872 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-05T23:27:29.942 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-05T23:27:29.943 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-05T23:27:29.943 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000245338 s, 2.1 MB/s 2024-04-05T23:27:29.943 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-05T23:27:30.003 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-04-05T23:27:30.063 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-05T23:27:30.063 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:30.063 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 212272 Links: 1 2024-04-05T23:27:30.063 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:30.063 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:30.063 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-05 23:27:18.178491042 +0000 2024-04-05T23:27:30.063 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-05 23:18:36.808612244 +0000 2024-04-05T23:27:30.063 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-05 23:18:36.808612244 +0000 2024-04-05T23:27:30.063 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-04-05T23:27:30.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-05T23:27:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-05T23:27:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-05T23:27:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000183541 s, 2.8 MB/s 2024-04-05T23:27:30.132 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-05T23:27:30.192 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-04-05T23:27:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-05T23:27:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:30.252 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 212330 Links: 1 2024-04-05T23:27:30.252 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:30.252 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:30.252 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-05 23:27:18.178491042 +0000 2024-04-05T23:27:30.252 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-05 23:18:37.164603101 +0000 2024-04-05T23:27:30.252 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-05 23:18:37.164603101 +0000 2024-04-05T23:27:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-04-05T23:27:30.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-05T23:27:30.320 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-05T23:27:30.320 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-05T23:27:30.320 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000234463 s, 2.2 MB/s 2024-04-05T23:27:30.321 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-05T23:27:30.381 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-05T23:27:30.381 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2024-04-05T23:27:30.398 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-05T23:27:30.398 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_1 2024-04-05T23:27:30.455 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-05T23:27:30.455 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:30.456 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 206437 Links: 1 2024-04-05T23:27:30.456 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:30.456 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:30.456 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-05 23:27:10.284145430 +0000 2024-04-05T23:27:30.456 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-05 23:18:19.270536873 +0000 2024-04-05T23:27:30.456 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-05 23:18:19.270536873 +0000 2024-04-05T23:27:30.456 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2024-04-05T23:27:30.456 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-05T23:27:30.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: Reconfiguring mon.c (monmap changed)... 2024-04-05T23:27:30.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: Reconfiguring daemon mon.c on smithi130 2024-04-05T23:27:30.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:30.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:30.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:30.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:30.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:30.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:30.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:30.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T23:27:30.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-05T23:27:30.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T23:27:30.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: Reconfiguring mon.c (monmap changed)... 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: Reconfiguring daemon mon.c on smithi130 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T23:27:30.508 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: Reconfiguring mon.c (monmap changed)... 2024-04-05T23:27:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: Reconfiguring daemon mon.c on smithi130 2024-04-05T23:27:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:30.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:30.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:30.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:30.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T23:27:30.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-05T23:27:30.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T23:27:30.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:30.608 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-05T23:27:30.608 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-05T23:27:30.608 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000280219 s, 1.8 MB/s 2024-04-05T23:27:30.609 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-05T23:27:30.671 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_2 2024-04-05T23:27:30.729 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-05T23:27:30.729 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:30.730 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 209683 Links: 1 2024-04-05T23:27:30.730 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:30.730 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:30.730 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-05 23:27:30.693777753 +0000 2024-04-05T23:27:30.730 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-05 23:18:19.609530787 +0000 2024-04-05T23:27:30.730 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-05 23:18:19.609530787 +0000 2024-04-05T23:27:30.730 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2024-04-05T23:27:30.731 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-05T23:27:30.799 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:30 smithi130 systemd[1]: Starting Ceph mgr.z for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:27:30.802 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-05T23:27:30.802 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-05T23:27:30.802 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000208304 s, 2.5 MB/s 2024-04-05T23:27:30.803 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-05T23:27:30.864 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_3 2024-04-05T23:27:30.922 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-05T23:27:30.922 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:30.923 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 208740 Links: 1 2024-04-05T23:27:30.923 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:30.923 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:30.923 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-05 23:27:30.693777753 +0000 2024-04-05T23:27:30.923 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-05 23:18:19.957524539 +0000 2024-04-05T23:27:30.923 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-05 23:18:19.957524539 +0000 2024-04-05T23:27:30.923 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2024-04-05T23:27:30.923 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-05T23:27:30.989 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-05T23:27:30.989 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-05T23:27:30.989 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000211505 s, 2.4 MB/s 2024-04-05T23:27:30.990 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-05T23:27:31.048 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_4 2024-04-05T23:27:31.107 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-05T23:27:31.107 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T23:27:31.107 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 208816 Links: 1 2024-04-05T23:27:31.107 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:27:31.107 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T23:27:31.107 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-05 23:27:30.693777753 +0000 2024-04-05T23:27:31.107 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-05 23:18:20.303518327 +0000 2024-04-05T23:27:31.107 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-05 23:18:20.303518327 +0000 2024-04-05T23:27:31.108 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2024-04-05T23:27:31.108 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-05T23:27:31.174 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-05T23:27:31.174 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-05T23:27:31.174 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000230037 s, 2.2 MB/s 2024-04-05T23:27:31.175 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-05T23:27:31.234 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-04-05T23:27:31.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-05T23:27:31.520 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-mgr[81477]: -- 172.21.15.130:0/3574683526 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d67f3ba1a0 con 0x55d67f390800 2024-04-05T23:27:31.520 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:31 smithi130 bash[81349]: 843859955bac1e333471594e883c27c9124dd541072f3f709119ef9bc022e40e 2024-04-05T23:27:31.520 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:31 smithi130 systemd[1]: Started Ceph mgr.z for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:27:31.521 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-mon[79417]: Deploying daemon mgr.z on smithi130 2024-04-05T23:27:31.521 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:31.521 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:31.521 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:31.521 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T23:27:31.521 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-05T23:27:31.521 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T23:27:31.521 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:31 smithi016 ceph-mon[38248]: Deploying daemon mgr.z on smithi130 2024-04-05T23:27:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T23:27:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-05T23:27:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T23:27:31.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:31.599 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:27:31.613 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:31.526+0000 7fe5d7192200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-05T23:27:31.637 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:31 smithi027 ceph-mon[78597]: Deploying daemon mgr.z on smithi130 2024-04-05T23:27:31.637 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:31.637 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:31.637 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:31.637 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T23:27:31.637 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-05T23:27:31.637 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T23:27:31.637 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:31.783 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:31.619+0000 7fe5d7192200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-05T23:27:31.784 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:31.707+0000 7fe5d7192200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-05T23:27:31.784 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:31.790+0000 7fe5d7192200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-05T23:27:32.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:32 smithi016 ceph-mon[38248]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:32.552 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:32 smithi016 ceph-mon[38248]: Deploying daemon mgr.y on smithi027 2024-04-05T23:27:32.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:32 smithi130 ceph-mon[79417]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:32.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:32 smithi130 ceph-mon[79417]: Deploying daemon mgr.y on smithi027 2024-04-05T23:27:32.585 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:32.426+0000 7fe5d7192200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:27:32.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:32 smithi027 ceph-mon[78597]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:32.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:32 smithi027 ceph-mon[78597]: Deploying daemon mgr.y on smithi027 2024-04-05T23:27:32.717 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:32 smithi027 systemd[1]: Starting Ceph mgr.y for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:27:33.022 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:32.677+0000 7fe5d7192200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-05T23:27:33.022 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:32.762+0000 7fe5d7192200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-05T23:27:33.022 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:32.869+0000 7fe5d7192200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-05T23:27:33.022 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:33 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:33.028+0000 7fe5d7192200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-05T23:27:33.334 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:33 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:33.156+0000 7fe5d7192200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-05T23:27:33.335 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:33 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:33.241+0000 7fe5d7192200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-05T23:27:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-05T23:27:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-05T23:27:33.382 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-05T23:27:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-05T23:27:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0327034 s, 321 MB/s 2024-04-05T23:27:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-05T23:27:33.409 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:33 smithi027 ceph-mgr[80070]: -- 172.21.15.27:0/152165330 <== mon.2 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e50046c1a0 con 0x55e500442800 2024-04-05T23:27:33.409 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:33 smithi027 bash[80032]: 8a6b96cd55488daba30a61ae7844f1c6ac74cbcf0d8b84dac430b512707c0b3f 2024-04-05T23:27:33.409 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:33 smithi027 systemd[1]: Started Ceph mgr.y for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:27:33.508 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:33 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:33.415+0000 7f65fbe84200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-05T23:27:33.717 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:33 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:33.515+0000 7f65fbe84200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-05T23:27:33.717 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:33 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:33.623+0000 7f65fbe84200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-05T23:27:33.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-04-05T23:27:33.967 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:33 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:33.733+0000 7f65fbe84200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-05T23:27:34.084 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:33 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:33.651+0000 7fe5d7192200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-05T23:27:34.142 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:27:34.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:34 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:34 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:34 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:34 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:34 smithi016 ceph-mon[38248]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:34.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:34 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:34.386 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:34 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:34.098+0000 7fe5d7192200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-05T23:27:34.386 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:34 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:34.299+0000 7fe5d7192200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-05T23:27:34.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:34 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:34 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:34 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:34 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:34 smithi130 ceph-mon[79417]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:34.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:34 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:34.429 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.429 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.429 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.429 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:34.429 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-mon[78597]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:34.429 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:34.689 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:34.435+0000 7f65fbe84200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:27:34.834 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:34 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:34.393+0000 7fe5d7192200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-05T23:27:34.967 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:34.696+0000 7f65fbe84200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-05T23:27:34.967 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:34.785+0000 7f65fbe84200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-05T23:27:34.967 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:34 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:34.898+0000 7f65fbe84200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-05T23:27:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:35.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:35.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T23:27:35.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T23:27:35.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.467+0000 7fc5a5600700 1 -- 172.21.15.16:0/1723079513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5a0100ef0 msgr2=0x7fc5a0101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.467+0000 7fc5a5600700 1 --2- 172.21.15.16:0/1723079513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5a0100ef0 0x7fc5a0101330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc590009990 tx=0x7fc590009ca0 comp rx=0 tx=0).stop 2024-04-05T23:27:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.468+0000 7fc5a5600700 1 -- 172.21.15.16:0/1723079513 shutdown_connections 2024-04-05T23:27:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.468+0000 7fc5a5600700 1 --2- 172.21.15.16:0/1723079513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5a0100ef0 0x7fc5a0101330 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.468+0000 7fc5a5600700 1 --2- 172.21.15.16:0/1723079513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5a0102a90 0x7fc5a01093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.468+0000 7fc5a5600700 1 --2- 172.21.15.16:0/1723079513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5a01020d0 0x7fc5a0102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.468+0000 7fc5a5600700 1 -- 172.21.15.16:0/1723079513 >> 172.21.15.16:0/1723079513 conn(0x7fc5a00fc4e0 msgr2=0x7fc5a00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.468+0000 7fc5a5600700 1 -- 172.21.15.16:0/1723079513 shutdown_connections 2024-04-05T23:27:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.468+0000 7fc5a5600700 1 -- 172.21.15.16:0/1723079513 wait complete. 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.469+0000 7fc5a5600700 1 Processor -- start 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.469+0000 7fc5a5600700 1 -- start start 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.469+0000 7fc5a5600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5a0100ef0 0x7fc5a0196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.469+0000 7fc5a5600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5a01020d0 0x7fc5a0197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.469+0000 7fc5a5600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5a0102a90 0x7fc5a019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.469+0000 7fc5a5600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a019e9d0 con 0x7fc5a01020d0 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.469+0000 7fc5a5600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a019eb10 con 0x7fc5a0100ef0 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.469+0000 7fc5a5600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a019eca0 con 0x7fc5a0102a90 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5a0100ef0 0x7fc5a0196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5a0100ef0 0x7fc5a0196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:43330/0 (socket says 172.21.15.16:43330) 2024-04-05T23:27:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59ea00700 1 -- 172.21.15.16:0/3886943975 learned_addr learned my addr 172.21.15.16:0/3886943975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59f400700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5a0102a90 0x7fc5a019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59e000700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5a01020d0 0x7fc5a0197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59ea00700 1 -- 172.21.15.16:0/3886943975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5a0102a90 msgr2=0x7fc5a019e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59ea00700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5a0102a90 0x7fc5a019e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59ea00700 1 -- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5a01020d0 msgr2=0x7fc5a0197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59ea00700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5a01020d0 0x7fc5a0197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59ea00700 1 -- 172.21.15.16:0/3886943975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5900095f0 con 0x7fc5a0100ef0 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59e000700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5a01020d0 0x7fc5a0197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc59ea00700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5a0100ef0 0x7fc5a0196ee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc590000c00 tx=0x7fc590009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc597400700 1 -- 172.21.15.16:0/3886943975 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59001d070 con 0x7fc5a0100ef0 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.470+0000 7fc597400700 1 -- 172.21.15.16:0/3886943975 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc590021410 con 0x7fc5a0100ef0 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.471+0000 7fc597400700 1 -- 172.21.15.16:0/3886943975 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc590017650 con 0x7fc5a0100ef0 2024-04-05T23:27:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.471+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a019ef20 con 0x7fc5a0100ef0 2024-04-05T23:27:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.471+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a019f280 con 0x7fc5a0100ef0 2024-04-05T23:27:35.467 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:35.062+0000 7f65fbe84200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-05T23:27:35.467 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:35.195+0000 7f65fbe84200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:35.284+0000 7f65fbe84200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T23:27:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T23:27:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.472+0000 7fc597400700 1 -- 172.21.15.16:0/3886943975 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 45597+0+0 (secure 0 0 0) 0x7fc590021580 con 0x7fc5a0100ef0 2024-04-05T23:27:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.472+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a0108480 con 0x7fc5a0100ef0 2024-04-05T23:27:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.472+0000 7fc597400700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc5840389d0 0x7fc58403ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.472+0000 7fc597400700 1 -- 172.21.15.16:0/3886943975 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc59004c600 con 0x7fc5a0100ef0 2024-04-05T23:27:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.473+0000 7fc59e000700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc5840389d0 0x7fc58403ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.473+0000 7fc59e000700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc5840389d0 0x7fc58403ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009cb0 tx=0x7fc58c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:35.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.478+0000 7fc597400700 1 -- 172.21.15.16:0/3886943975 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc590029540 con 0x7fc5a0100ef0 2024-04-05T23:27:35.584 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:35.160+0000 7fe5d7192200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:35.257+0000 7fe5d7192200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:35.404+0000 7fe5d7192200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T23:27:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T23:27:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:35.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:35.704+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5a002ce80 con 0x7fc5840389d0 2024-04-05T23:27:35.967 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:35 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:35.674+0000 7f65fbe84200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-05T23:27:36.335 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:36.037+0000 7fe5d7192200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-05T23:27:36.335 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:36.221+0000 7fe5d7192200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-05T23:27:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: Reconfiguring mgr.x (unknown last config time)... 2024-04-05T23:27:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: Reconfiguring daemon mgr.x on smithi016 2024-04-05T23:27:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:27:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:27:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:36.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:36.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:36.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:36.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:36.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:36.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:36.371 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:36.079+0000 7f65fbe84200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-05T23:27:36.371 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:36.281+0000 7f65fbe84200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-05T23:27:36.371 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: Reconfiguring mgr.x (unknown last config time)... 2024-04-05T23:27:36.371 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: Reconfiguring daemon mgr.x on smithi016 2024-04-05T23:27:36.371 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:36.371 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:27:36.372 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:27:36.372 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:36.372 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:36.372 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:36.372 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:36.372 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:36.372 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:36.372 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: Reconfiguring mgr.x (unknown last config time)... 2024-04-05T23:27:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: Reconfiguring daemon mgr.x on smithi016 2024-04-05T23:27:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:36.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:27:36.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:27:36.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:36.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:36.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:36.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:36.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:36.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:36.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:36.717 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:36.378+0000 7f65fbe84200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-05T23:27:36.834 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:36 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:36.415+0000 7fe5d7192200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-05T23:27:37.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:37 smithi016 ceph-mon[38248]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:37.424 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:37 smithi027 ceph-mon[78597]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:37.425 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:37 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:37.175+0000 7f65fbe84200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-05T23:27:37.425 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:37 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:37.276+0000 7f65fbe84200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-05T23:27:37.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:37 smithi130 ceph-mon[79417]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:37.585 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:37 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:37.349+0000 7fe5d7192200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-05T23:27:37.717 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:37 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:37.430+0000 7f65fbe84200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-05T23:27:37.905 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:37 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:37.634+0000 7fe5d7192200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:27:37.906 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:37 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:37.721+0000 7fe5d7192200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-05T23:27:38.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:38.297+0000 7fc597400700 1 -- 172.21.15.16:0/3886943975 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 14) v1 ==== 90707+0+0 (secure 0 0 0) 0x7fc590014d40 con 0x7fc5a0100ef0 2024-04-05T23:27:38.294 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:27:37 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:27:37.912+0000 7fe5d7192200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-05T23:27:38.467 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:38.086+0000 7f65fbe84200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-05T23:27:38.467 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:38.275+0000 7f65fbe84200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-05T23:27:38.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:38.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:38.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: Standby manager daemon z started 2024-04-05T23:27:38.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-05T23:27:38.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:27:38.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-05T23:27:38.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:27:38.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2639566522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fd073ed-664a-4b48-912a-3a2b1708c3a2"}]: dispatch 2024-04-05T23:27:38.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2639566522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5fd073ed-664a-4b48-912a-3a2b1708c3a2"}]': finished 2024-04-05T23:27:38.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: osdmap e5: 1 total, 0 up, 1 in 2024-04-05T23:27:38.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: Standby manager daemon z started 2024-04-05T23:27:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-05T23:27:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:27:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-05T23:27:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:27:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2639566522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fd073ed-664a-4b48-912a-3a2b1708c3a2"}]: dispatch 2024-04-05T23:27:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2639566522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5fd073ed-664a-4b48-912a-3a2b1708c3a2"}]': finished 2024-04-05T23:27:38.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: osdmap e5: 1 total, 0 up, 1 in 2024-04-05T23:27:38.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:38.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:38.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:38.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: Standby manager daemon z started 2024-04-05T23:27:38.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-05T23:27:38.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:27:38.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-05T23:27:38.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.130:0/829113691' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:27:38.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2639566522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fd073ed-664a-4b48-912a-3a2b1708c3a2"}]: dispatch 2024-04-05T23:27:38.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2639566522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5fd073ed-664a-4b48-912a-3a2b1708c3a2"}]': finished 2024-04-05T23:27:38.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: osdmap e5: 1 total, 0 up, 1 in 2024-04-05T23:27:38.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:38.967 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:38 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:38.478+0000 7f65fbe84200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-05T23:27:39.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:39 smithi016 ceph-mon[38248]: mgrmap e14: x(active, since 61s), standbys: z 2024-04-05T23:27:39.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:39 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-05T23:27:39.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:39 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3953856983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:27:39.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:39 smithi130 ceph-mon[79417]: mgrmap e14: x(active, since 61s), standbys: z 2024-04-05T23:27:39.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:39 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-05T23:27:39.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:39 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3953856983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:27:39.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:39 smithi027 ceph-mon[78597]: mgrmap e14: x(active, since 61s), standbys: z 2024-04-05T23:27:39.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:39 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-05T23:27:39.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:39 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3953856983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:27:39.717 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:39 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:39.455+0000 7f65fbe84200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-05T23:27:40.039 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:39 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:39.753+0000 7f65fbe84200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:27:40.039 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:39 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:39.845+0000 7f65fbe84200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-05T23:27:40.039 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:27:40 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:27:40.045+0000 7f65fbe84200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-05T23:27:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:40.320+0000 7fc597400700 1 -- 172.21.15.16:0/3886943975 <== mon.2 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 15) v1 ==== 135817+0+0 (secure 0 0 0) 0x7fc590020d00 con 0x7fc5a0100ef0 2024-04-05T23:27:40.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:40 smithi016 ceph-mon[38248]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:40.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:40 smithi016 ceph-mon[38248]: Standby manager daemon y started 2024-04-05T23:27:40.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:40 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-05T23:27:40.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:40 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:27:40.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:40 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-05T23:27:40.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:40 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:27:40.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:40 smithi130 ceph-mon[79417]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:40.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:40 smithi130 ceph-mon[79417]: Standby manager daemon y started 2024-04-05T23:27:40.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:40 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-05T23:27:40.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:40 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:27:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:40 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-05T23:27:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:40 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:27:40.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:40 smithi027 ceph-mon[78597]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:40.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:40 smithi027 ceph-mon[78597]: Standby manager daemon y started 2024-04-05T23:27:40.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:40 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-05T23:27:40.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:40 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:27:40.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:40 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-05T23:27:40.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:40 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.27:0/3105171675' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:27:41.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:41 smithi016 ceph-mon[38248]: mgrmap e15: x(active, since 63s), standbys: y, z 2024-04-05T23:27:41.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:41 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-05T23:27:41.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:41 smithi130 ceph-mon[79417]: mgrmap e15: x(active, since 63s), standbys: y, z 2024-04-05T23:27:41.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:41 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-05T23:27:41.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:41 smithi027 ceph-mon[78597]: mgrmap e15: x(active, since 63s), standbys: y, z 2024-04-05T23:27:41.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:41 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-05T23:27:42.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:42 smithi016 ceph-mon[38248]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:42.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:42 smithi130 ceph-mon[79417]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:42 smithi027 ceph-mon[78597]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:44.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:44 smithi016 ceph-mon[38248]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:44.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:44 smithi130 ceph-mon[79417]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:44.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:44 smithi027 ceph-mon[78597]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:45.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T23:27:45.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:45.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T23:27:45.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T23:27:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:46.572 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:46 smithi016 ceph-mon[38248]: Deploying daemon osd.0 on smithi016 2024-04-05T23:27:46.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:46 smithi016 ceph-mon[38248]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:46.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:46 smithi130 ceph-mon[79417]: Deploying daemon osd.0 on smithi016 2024-04-05T23:27:46.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:46 smithi130 ceph-mon[79417]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:46 smithi027 ceph-mon[78597]: Deploying daemon osd.0 on smithi016 2024-04-05T23:27:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:46 smithi027 ceph-mon[78597]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:48.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:48 smithi016 ceph-mon[38248]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:48.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:48 smithi130 ceph-mon[79417]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:48.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:48 smithi027 ceph-mon[78597]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:50.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:50 smithi016 ceph-mon[38248]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:50.497 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:50.498 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:50.498 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:50.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:50 smithi130 ceph-mon[79417]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:50.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:50.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:50.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:50.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:50 smithi027 ceph-mon[78597]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:50.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:50.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:50.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:52.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:52 smithi016 ceph-mon[38248]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:52.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:52.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:52.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:52.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:52.542 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:52 smithi130 ceph-mon[79417]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:52.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.605+0000 7fc597400700 1 -- 172.21.15.16:0/3886943975 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc5a002ce80 con 0x7fc5840389d0 2024-04-05T23:27:52.598 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-04-05T23:27:52.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.607+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc5840389d0 msgr2=0x7fc58403ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:52.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.607+0000 7fc5a5600700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc5840389d0 0x7fc58403ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009cb0 tx=0x7fc58c009260 comp rx=0 tx=0).stop 2024-04-05T23:27:52.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.607+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5a0100ef0 msgr2=0x7fc5a0196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.607+0000 7fc5a5600700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5a0100ef0 0x7fc5a0196ee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc590000c00 tx=0x7fc590009580 comp rx=0 tx=0).stop 2024-04-05T23:27:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.608+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 shutdown_connections 2024-04-05T23:27:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.608+0000 7fc5a5600700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc5840389d0 0x7fc58403ae90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.608+0000 7fc5a5600700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5a0102a90 0x7fc5a019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.608+0000 7fc5a5600700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5a0100ef0 0x7fc5a0196ee0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.608+0000 7fc5a5600700 1 --2- 172.21.15.16:0/3886943975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5a01020d0 0x7fc5a0197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.608+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 >> 172.21.15.16:0/3886943975 conn(0x7fc5a00fc4e0 msgr2=0x7fc5a01055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.608+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 shutdown_connections 2024-04-05T23:27:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:27:52.608+0000 7fc5a5600700 1 -- 172.21.15.16:0/3886943975 wait complete. 2024-04-05T23:27:52.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:52 smithi027 ceph-mon[78597]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:52.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:52.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:52.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:52.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:52.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:52.994 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-05T23:27:53.000 INFO:tasks.cephadm:Deploying osd.1 on smithi027 with /dev/vg_nvme/lv_4... 2024-04-05T23:27:53.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-05T23:27:53.078 INFO:journalctl@ceph.osd.0.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:27:53.238 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:27:53.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:53.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:53.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:53.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:53 smithi027 ceph-mon[78597]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:53.757 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:53.757 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:53.757 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:53.757 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:53 smithi016 ceph-mon[38248]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:53.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:53.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:53.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:27:53.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:53 smithi130 ceph-mon[79417]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:54.008 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:27:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-0[57079]: 2024-04-05T23:27:53.810+0000 7fdfaa1b37c0 -1 osd.0 0 log_to_monitors true 2024-04-05T23:27:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-05T23:27:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-05T23:27:54.619 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-05T23:27:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-05T23:27:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0159356 s, 658 MB/s 2024-04-05T23:27:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-05T23:27:54.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:54 smithi016 ceph-mon[38248]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T23:27:54.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:54 smithi130 ceph-mon[79417]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T23:27:54.939 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:54 smithi027 ceph-mon[78597]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T23:27:54.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi027:vg_nvme/lv_4 2024-04-05T23:27:55.181 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:27:55.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:55 smithi016 ceph-mon[38248]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T23:27:55.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:55 smithi016 ceph-mon[38248]: osdmap e6: 1 total, 0 up, 1 in 2024-04-05T23:27:55.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:55 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:55.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:55 smithi016 ceph-mon[38248]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:27:55.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:55 smithi016 ceph-mon[38248]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:55.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:55 smithi027 ceph-mon[78597]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T23:27:55.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:55 smithi027 ceph-mon[78597]: osdmap e6: 1 total, 0 up, 1 in 2024-04-05T23:27:55.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:55 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:55.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:55 smithi027 ceph-mon[78597]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:27:55.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:55 smithi027 ceph-mon[78597]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:56.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:55 smithi130 ceph-mon[79417]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T23:27:56.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:55 smithi130 ceph-mon[79417]: osdmap e6: 1 total, 0 up, 1 in 2024-04-05T23:27:56.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:55 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:56.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:55 smithi130 ceph-mon[79417]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:27:56.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:55 smithi130 ceph-mon[79417]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.149+0000 7fbf29464700 1 -- 172.21.15.27:0/2851564203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf24106cb0 msgr2=0x7fbf241090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.149+0000 7fbf29464700 1 --2- 172.21.15.27:0/2851564203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf24106cb0 0x7fbf241090a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c00b0a0 tx=0x7fbf1c00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:27:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.149+0000 7fbf29464700 1 -- 172.21.15.27:0/2851564203 shutdown_connections 2024-04-05T23:27:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.149+0000 7fbf29464700 1 --2- 172.21.15.27:0/2851564203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf24106cb0 0x7fbf241090a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.149+0000 7fbf29464700 1 --2- 172.21.15.27:0/2851564203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf24101a50 0x7fbf24103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.149+0000 7fbf29464700 1 --2- 172.21.15.27:0/2851564203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf24104380 0x7fbf24106770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.149+0000 7fbf29464700 1 -- 172.21.15.27:0/2851564203 >> 172.21.15.27:0/2851564203 conn(0x7fbf240fb3e0 msgr2=0x7fbf240fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.149+0000 7fbf29464700 1 -- 172.21.15.27:0/2851564203 shutdown_connections 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.149+0000 7fbf29464700 1 -- 172.21.15.27:0/2851564203 wait complete. 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf29464700 1 Processor -- start 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf29464700 1 -- start start 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf29464700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf24101a50 0x7fbf24196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf29464700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf24104380 0x7fbf24197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf29464700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf24106cb0 0x7fbf2419e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf29464700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2419eae0 con 0x7fbf24104380 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf29464700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2419ec20 con 0x7fbf24101a50 2024-04-05T23:27:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf29464700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2419edb0 con 0x7fbf24106cb0 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf227fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf24104380 0x7fbf24197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf227fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf24104380 0x7fbf24197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.27:42176/0 (socket says 172.21.15.27:42176) 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf227fc700 1 -- 172.21.15.27:0/1694715985 learned_addr learned my addr 172.21.15.27:0/1694715985 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf237fe700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf24106cb0 0x7fbf2419e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.150+0000 7fbf22ffd700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf24101a50 0x7fbf24196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf227fc700 1 -- 172.21.15.27:0/1694715985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf24106cb0 msgr2=0x7fbf2419e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf227fc700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf24106cb0 0x7fbf2419e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf227fc700 1 -- 172.21.15.27:0/1694715985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf24101a50 msgr2=0x7fbf24196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf227fc700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf24101a50 0x7fbf24196ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf227fc700 1 -- 172.21.15.27:0/1694715985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c009d00 con 0x7fbf24104380 2024-04-05T23:27:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf22ffd700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf24101a50 0x7fbf24196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:27:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf227fc700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf24104380 0x7fbf24197530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00d980 tx=0x7fbf0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf1bfff700 1 -- 172.21.15.27:0/1694715985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c0098d0 con 0x7fbf24104380 2024-04-05T23:27:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2419f090 con 0x7fbf24104380 2024-04-05T23:27:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2419f5e0 con 0x7fbf24104380 2024-04-05T23:27:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf1bfff700 1 -- 172.21.15.27:0/1694715985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf0c010460 con 0x7fbf24104380 2024-04-05T23:27:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.151+0000 7fbf1bfff700 1 -- 172.21.15.27:0/1694715985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c010ea0 con 0x7fbf24104380 2024-04-05T23:27:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.152+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf240fcf60 con 0x7fbf24104380 2024-04-05T23:27:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.154+0000 7fbf1bfff700 1 -- 172.21.15.27:0/1694715985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 135817+0+0 (secure 0 0 0) 0x7fbf0c0105d0 con 0x7fbf24104380 2024-04-05T23:27:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.154+0000 7fbf1bfff700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf100a1090 0x7fbf100a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:27:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.154+0000 7fbf1bfff700 1 -- 172.21.15.27:0/1694715985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1487+0+0 (secure 0 0 0) 0x7fbf0c0cb050 con 0x7fbf24104380 2024-04-05T23:27:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.155+0000 7fbf22ffd700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf100a1090 0x7fbf100a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:27:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.155+0000 7fbf22ffd700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf100a1090 0x7fbf100a3550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009de0 tx=0x7fbf14009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:27:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.159+0000 7fbf1bfff700 1 -- 172.21.15.27:0/1694715985 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbf0c0962d0 con 0x7fbf24104380 2024-04-05T23:27:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:27:56.372+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf2404ebb0 con 0x7fbf100a1090 2024-04-05T23:27:56.833 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-0[57079]: 2024-04-05T23:27:56.496+0000 7fdf9de00700 -1 osd.0 0 waiting for initial osdmap 2024-04-05T23:27:56.833 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-0[57079]: 2024-04-05T23:27:56.515+0000 7fdf94600700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:27:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: osdmap e7: 1 total, 0 up, 1 in 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi016 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi016 to 17274M 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:56.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:56.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:27:56.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:27:56.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:56.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:56 smithi016 ceph-mon[38248]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' 2024-04-05T23:27:56.854 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:27:56.854 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: osdmap e7: 1 total, 0 up, 1 in 2024-04-05T23:27:56.854 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:56.854 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:56.854 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi016 2024-04-05T23:27:56.854 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:56.854 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:56.854 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi016 to 17274M 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:56.855 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:56 smithi027 ceph-mon[78597]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' 2024-04-05T23:27:57.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:27:57.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: osdmap e7: 1 total, 0 up, 1 in 2024-04-05T23:27:57.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:57.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:57.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi016 2024-04-05T23:27:57.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:57.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:57.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi016 to 17274M 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:27:57.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:56 smithi130 ceph-mon[79417]: from='osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669]' entity='osd.0' 2024-04-05T23:27:57.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:57 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:27:57.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:57 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:27:57.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:57 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:57.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:57 smithi027 ceph-mon[78597]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:57.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:57 smithi027 ceph-mon[78597]: osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] boot 2024-04-05T23:27:57.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:57 smithi027 ceph-mon[78597]: osdmap e8: 1 total, 1 up, 1 in 2024-04-05T23:27:57.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:57 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:58.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:57 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:27:58.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:57 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:27:58.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:57 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:58.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:57 smithi016 ceph-mon[38248]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:58.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:57 smithi016 ceph-mon[38248]: osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] boot 2024-04-05T23:27:58.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:57 smithi016 ceph-mon[38248]: osdmap e8: 1 total, 1 up, 1 in 2024-04-05T23:27:58.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:57 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:58.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:57 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:27:58.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:57 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:27:58.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:57 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:58.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:57 smithi130 ceph-mon[79417]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T23:27:58.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:57 smithi130 ceph-mon[79417]: osd.0 [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] boot 2024-04-05T23:27:58.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:57 smithi130 ceph-mon[79417]: osdmap e8: 1 total, 1 up, 1 in 2024-04-05T23:27:58.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:57 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:27:58.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:58 smithi027 ceph-mon[78597]: from='client.? 172.21.15.27:0/1758221088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20981429-c950-4821-abbc-06a9d380c25d"}]: dispatch 2024-04-05T23:27:58.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:58 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20981429-c950-4821-abbc-06a9d380c25d"}]: dispatch 2024-04-05T23:27:58.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:58 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20981429-c950-4821-abbc-06a9d380c25d"}]': finished 2024-04-05T23:27:58.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:58 smithi027 ceph-mon[78597]: osdmap e9: 2 total, 1 up, 2 in 2024-04-05T23:27:58.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:58 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:27:59.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:58 smithi016 ceph-mon[38248]: from='client.? 172.21.15.27:0/1758221088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20981429-c950-4821-abbc-06a9d380c25d"}]: dispatch 2024-04-05T23:27:59.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:58 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20981429-c950-4821-abbc-06a9d380c25d"}]: dispatch 2024-04-05T23:27:59.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:58 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20981429-c950-4821-abbc-06a9d380c25d"}]': finished 2024-04-05T23:27:59.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:58 smithi016 ceph-mon[38248]: osdmap e9: 2 total, 1 up, 2 in 2024-04-05T23:27:59.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:27:59.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:58 smithi130 ceph-mon[79417]: from='client.? 172.21.15.27:0/1758221088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20981429-c950-4821-abbc-06a9d380c25d"}]: dispatch 2024-04-05T23:27:59.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:58 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20981429-c950-4821-abbc-06a9d380c25d"}]: dispatch 2024-04-05T23:27:59.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:58 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20981429-c950-4821-abbc-06a9d380c25d"}]': finished 2024-04-05T23:27:59.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:58 smithi130 ceph-mon[79417]: osdmap e9: 2 total, 1 up, 2 in 2024-04-05T23:27:59.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:58 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:27:59.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:59 smithi027 ceph-mon[78597]: from='client.? 172.21.15.27:0/2663400997' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:27:59.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:27:59 smithi027 ceph-mon[78597]: pgmap v40: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:00.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:59 smithi016 ceph-mon[38248]: from='client.? 172.21.15.27:0/2663400997' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:00.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:27:59 smithi016 ceph-mon[38248]: pgmap v40: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:00.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:59 smithi130 ceph-mon[79417]: from='client.? 172.21.15.27:0/2663400997' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:00.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:27:59 smithi130 ceph-mon[79417]: pgmap v40: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:02.563 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:02 smithi027 ceph-mon[78597]: pgmap v41: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:02.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:02 smithi016 ceph-mon[38248]: pgmap v41: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:02.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:02 smithi130 ceph-mon[79417]: pgmap v41: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:04.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:04 smithi016 ceph-mon[38248]: pgmap v42: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:04 smithi130 ceph-mon[79417]: pgmap v42: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:04.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:04 smithi027 ceph-mon[78597]: pgmap v42: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:05 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T23:28:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:05 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:05 smithi027 ceph-mon[78597]: Deploying daemon osd.1 on smithi027 2024-04-05T23:28:05.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T23:28:05.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:05.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:05 smithi016 ceph-mon[38248]: Deploying daemon osd.1 on smithi027 2024-04-05T23:28:05.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:05 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T23:28:05.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:05 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:05.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:05 smithi130 ceph-mon[79417]: Deploying daemon osd.1 on smithi027 2024-04-05T23:28:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:06 smithi027 ceph-mon[78597]: pgmap v43: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:06.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:06 smithi016 ceph-mon[38248]: pgmap v43: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:06.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:06 smithi130 ceph-mon[79417]: pgmap v43: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:08 smithi027 ceph-mon[78597]: pgmap v44: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:08 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:08 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:08 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:08 smithi016 ceph-mon[38248]: pgmap v44: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:08.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:08 smithi130 ceph-mon[79417]: pgmap v44: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:08.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:08 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:08.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:08 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:08.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:08 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:10.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.380+0000 7fbf1bfff700 1 -- 172.21.15.27:0/1694715985 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbf2404ebb0 con 0x7fbf100a1090 2024-04-05T23:28:10.374 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 1 on host 'smithi027' 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf100a1090 msgr2=0x7fbf100a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf100a1090 0x7fbf100a3550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009de0 tx=0x7fbf14009190 comp rx=0 tx=0).stop 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf24104380 msgr2=0x7fbf24197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf24104380 0x7fbf24197530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00d980 tx=0x7fbf0c00dc90 comp rx=0 tx=0).stop 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 shutdown_connections 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf100a1090 0x7fbf100a3550 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf24106cb0 0x7fbf2419e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf24101a50 0x7fbf24196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 --2- 172.21.15.27:0/1694715985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf24104380 0x7fbf24197530 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 >> 172.21.15.27:0/1694715985 conn(0x7fbf240fb3e0 msgr2=0x7fbf241022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:28:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 shutdown_connections 2024-04-05T23:28:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:10.383+0000 7fbf29464700 1 -- 172.21.15.27:0/1694715985 wait complete. 2024-04-05T23:28:10.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:10 smithi016 ceph-mon[38248]: pgmap v45: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:10.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:10.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:10.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:10.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:10.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:10.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:10 smithi130 ceph-mon[79417]: pgmap v45: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:10.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:10.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:10.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:10.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:10.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:10.683 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:10 smithi027 ceph-mon[78597]: pgmap v45: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:10.684 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:10.684 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:10.684 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:10.684 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:10.684 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:10.705 DEBUG:teuthology.orchestra.run.smithi027:osd.1> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-05T23:28:10.707 INFO:tasks.cephadm:Deploying osd.2 on smithi130 with /dev/vg_nvme/lv_4... 2024-04-05T23:28:10.707 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-05T23:28:10.740 INFO:journalctl@ceph.osd.1.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:28:10.933 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:28:11.337 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:11.337 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:11.337 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:11.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:11.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:11.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:11.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:11.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:11.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:12.230 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-05T23:28:12.230 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-05T23:28:12.230 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-05T23:28:12.230 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-05T23:28:12.230 INFO:teuthology.orchestra.run.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011062 s, 948 MB/s 2024-04-05T23:28:12.230 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 2024-04-05T23:28:12.231 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-05T23:28:12.353 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:12 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-1[83584]: 2024-04-05T23:28:12.089+0000 7f47f6cda7c0 -1 osd.1 0 log_to_monitors true 2024-04-05T23:28:12.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:12 smithi130 ceph-mon[79417]: pgmap v46: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:12.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:12 smithi130 ceph-mon[79417]: from='osd.1 [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T23:28:12.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:12 smithi130 ceph-mon[79417]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T23:28:12.606 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:12 smithi027 ceph-mon[78597]: pgmap v46: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:12.606 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:12 smithi027 ceph-mon[78597]: from='osd.1 [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T23:28:12.606 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:12 smithi027 ceph-mon[78597]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T23:28:12.622 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi130:vg_nvme/lv_4 2024-04-05T23:28:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:12 smithi016 ceph-mon[38248]: pgmap v46: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:12 smithi016 ceph-mon[38248]: from='osd.1 [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T23:28:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:12 smithi016 ceph-mon[38248]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T23:28:12.837 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:28:13.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: osdmap e10: 2 total, 1 up, 2 in 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='osd.1 [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:13.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:13.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: osdmap e10: 2 total, 1 up, 2 in 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='osd.1 [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:13.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.796 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.803+0000 7faa65477700 1 -- 172.21.15.130:0/3998342403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601019a0 msgr2=0x7faa60103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:13.796 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.803+0000 7faa65477700 1 --2- 172.21.15.130:0/3998342403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601019a0 0x7faa60103d90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faa54009990 tx=0x7faa54009ca0 comp rx=0 tx=0).stop 2024-04-05T23:28:13.797 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.803+0000 7faa65477700 1 -- 172.21.15.130:0/3998342403 shutdown_connections 2024-04-05T23:28:13.797 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.803+0000 7faa65477700 1 --2- 172.21.15.130:0/3998342403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa60106c00 0x7faa60109070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:13.797 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.803+0000 7faa65477700 1 --2- 172.21.15.130:0/3998342403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601019a0 0x7faa60103d90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:13.797 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.803+0000 7faa65477700 1 --2- 172.21.15.130:0/3998342403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa601042d0 0x7faa601066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:13.797 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.803+0000 7faa65477700 1 -- 172.21.15.130:0/3998342403 >> 172.21.15.130:0/3998342403 conn(0x7faa600fb3b0 msgr2=0x7faa600fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:28:13.797 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.803+0000 7faa65477700 1 -- 172.21.15.130:0/3998342403 shutdown_connections 2024-04-05T23:28:13.797 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.803+0000 7faa65477700 1 -- 172.21.15.130:0/3998342403 wait complete. 2024-04-05T23:28:13.797 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa65477700 1 Processor -- start 2024-04-05T23:28:13.797 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa65477700 1 -- start start 2024-04-05T23:28:13.799 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa65477700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa601019a0 0x7faa60196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:13.799 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa65477700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601042d0 0x7faa601973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:13.799 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa65477700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa60106c00 0x7faa6019e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:13.799 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa65477700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6019e9a0 con 0x7faa601019a0 2024-04-05T23:28:13.799 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa65477700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6019eae0 con 0x7faa60106c00 2024-04-05T23:28:13.799 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa65477700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6019ec70 con 0x7faa601042d0 2024-04-05T23:28:13.799 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa5e7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601042d0 0x7faa601973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa5e7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601042d0 0x7faa601973f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:33106/0 (socket says 172.21.15.130:33106) 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa5e7fc700 1 -- 172.21.15.130:0/2954166847 learned_addr learned my addr 172.21.15.130:0/2954166847 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa5f7fe700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa60106c00 0x7faa6019e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa5e7fc700 1 -- 172.21.15.130:0/2954166847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa60106c00 msgr2=0x7faa6019e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa5e7fc700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa60106c00 0x7faa6019e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa5e7fc700 1 -- 172.21.15.130:0/2954166847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa601019a0 msgr2=0x7faa60196eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa5e7fc700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa601019a0 0x7faa60196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.804+0000 7faa5e7fc700 1 -- 172.21.15.130:0/2954166847 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa540095f0 con 0x7faa601042d0 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.805+0000 7faa5e7fc700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601042d0 0x7faa601973f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faa5000d9b0 tx=0x7faa5000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.805+0000 7faa47fff700 1 -- 172.21.15.130:0/2954166847 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa50004220 con 0x7faa601042d0 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.805+0000 7faa47fff700 1 -- 172.21.15.130:0/2954166847 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa50007640 con 0x7faa601042d0 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.805+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6019ef50 con 0x7faa601042d0 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.805+0000 7faa47fff700 1 -- 172.21.15.130:0/2954166847 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa50005650 con 0x7faa601042d0 2024-04-05T23:28:13.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.805+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6019f410 con 0x7faa601042d0 2024-04-05T23:28:13.801 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.806+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa600fcf30 con 0x7faa601042d0 2024-04-05T23:28:13.801 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.807+0000 7faa47fff700 1 -- 172.21.15.130:0/2954166847 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 15) v1 ==== 135817+0+0 (secure 0 0 0) 0x7faa50004750 con 0x7faa601042d0 2024-04-05T23:28:13.801 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.807+0000 7faa47fff700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faa480a1110 0x7faa480a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:13.801 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.807+0000 7faa47fff700 1 -- 172.21.15.130:0/2954166847 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2029+0+0 (secure 0 0 0) 0x7faa500cad60 con 0x7faa601042d0 2024-04-05T23:28:13.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.807+0000 7faa5effd700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faa480a1110 0x7faa480a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:13.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.808+0000 7faa5effd700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faa480a1110 0x7faa480a35d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa540037f0 tx=0x7faa54005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:28:13.804 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:13.812+0000 7faa47fff700 1 -- 172.21.15.130:0/2954166847 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faa50095e60 con 0x7faa601042d0 2024-04-05T23:28:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T23:28:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: osdmap e10: 2 total, 1 up, 2 in 2024-04-05T23:28:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='osd.1 [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:28:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:28:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:13.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:13.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:13.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:14.003 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:14.009+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7faa6002cdb0 con 0x7faa480a1110 2024-04-05T23:28:14.467 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-1[83584]: 2024-04-05T23:28:14.220+0000 7f47ebb3e700 -1 osd.1 0 waiting for initial osdmap 2024-04-05T23:28:14.467 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-1[83584]: 2024-04-05T23:28:14.228+0000 7f47e412c700 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:28:14.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi027 2024-04-05T23:28:14.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi027 to 17113M 2024-04-05T23:28:14.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: pgmap v48: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:14.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:28:14.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: osdmap e11: 2 total, 1 up, 2 in 2024-04-05T23:28:14.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:14.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:14.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:28:14.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:28:14.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:14.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:14 smithi027 ceph-mon[78597]: from='osd.1 ' entity='osd.1' 2024-04-05T23:28:14.735 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi027 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi027 to 17113M 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: pgmap v48: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: osdmap e11: 2 total, 1 up, 2 in 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:14.736 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:14 smithi130 ceph-mon[79417]: from='osd.1 ' entity='osd.1' 2024-04-05T23:28:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi027 2024-04-05T23:28:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi027 to 17113M 2024-04-05T23:28:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: pgmap v48: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:28:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: osdmap e11: 2 total, 1 up, 2 in 2024-04-05T23:28:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:28:14.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:28:14.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:14.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:14 smithi016 ceph-mon[38248]: from='osd.1 ' entity='osd.1' 2024-04-05T23:28:15.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:15 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:28:15.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:15 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:28:15.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:15 smithi027 ceph-mon[78597]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:28:15.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:15 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:15.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:15 smithi027 ceph-mon[78597]: osd.1 [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] boot 2024-04-05T23:28:15.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:15 smithi027 ceph-mon[78597]: osdmap e12: 2 total, 2 up, 2 in 2024-04-05T23:28:15.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:15 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:15.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:15 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:28:15.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:15 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:28:15.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:15 smithi016 ceph-mon[38248]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:28:15.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:15.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:15 smithi016 ceph-mon[38248]: osd.1 [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] boot 2024-04-05T23:28:15.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:15 smithi016 ceph-mon[38248]: osdmap e12: 2 total, 2 up, 2 in 2024-04-05T23:28:15.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:15.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:15 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:28:15.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:15 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:28:15.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:15 smithi130 ceph-mon[79417]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:28:15.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:15.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:15 smithi130 ceph-mon[79417]: osd.1 [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] boot 2024-04-05T23:28:15.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:15 smithi130 ceph-mon[79417]: osdmap e12: 2 total, 2 up, 2 in 2024-04-05T23:28:15.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:28:16.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:16 smithi027 ceph-mon[78597]: pgmap v51: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:16.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:16 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/1180721743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794cd3d0-bb75-41a5-9ea9-e946ddea82b7"}]: dispatch 2024-04-05T23:28:16.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:16 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794cd3d0-bb75-41a5-9ea9-e946ddea82b7"}]: dispatch 2024-04-05T23:28:16.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:16 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "794cd3d0-bb75-41a5-9ea9-e946ddea82b7"}]': finished 2024-04-05T23:28:16.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:16 smithi027 ceph-mon[78597]: osdmap e13: 3 total, 2 up, 3 in 2024-04-05T23:28:16.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:16 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:16.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:16 smithi016 ceph-mon[38248]: pgmap v51: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:16.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:16 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/1180721743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794cd3d0-bb75-41a5-9ea9-e946ddea82b7"}]: dispatch 2024-04-05T23:28:16.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:16 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794cd3d0-bb75-41a5-9ea9-e946ddea82b7"}]: dispatch 2024-04-05T23:28:16.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:16 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "794cd3d0-bb75-41a5-9ea9-e946ddea82b7"}]': finished 2024-04-05T23:28:16.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:16 smithi016 ceph-mon[38248]: osdmap e13: 3 total, 2 up, 3 in 2024-04-05T23:28:16.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:16 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:16.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:16 smithi130 ceph-mon[79417]: pgmap v51: 0 pgs: ; 0 B data, 34 MiB used, 89 GiB / 89 GiB avail 2024-04-05T23:28:16.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:16 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/1180721743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794cd3d0-bb75-41a5-9ea9-e946ddea82b7"}]: dispatch 2024-04-05T23:28:16.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:16 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794cd3d0-bb75-41a5-9ea9-e946ddea82b7"}]: dispatch 2024-04-05T23:28:16.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:16 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "794cd3d0-bb75-41a5-9ea9-e946ddea82b7"}]': finished 2024-04-05T23:28:16.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:16 smithi130 ceph-mon[79417]: osdmap e13: 3 total, 2 up, 3 in 2024-04-05T23:28:16.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:16 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:17.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:17 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/2509837477' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:17.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:17 smithi027 ceph-mon[78597]: osdmap e14: 3 total, 2 up, 3 in 2024-04-05T23:28:17.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:17 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:17.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:17 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/2509837477' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:17.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:17 smithi016 ceph-mon[38248]: osdmap e14: 3 total, 2 up, 3 in 2024-04-05T23:28:17.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:17 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:17.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:17 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/2509837477' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:17.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:17 smithi130 ceph-mon[79417]: osdmap e14: 3 total, 2 up, 3 in 2024-04-05T23:28:17.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:17 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:18.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:18 smithi027 ceph-mon[78597]: pgmap v54: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:18.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:18 smithi016 ceph-mon[38248]: pgmap v54: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:18.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:18 smithi130 ceph-mon[79417]: pgmap v54: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:20.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:20 smithi027 ceph-mon[78597]: pgmap v55: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:20.745 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:20 smithi130 ceph-mon[79417]: pgmap v55: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:20.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:20 smithi016 ceph-mon[38248]: pgmap v55: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:22.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:22 smithi027 ceph-mon[78597]: pgmap v56: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:22.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:22 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T23:28:22.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:22 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:22.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:22 smithi130 ceph-mon[79417]: pgmap v56: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:22.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:22 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T23:28:22.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:22 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:22.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:22 smithi016 ceph-mon[38248]: pgmap v56: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:22.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:22 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T23:28:22.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:22 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:23.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:23 smithi027 ceph-mon[78597]: Deploying daemon osd.2 on smithi130 2024-04-05T23:28:23.737 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:23 smithi130 ceph-mon[79417]: Deploying daemon osd.2 on smithi130 2024-04-05T23:28:23.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:23 smithi016 ceph-mon[38248]: Deploying daemon osd.2 on smithi130 2024-04-05T23:28:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:24 smithi027 ceph-mon[78597]: pgmap v57: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:24.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:24 smithi016 ceph-mon[38248]: pgmap v57: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:24.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:24 smithi130 ceph-mon[79417]: pgmap v57: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:26.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:26 smithi130 ceph-mon[79417]: pgmap v58: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:26.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:26.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:26.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:26 smithi027 ceph-mon[78597]: pgmap v58: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:26.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:26 smithi016 ceph-mon[38248]: pgmap v58: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:26.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:26.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:26.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:27.912 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.919+0000 7faa47fff700 1 -- 172.21.15.130:0/2954166847 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faa6002cdb0 con 0x7faa480a1110 2024-04-05T23:28:27.912 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 2 on host 'smithi130' 2024-04-05T23:28:27.914 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faa480a1110 msgr2=0x7faa480a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:27.914 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faa480a1110 0x7faa480a35d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa540037f0 tx=0x7faa54005fb0 comp rx=0 tx=0).stop 2024-04-05T23:28:27.914 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601042d0 msgr2=0x7faa601973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:27.914 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601042d0 0x7faa601973f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faa5000d9b0 tx=0x7faa5000dcc0 comp rx=0 tx=0).stop 2024-04-05T23:28:27.915 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 shutdown_connections 2024-04-05T23:28:27.915 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7faa480a1110 0x7faa480a35d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:27.915 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa601019a0 0x7faa60196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:27.915 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa60106c00 0x7faa6019e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:27.915 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 --2- 172.21.15.130:0/2954166847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa601042d0 0x7faa601973f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:27.915 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 >> 172.21.15.130:0/2954166847 conn(0x7faa600fb3b0 msgr2=0x7faa601023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:28:27.915 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.921+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 shutdown_connections 2024-04-05T23:28:27.915 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:28:27.922+0000 7faa65477700 1 -- 172.21.15.130:0/2954166847 wait complete. 2024-04-05T23:28:28.066 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.066 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.066 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:28.066 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:28.067 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.067 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:27 smithi130 ceph-mon[79417]: pgmap v59: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:28.067 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.067 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:28.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:28.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:27 smithi027 ceph-mon[78597]: pgmap v59: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:28.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.276 DEBUG:teuthology.orchestra.run.smithi130:osd.2> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-05T23:28:28.278 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_3... 2024-04-05T23:28:28.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-05T23:28:28.310 INFO:journalctl@ceph.osd.2.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:28:28.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:28.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:28.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:27 smithi016 ceph-mon[38248]: pgmap v59: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:28.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:28:29.059 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:29.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:29.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:30.041 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-2[85197]: 2024-04-05T23:28:29.693+0000 7fc4e77317c0 -1 osd.2 0 log_to_monitors true 2024-04-05T23:28:30.041 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:29 smithi130 ceph-mon[79417]: pgmap v60: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:30.041 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:29 smithi130 ceph-mon[79417]: from='osd.2 [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T23:28:30.041 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:29 smithi130 ceph-mon[79417]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T23:28:30.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:29 smithi027 ceph-mon[78597]: pgmap v60: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:30.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:29 smithi027 ceph-mon[78597]: from='osd.2 [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T23:28:30.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:29 smithi027 ceph-mon[78597]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T23:28:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-05T23:28:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-05T23:28:30.320 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-05T23:28:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-05T23:28:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0385789 s, 272 MB/s 2024-04-05T23:28:30.321 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 2024-04-05T23:28:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-05T23:28:30.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:29 smithi016 ceph-mon[38248]: pgmap v60: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:30.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:29 smithi016 ceph-mon[38248]: from='osd.2 [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T23:28:30.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:29 smithi016 ceph-mon[38248]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T23:28:30.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: osdmap e15: 3 total, 2 up, 3 in 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='osd.2 [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:31.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:31.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.093 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:28:31.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: osdmap e15: 3 total, 2 up, 3 in 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='osd.2 [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:31.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: osdmap e15: 3 total, 2 up, 3 in 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='osd.2 [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:31.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:32.084 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-2[85197]: 2024-04-05T23:28:31.727+0000 7fc4dc595700 -1 osd.2 0 waiting for initial osdmap 2024-04-05T23:28:32.084 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-2[85197]: 2024-04-05T23:28:31.734+0000 7fc4d4b83700 -1 osd.2 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:28:32.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi130 2024-04-05T23:28:32.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi130 to 17113M 2024-04-05T23:28:32.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-mon[79417]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:28:32.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-mon[79417]: osdmap e16: 3 total, 2 up, 3 in 2024-04-05T23:28:32.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:32.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:32.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-mon[79417]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:32.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-mon[79417]: from='osd.2 ' entity='osd.2' 2024-04-05T23:28:32.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:32.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:31 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi130 2024-04-05T23:28:32.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:31 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi130 to 17113M 2024-04-05T23:28:32.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:31 smithi027 ceph-mon[78597]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:28:32.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:31 smithi027 ceph-mon[78597]: osdmap e16: 3 total, 2 up, 3 in 2024-04-05T23:28:32.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:32.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:32.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:31 smithi027 ceph-mon[78597]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:32.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:31 smithi027 ceph-mon[78597]: from='osd.2 ' entity='osd.2' 2024-04-05T23:28:32.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.227+0000 7f8ddae00700 1 -- 172.21.15.16:0/3764805760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 msgr2=0x7f8dd4101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.227+0000 7f8ddae00700 1 --2- 172.21.15.16:0/3764805760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 0x7f8dd4101330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009990 tx=0x7f8dc4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.228+0000 7f8ddae00700 1 -- 172.21.15.16:0/3764805760 shutdown_connections 2024-04-05T23:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.228+0000 7f8ddae00700 1 --2- 172.21.15.16:0/3764805760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 0x7f8dd4101330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.228+0000 7f8ddae00700 1 --2- 172.21.15.16:0/3764805760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8dd4102a90 0x7f8dd41093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.228+0000 7f8ddae00700 1 --2- 172.21.15.16:0/3764805760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd41020d0 0x7f8dd4102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.228+0000 7f8ddae00700 1 -- 172.21.15.16:0/3764805760 >> 172.21.15.16:0/3764805760 conn(0x7f8dd40fc4e0 msgr2=0x7f8dd40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.228+0000 7f8ddae00700 1 -- 172.21.15.16:0/3764805760 shutdown_connections 2024-04-05T23:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.228+0000 7f8ddae00700 1 -- 172.21.15.16:0/3764805760 wait complete. 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.229+0000 7f8ddae00700 1 Processor -- start 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.229+0000 7f8ddae00700 1 -- start start 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8ddae00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 0x7f8dd4196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8ddae00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd41020d0 0x7f8dd4197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8ddae00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8dd4102a90 0x7f8dd419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8ddae00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd419e9d0 con 0x7f8dd4100ef0 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8ddae00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd419eb10 con 0x7f8dd41020d0 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8ddae00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd419eca0 con 0x7f8dd4102a90 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 0x7f8dd4196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 0x7f8dd4196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37794/0 (socket says 172.21.15.16:37794) 2024-04-05T23:28:32.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd3e00700 1 -- 172.21.15.16:0/2045343109 learned_addr learned my addr 172.21.15.16:0/2045343109 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:28:32.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd8e00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8dd4102a90 0x7f8dd419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:32.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd3400700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd41020d0 0x7f8dd4197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:32.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd3e00700 1 -- 172.21.15.16:0/2045343109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8dd4102a90 msgr2=0x7f8dd419e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:32.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd3e00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8dd4102a90 0x7f8dd419e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:32.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd3e00700 1 -- 172.21.15.16:0/2045343109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd41020d0 msgr2=0x7f8dd4197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:32.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd3e00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd41020d0 0x7f8dd4197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:32.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.230+0000 7f8dd3e00700 1 -- 172.21.15.16:0/2045343109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc40095f0 con 0x7f8dd4100ef0 2024-04-05T23:28:32.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.231+0000 7f8dd3e00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 0x7f8dd4196ee0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40095c0 tx=0x7f8dc4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:28:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.231+0000 7f8dd0c00700 1 -- 172.21.15.16:0/2045343109 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc401d070 con 0x7f8dd4100ef0 2024-04-05T23:28:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.231+0000 7f8dd0c00700 1 -- 172.21.15.16:0/2045343109 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8dc4021410 con 0x7f8dd4100ef0 2024-04-05T23:28:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.231+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd419ef20 con 0x7f8dd4100ef0 2024-04-05T23:28:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.231+0000 7f8dd0c00700 1 -- 172.21.15.16:0/2045343109 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4017650 con 0x7f8dd4100ef0 2024-04-05T23:28:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.231+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd419f280 con 0x7f8dd4100ef0 2024-04-05T23:28:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.232+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd4108480 con 0x7f8dd4100ef0 2024-04-05T23:28:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.234+0000 7f8dd0c00700 1 -- 172.21.15.16:0/2045343109 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 135817+0+0 (secure 0 0 0) 0x7f8dc4021580 con 0x7f8dd4100ef0 2024-04-05T23:28:32.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.236+0000 7f8dd0c00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8db80a1090 0x7f8db80a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:32.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.237+0000 7f8dd0c00700 1 -- 172.21.15.16:0/2045343109 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2571+0+0 (secure 0 0 0) 0x7f8dc40cc6a0 con 0x7f8dd4100ef0 2024-04-05T23:28:32.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.237+0000 7f8dd3400700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8db80a1090 0x7f8db80a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:32.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.237+0000 7f8dd3400700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8db80a1090 0x7f8db80a3550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0005950 tx=0x7f8dc00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:28:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.241+0000 7f8dd0c00700 1 -- 172.21.15.16:0/2045343109 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8dc4097570 con 0x7f8dd4100ef0 2024-04-05T23:28:32.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:31 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi130 2024-04-05T23:28:32.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:31 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi130 to 17113M 2024-04-05T23:28:32.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:31 smithi016 ceph-mon[38248]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:28:32.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:31 smithi016 ceph-mon[38248]: osdmap e16: 3 total, 2 up, 3 in 2024-04-05T23:28:32.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:32.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:32.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:31 smithi016 ceph-mon[38248]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-05T23:28:32.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:31 smithi016 ceph-mon[38248]: from='osd.2 ' entity='osd.2' 2024-04-05T23:28:32.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:32.467+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8dd402ce50 con 0x7f8db80a1090 2024-04-05T23:28:33.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:32 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:28:33.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:32 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:28:33.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:32 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:28:33.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:32 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:28:33.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:32 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:33.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:32 smithi027 ceph-mon[78597]: osd.2 [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] boot 2024-04-05T23:28:33.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:32 smithi027 ceph-mon[78597]: osdmap e17: 3 total, 3 up, 3 in 2024-04-05T23:28:33.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:32 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:33.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:32 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:28:33.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:32 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:28:33.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:32 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:28:33.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:32 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:28:33.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:32 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:33.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:32 smithi016 ceph-mon[38248]: osd.2 [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] boot 2024-04-05T23:28:33.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:32 smithi016 ceph-mon[38248]: osdmap e17: 3 total, 3 up, 3 in 2024-04-05T23:28:33.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:32 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:33.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:32 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:28:33.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:32 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:28:33.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:32 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:28:33.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:32 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:28:33.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:32 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:33.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:32 smithi130 ceph-mon[79417]: osd.2 [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] boot 2024-04-05T23:28:33.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:32 smithi130 ceph-mon[79417]: osdmap e17: 3 total, 3 up, 3 in 2024-04-05T23:28:33.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:32 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:28:34.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:33 smithi016 ceph-mon[38248]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:28:34.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:33 smithi016 ceph-mon[38248]: pgmap v65: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:34.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-05T23:28:34.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:33 smithi027 ceph-mon[78597]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:28:34.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:33 smithi027 ceph-mon[78597]: pgmap v65: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:34.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-05T23:28:34.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:33 smithi130 ceph-mon[79417]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:28:34.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:33 smithi130 ceph-mon[79417]: pgmap v65: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:34.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-05T23:28:35.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:34 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-05T23:28:35.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:34 smithi027 ceph-mon[78597]: osdmap e18: 3 total, 3 up, 3 in 2024-04-05T23:28:35.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:34 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T23:28:35.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:34 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4062585464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ef270b8-eac6-4738-82f3-7aa26c536370"}]: dispatch 2024-04-05T23:28:35.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:34 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-05T23:28:35.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:34 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4062585464' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ef270b8-eac6-4738-82f3-7aa26c536370"}]': finished 2024-04-05T23:28:35.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:34 smithi027 ceph-mon[78597]: osdmap e19: 4 total, 3 up, 4 in 2024-04-05T23:28:35.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:34 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:34 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-05T23:28:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:34 smithi016 ceph-mon[38248]: osdmap e18: 3 total, 3 up, 3 in 2024-04-05T23:28:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:34 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T23:28:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:34 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4062585464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ef270b8-eac6-4738-82f3-7aa26c536370"}]: dispatch 2024-04-05T23:28:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:34 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-05T23:28:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:34 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4062585464' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ef270b8-eac6-4738-82f3-7aa26c536370"}]': finished 2024-04-05T23:28:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:34 smithi016 ceph-mon[38248]: osdmap e19: 4 total, 3 up, 4 in 2024-04-05T23:28:35.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:34 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:35.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:34 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-05T23:28:35.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:34 smithi130 ceph-mon[79417]: osdmap e18: 3 total, 3 up, 3 in 2024-04-05T23:28:35.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:34 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T23:28:35.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:34 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4062585464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ef270b8-eac6-4738-82f3-7aa26c536370"}]: dispatch 2024-04-05T23:28:35.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:34 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-05T23:28:35.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:34 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4062585464' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ef270b8-eac6-4738-82f3-7aa26c536370"}]': finished 2024-04-05T23:28:35.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:34 smithi130 ceph-mon[79417]: osdmap e19: 4 total, 3 up, 4 in 2024-04-05T23:28:35.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:34 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:35.833 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:28:35 smithi016 sudo[62870]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T23:28:35.833 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:28:35 smithi016 sudo[62870]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:35.833 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:28:35 smithi016 sudo[62870]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:35.833 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:28:35 smithi016 sudo[62870]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:35.966+0000 7f8dd0c00700 1 -- 172.21.15.16:0/2045343109 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f8dc4097940 con 0x7f8dd4100ef0 2024-04-05T23:28:36.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:35 smithi027 ceph-mon[78597]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:36.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:35 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3445906194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:36.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:35 smithi027 ceph-mon[78597]: osdmap e20: 4 total, 3 up, 4 in 2024-04-05T23:28:36.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:36.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:35 smithi016 ceph-mon[38248]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:36.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:35 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3445906194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:36.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:35 smithi016 ceph-mon[38248]: osdmap e20: 4 total, 3 up, 4 in 2024-04-05T23:28:36.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:36.333 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:28:36 smithi016 sudo[62924]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T23:28:36.333 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:28:36 smithi016 sudo[62924]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:36.333 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:28:36 smithi016 sudo[62924]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:36.334 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:28:36 smithi016 sudo[62924]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:35 smithi130 ceph-mon[79417]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:35 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3445906194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:35 smithi130 ceph-mon[79417]: osdmap e20: 4 total, 3 up, 4 in 2024-04-05T23:28:36.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:36.953 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:36 smithi027 sudo[89053]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T23:28:36.954 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:36 smithi027 sudo[89053]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:36.954 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:36 smithi027 sudo[89053]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:36.954 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:36 smithi027 sudo[89053]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:37.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:36 smithi027 ceph-mon[78597]: mgrmap e16: x(active, since 118s), standbys: y, z 2024-04-05T23:28:37.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:36 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:37.217 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:37 smithi027 sudo[89056]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T23:28:37.217 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:37 smithi027 sudo[89056]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:37.217 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:37 smithi027 sudo[89056]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:37.217 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:28:37 smithi027 sudo[89056]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:37.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:36 smithi016 ceph-mon[38248]: mgrmap e16: x(active, since 118s), standbys: y, z 2024-04-05T23:28:37.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:36 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:37.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:36 smithi130 ceph-mon[79417]: mgrmap e16: x(active, since 118s), standbys: y, z 2024-04-05T23:28:37.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:36 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:37.834 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:37 smithi130 sudo[90655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T23:28:37.835 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:37 smithi130 sudo[90655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:37.835 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:37 smithi130 sudo[90655]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:37.835 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:37 smithi130 sudo[90655]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:37.835 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:37 smithi130 sudo[90658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T23:28:37.835 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:37 smithi130 sudo[90658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:37.835 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:37 smithi130 sudo[90658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:37.835 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:28:37 smithi130 sudo[90658]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:38.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 sudo[63873]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T23:28:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 sudo[63873]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 sudo[63873]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:38.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 sudo[63873]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:39.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 sudo[64073]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T23:28:39.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 sudo[64073]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:39.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 sudo[64073]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:39.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 sudo[64073]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:39.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:39.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 ceph-mon[38248]: pgmap v70: 1 pgs: 1 active+clean; 449 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:39.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:39.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:38 smithi016 ceph-mon[38248]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T23:28:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:38 smithi130 ceph-mon[79417]: pgmap v70: 1 pgs: 1 active+clean; 449 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:38 smithi130 ceph-mon[79417]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T23:28:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:38 smithi130 sudo[90661]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T23:28:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:38 smithi130 sudo[90661]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:38 smithi130 sudo[90661]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:38 smithi130 sudo[90661]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:39.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:39.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:38 smithi027 ceph-mon[78597]: pgmap v70: 1 pgs: 1 active+clean; 449 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:39.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:39.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:38 smithi027 ceph-mon[78597]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T23:28:39.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:39 smithi130 sudo[90664]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T23:28:39.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:39 smithi130 sudo[90664]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:39.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:39 smithi130 sudo[90664]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:39.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:39 smithi130 sudo[90664]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:39.969 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:39 smithi027 sudo[89059]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T23:28:39.970 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:39 smithi027 sudo[89059]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:39.970 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:39 smithi027 sudo[89059]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:39.970 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:39 smithi027 sudo[89059]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:28:40.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:28:40.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:28:40.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:40 smithi016 ceph-mon[38248]: pgmap v71: 1 pgs: 1 active+clean; 449 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 sudo[89062]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T23:28:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 sudo[89062]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T23:28:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 sudo[89062]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T23:28:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 sudo[89062]: pam_unix(sudo:session): session closed for user root 2024-04-05T23:28:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T23:28:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:28:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:28:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:28:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T23:28:40.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:28:40.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:28:40.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:28:40.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T23:28:40.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:28:40.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:28:40.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:28:40.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:40 smithi027 ceph-mon[78597]: pgmap v71: 1 pgs: 1 active+clean; 449 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:40.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T23:28:40.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:28:40.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:28:40.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:40 smithi130 ceph-mon[79417]: pgmap v71: 1 pgs: 1 active+clean; 449 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:41.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:41 smithi027 ceph-mon[78597]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T23:28:41.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:41 smithi027 ceph-mon[78597]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T23:28:41.518 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:41 smithi016 ceph-mon[38248]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T23:28:41.518 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:41 smithi016 ceph-mon[38248]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T23:28:41.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:41 smithi130 ceph-mon[79417]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T23:28:41.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:41 smithi130 ceph-mon[79417]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T23:28:42.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:42 smithi027 ceph-mon[78597]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 344 KiB/s wr, 23 op/s 2024-04-05T23:28:42.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:42 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T23:28:42.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:42 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:42.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:42 smithi016 ceph-mon[38248]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 344 KiB/s wr, 23 op/s 2024-04-05T23:28:42.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:42 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T23:28:42.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:42 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:42.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:42 smithi130 ceph-mon[79417]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 344 KiB/s wr, 23 op/s 2024-04-05T23:28:42.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:42 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T23:28:42.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:42 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:43.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:43 smithi016 ceph-mon[38248]: Deploying daemon osd.3 on smithi016 2024-04-05T23:28:43.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:43 smithi027 ceph-mon[78597]: Deploying daemon osd.3 on smithi016 2024-04-05T23:28:43.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:43 smithi130 ceph-mon[79417]: Deploying daemon osd.3 on smithi016 2024-04-05T23:28:44.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:44 smithi016 ceph-mon[38248]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 250 KiB/s wr, 17 op/s 2024-04-05T23:28:44.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:44 smithi027 ceph-mon[78597]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 250 KiB/s wr, 17 op/s 2024-04-05T23:28:44.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:44 smithi130 ceph-mon[79417]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 250 KiB/s wr, 17 op/s 2024-04-05T23:28:46.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:46 smithi016 ceph-mon[38248]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-05T23:28:46.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:46 smithi130 ceph-mon[79417]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-05T23:28:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:46 smithi027 ceph-mon[78597]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-05T23:28:47.931 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:47.931 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:47.931 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:47.931 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:47 smithi016 ceph-mon[38248]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 161 KiB/s wr, 11 op/s 2024-04-05T23:28:48.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:48.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:48.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:48.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:47 smithi027 ceph-mon[78597]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 161 KiB/s wr, 11 op/s 2024-04-05T23:28:48.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:48.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:48.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:48.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:47 smithi130 ceph-mon[79417]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 161 KiB/s wr, 11 op/s 2024-04-05T23:28:49.913 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-04-05T23:28:49.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.919+0000 7f8dd0c00700 1 -- 172.21.15.16:0/2045343109 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8dd402ce50 con 0x7f8db80a1090 2024-04-05T23:28:49.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8db80a1090 msgr2=0x7f8db80a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:49.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8db80a1090 0x7f8db80a3550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0005950 tx=0x7f8dc00058e0 comp rx=0 tx=0).stop 2024-04-05T23:28:49.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 msgr2=0x7f8dd4196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:49.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 0x7f8dd4196ee0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40095c0 tx=0x7f8dc4005740 comp rx=0 tx=0).stop 2024-04-05T23:28:49.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 shutdown_connections 2024-04-05T23:28:49.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8db80a1090 0x7f8db80a3550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:49.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dd4100ef0 0x7f8dd4196ee0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:49.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8dd4102a90 0x7f8dd419e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:49.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 --2- 172.21.15.16:0/2045343109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd41020d0 0x7f8dd4197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:49.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 >> 172.21.15.16:0/2045343109 conn(0x7f8dd40fc4e0 msgr2=0x7f8dd41055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:28:49.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 shutdown_connections 2024-04-05T23:28:49.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:28:49.922+0000 7f8ddae00700 1 -- 172.21.15.16:0/2045343109 wait complete. 2024-04-05T23:28:50.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:50.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:50.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:50.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:50.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:50.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:49 smithi016 ceph-mon[38248]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-04-05T23:28:50.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:50.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:50.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:50.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:50.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:50.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:49 smithi130 ceph-mon[79417]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-04-05T23:28:50.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:50.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:50.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:50.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:50.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:50.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:49 smithi027 ceph-mon[78597]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-04-05T23:28:50.363 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-05T23:28:50.369 INFO:tasks.cephadm:Deploying osd.4 on smithi027 with /dev/vg_nvme/lv_3... 2024-04-05T23:28:50.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-05T23:28:50.442 INFO:journalctl@ceph.osd.3.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:28:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:28:51.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:51.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:51.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:51.083 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 05 23:28:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-3[65112]: 2024-04-05T23:28:50.919+0000 7f5b7e7137c0 -1 osd.3 0 log_to_monitors true 2024-04-05T23:28:51.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:51.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:51.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:51.103 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:28:51.103 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:51.103 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-05T23:28:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-05T23:28:51.962 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-05T23:28:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-05T23:28:51.962 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109454 s, 958 MB/s 2024-04-05T23:28:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-05T23:28:52.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:51 smithi016 ceph-mon[38248]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T23:28:52.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:51 smithi016 ceph-mon[38248]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-04-05T23:28:52.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:51 smithi130 ceph-mon[79417]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T23:28:52.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:51 smithi130 ceph-mon[79417]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-04-05T23:28:52.137 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:51 smithi027 ceph-mon[78597]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T23:28:52.137 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:51 smithi027 ceph-mon[78597]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-04-05T23:28:52.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi027:vg_nvme/lv_3 2024-04-05T23:28:52.556 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:28:52.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:52 smithi027 ceph-mon[78597]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T23:28:52.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:52 smithi027 ceph-mon[78597]: osdmap e21: 4 total, 3 up, 4 in 2024-04-05T23:28:52.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:52.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:52 smithi027 ceph-mon[78597]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:28:53.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:52 smithi016 ceph-mon[38248]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T23:28:53.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:52 smithi016 ceph-mon[38248]: osdmap e21: 4 total, 3 up, 4 in 2024-04-05T23:28:53.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:53.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:52 smithi016 ceph-mon[38248]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:28:53.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:52 smithi130 ceph-mon[79417]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T23:28:53.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:52 smithi130 ceph-mon[79417]: osdmap e21: 4 total, 3 up, 4 in 2024-04-05T23:28:53.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:53.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:52 smithi130 ceph-mon[79417]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.609+0000 7f95d082e700 1 -- 172.21.15.27:0/1545047123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 msgr2=0x7f95cc100260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.609+0000 7f95d082e700 1 --2- 172.21.15.27:0/1545047123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 0x7f95cc100260 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009990 tx=0x7f95bc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.609+0000 7f95d082e700 1 -- 172.21.15.27:0/1545047123 shutdown_connections 2024-04-05T23:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.609+0000 7f95d082e700 1 --2- 172.21.15.27:0/1545047123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc101960 0x7f95cc108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.609+0000 7f95d082e700 1 --2- 172.21.15.27:0/1545047123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 0x7f95cc100260 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.609+0000 7f95d082e700 1 --2- 172.21.15.27:0/1545047123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95cc100fa0 0x7f95cc101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.609+0000 7f95d082e700 1 -- 172.21.15.27:0/1545047123 >> 172.21.15.27:0/1545047123 conn(0x7f95cc0fb3e0 msgr2=0x7f95cc0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.609+0000 7f95d082e700 1 -- 172.21.15.27:0/1545047123 shutdown_connections 2024-04-05T23:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.609+0000 7f95d082e700 1 -- 172.21.15.27:0/1545047123 wait complete. 2024-04-05T23:28:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95d082e700 1 Processor -- start 2024-04-05T23:28:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95d082e700 1 -- start start 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95d082e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 0x7f95cc10a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95d082e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95cc100fa0 0x7f95cc108d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95d082e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc101960 0x7f95cc1092b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95d082e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc109a50 con 0x7f95cc100fa0 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95d082e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc109bc0 con 0x7f95cc0ffe40 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95d082e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc1a0ec0 con 0x7f95cc101960 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95ca59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 0x7f95cc10a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95ca59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 0x7f95cc10a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56620/0 (socket says 172.21.15.27:56620) 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95ca59c700 1 -- 172.21.15.27:0/1293750713 learned_addr learned my addr 172.21.15.27:0/1293750713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95cad9d700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc101960 0x7f95cc1092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95c9d9b700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95cc100fa0 0x7f95cc108d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95ca59c700 1 -- 172.21.15.27:0/1293750713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc101960 msgr2=0x7f95cc1092b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95ca59c700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc101960 0x7f95cc1092b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95ca59c700 1 -- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95cc100fa0 msgr2=0x7f95cc108d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95ca59c700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95cc100fa0 0x7f95cc108d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95ca59c700 1 -- 172.21.15.27:0/1293750713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc0095f0 con 0x7f95cc0ffe40 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95cad9d700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc101960 0x7f95cc1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.610+0000 7f95ca59c700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 0x7f95cc10a6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95bc005910 tx=0x7f95bc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.611+0000 7f95bb7fe700 1 -- 172.21.15.27:0/1293750713 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc01d070 con 0x7f95cc0ffe40 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.611+0000 7f95bb7fe700 1 -- 172.21.15.27:0/1293750713 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95bc003f60 con 0x7f95cc0ffe40 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.611+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95cc1a10f0 con 0x7f95cc0ffe40 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.611+0000 7f95bb7fe700 1 -- 172.21.15.27:0/1293750713 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc022cc0 con 0x7f95cc0ffe40 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.611+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95cc1a1560 con 0x7f95cc0ffe40 2024-04-05T23:28:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.612+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95cc107210 con 0x7f95cc0ffe40 2024-04-05T23:28:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.613+0000 7f95bb7fe700 1 -- 172.21.15.27:0/1293750713 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f95bc0040d0 con 0x7f95cc0ffe40 2024-04-05T23:28:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.613+0000 7f95bb7fe700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f95b40a11a0 0x7f95b40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:28:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.614+0000 7f95bb7fe700 1 -- 172.21.15.27:0/1293750713 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3389+0+0 (secure 0 0 0) 0x7f95bc0ccee0 con 0x7f95cc0ffe40 2024-04-05T23:28:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.614+0000 7f95c9d9b700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f95b40a11a0 0x7f95b40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:28:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.614+0000 7f95c9d9b700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f95b40a11a0 0x7f95b40a3660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f95c0007c60 tx=0x7f95c00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:28:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.617+0000 7f95bb7fe700 1 -- 172.21.15.27:0/1293750713 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f95bc097a20 con 0x7f95cc0ffe40 2024-04-05T23:28:53.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:28:53.829+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f95cc061320 con 0x7f95b40a11a0 2024-04-05T23:28:54.083 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-3[65112]: 2024-04-05T23:28:53.670+0000 7f5b74000700 -1 osd.3 0 waiting for initial osdmap 2024-04-05T23:28:54.083 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-3[65112]: 2024-04-05T23:28:53.693+0000 7f5b68a00700 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:28:54.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:28:54.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: osdmap e22: 4 total, 3 up, 4 in 2024-04-05T23:28:54.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi016 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi016 to 8637M 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 97 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:54.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' 2024-04-05T23:28:54.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:53 smithi016 ceph-mon[38248]: osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] boot 2024-04-05T23:28:54.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:28:54.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: osdmap e22: 4 total, 3 up, 4 in 2024-04-05T23:28:54.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:54.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:54.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi016 2024-04-05T23:28:54.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi016 to 8637M 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 97 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' 2024-04-05T23:28:54.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:53 smithi130 ceph-mon[79417]: osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] boot 2024-04-05T23:28:54.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:28:54.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: osdmap e22: 4 total, 3 up, 4 in 2024-04-05T23:28:54.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:54.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi016 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi016 to 8637M 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 97 MiB used, 268 GiB / 268 GiB avail 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: from='osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397]' entity='osd.3' 2024-04-05T23:28:54.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:53 smithi027 ceph-mon[78597]: osd.3 [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] boot 2024-04-05T23:28:54.944 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:54 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:28:54.944 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:54 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:28:54.944 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:54 smithi027 ceph-mon[78597]: osdmap e23: 4 total, 4 up, 4 in 2024-04-05T23:28:54.944 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:54.944 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:54 smithi027 ceph-mon[78597]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:28:54.944 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:28:54.944 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:28:54.944 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:54.945 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:54 smithi027 ceph-mon[78597]: osdmap e24: 4 total, 4 up, 4 in 2024-04-05T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:54 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:54 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:54 smithi016 ceph-mon[38248]: osdmap e23: 4 total, 4 up, 4 in 2024-04-05T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:54 smithi016 ceph-mon[38248]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:28:55.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:28:55.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:28:55.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:55.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:54 smithi016 ceph-mon[38248]: osdmap e24: 4 total, 4 up, 4 in 2024-04-05T23:28:55.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:54 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:28:55.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:54 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:28:55.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:54 smithi130 ceph-mon[79417]: osdmap e23: 4 total, 4 up, 4 in 2024-04-05T23:28:55.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:28:55.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:54 smithi130 ceph-mon[79417]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:28:55.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:28:55.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:28:55.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:28:55.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:54 smithi130 ceph-mon[79417]: osdmap e24: 4 total, 4 up, 4 in 2024-04-05T23:28:56.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:56 smithi027 ceph-mon[78597]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:28:56.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:56 smithi027 ceph-mon[78597]: osdmap e25: 4 total, 4 up, 4 in 2024-04-05T23:28:56.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:56 smithi027 ceph-mon[78597]: from='client.? 172.21.15.27:0/3323054245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e348d76d-044d-4a3a-8420-8e283130eb95"}]: dispatch 2024-04-05T23:28:56.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:56 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e348d76d-044d-4a3a-8420-8e283130eb95"}]: dispatch 2024-04-05T23:28:56.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:56 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e348d76d-044d-4a3a-8420-8e283130eb95"}]': finished 2024-04-05T23:28:56.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:56 smithi027 ceph-mon[78597]: osdmap e26: 5 total, 4 up, 5 in 2024-04-05T23:28:56.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:28:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:56 smithi016 ceph-mon[38248]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:28:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:56 smithi016 ceph-mon[38248]: osdmap e25: 4 total, 4 up, 4 in 2024-04-05T23:28:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:56 smithi016 ceph-mon[38248]: from='client.? 172.21.15.27:0/3323054245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e348d76d-044d-4a3a-8420-8e283130eb95"}]: dispatch 2024-04-05T23:28:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:56 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e348d76d-044d-4a3a-8420-8e283130eb95"}]: dispatch 2024-04-05T23:28:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:56 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e348d76d-044d-4a3a-8420-8e283130eb95"}]': finished 2024-04-05T23:28:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:56 smithi016 ceph-mon[38248]: osdmap e26: 5 total, 4 up, 5 in 2024-04-05T23:28:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:28:56.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:56 smithi130 ceph-mon[79417]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:28:56.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:56 smithi130 ceph-mon[79417]: osdmap e25: 4 total, 4 up, 4 in 2024-04-05T23:28:56.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:56 smithi130 ceph-mon[79417]: from='client.? 172.21.15.27:0/3323054245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e348d76d-044d-4a3a-8420-8e283130eb95"}]: dispatch 2024-04-05T23:28:56.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:56 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e348d76d-044d-4a3a-8420-8e283130eb95"}]: dispatch 2024-04-05T23:28:56.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:56 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e348d76d-044d-4a3a-8420-8e283130eb95"}]': finished 2024-04-05T23:28:56.836 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:56 smithi130 ceph-mon[79417]: osdmap e26: 5 total, 4 up, 5 in 2024-04-05T23:28:56.836 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:28:57.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:57 smithi027 ceph-mon[78597]: from='client.? 172.21.15.27:0/1382282545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:57.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:57 smithi027 ceph-mon[78597]: osdmap e27: 5 total, 4 up, 5 in 2024-04-05T23:28:57.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:57 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:28:57.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:57 smithi016 ceph-mon[38248]: from='client.? 172.21.15.27:0/1382282545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:57.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:57 smithi016 ceph-mon[38248]: osdmap e27: 5 total, 4 up, 5 in 2024-04-05T23:28:57.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:57 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:28:57.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:57 smithi130 ceph-mon[79417]: from='client.? 172.21.15.27:0/1382282545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:28:57.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:57 smithi130 ceph-mon[79417]: osdmap e27: 5 total, 4 up, 5 in 2024-04-05T23:28:57.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:57 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:28:58.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:28:58 smithi027 ceph-mon[78597]: pgmap v87: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 312 MiB used, 357 GiB / 358 GiB avail; 165 KiB/s, 0 objects/s recovering 2024-04-05T23:28:58.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:28:58 smithi016 ceph-mon[38248]: pgmap v87: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 312 MiB used, 357 GiB / 358 GiB avail; 165 KiB/s, 0 objects/s recovering 2024-04-05T23:28:58.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:28:58 smithi130 ceph-mon[79417]: pgmap v87: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 312 MiB used, 357 GiB / 358 GiB avail; 165 KiB/s, 0 objects/s recovering 2024-04-05T23:29:00.693 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:00 smithi027 ceph-mon[78597]: pgmap v88: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-05T23:29:00.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:00 smithi016 ceph-mon[38248]: pgmap v88: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-05T23:29:00.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:00 smithi130 ceph-mon[79417]: pgmap v88: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-05T23:29:02.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:02 smithi027 ceph-mon[78597]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T23:29:02.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T23:29:02.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:02.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:02 smithi016 ceph-mon[38248]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T23:29:02.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T23:29:02.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:02.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:02 smithi130 ceph-mon[79417]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T23:29:02.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T23:29:02.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:03.571 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:03 smithi027 ceph-mon[78597]: Deploying daemon osd.4 on smithi027 2024-04-05T23:29:03.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:03 smithi016 ceph-mon[38248]: Deploying daemon osd.4 on smithi027 2024-04-05T23:29:03.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:03 smithi130 ceph-mon[79417]: Deploying daemon osd.4 on smithi027 2024-04-05T23:29:04.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:04 smithi027 ceph-mon[78597]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-05T23:29:04.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:04 smithi016 ceph-mon[38248]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-05T23:29:04.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:04 smithi130 ceph-mon[79417]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-05T23:29:06.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:06 smithi027 ceph-mon[78597]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:06.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:06.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:06.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:06.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:06 smithi016 ceph-mon[38248]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:06.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:06.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:06.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:06.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:06 smithi130 ceph-mon[79417]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:06.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:06.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:06.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:07.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.885+0000 7f95bb7fe700 1 -- 172.21.15.27:0/1293750713 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f95cc061320 con 0x7f95b40a11a0 2024-04-05T23:29:07.878 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 4 on host 'smithi027' 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f95b40a11a0 msgr2=0x7f95b40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f95b40a11a0 0x7f95b40a3660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f95c0007c60 tx=0x7f95c00073d0 comp rx=0 tx=0).stop 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 msgr2=0x7f95cc10a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 0x7f95cc10a6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95bc005910 tx=0x7f95bc004730 comp rx=0 tx=0).stop 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 shutdown_connections 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f95b40a11a0 0x7f95b40a3660 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc101960 0x7f95cc1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95cc0ffe40 0x7f95cc10a6d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 --2- 172.21.15.27:0/1293750713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95cc100fa0 0x7f95cc108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 >> 172.21.15.27:0/1293750713 conn(0x7f95cc0fb3e0 msgr2=0x7f95cc104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 shutdown_connections 2024-04-05T23:29:07.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:07.887+0000 7f95d082e700 1 -- 172.21.15.27:0/1293750713 wait complete. 2024-04-05T23:29:08.067 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.067 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.067 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:08.068 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:08.068 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.068 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:07 smithi027 ceph-mon[78597]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:08.068 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.068 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.285 DEBUG:teuthology.orchestra.run.smithi027:osd.4> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-05T23:29:08.287 INFO:tasks.cephadm:Deploying osd.5 on smithi130 with /dev/vg_nvme/lv_3... 2024-04-05T23:29:08.287 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-05T23:29:08.320 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.320 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:08.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:08.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:07 smithi130 ceph-mon[79417]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:08.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.321 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.322 INFO:journalctl@ceph.osd.4.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:29:08.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:08.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:08.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:07 smithi016 ceph-mon[38248]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:08.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:08.518 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:29:09.112 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:08 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:09.302 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:08 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:09.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:08 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:09.717 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 05 23:29:09 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-4[92155]: 2024-04-05T23:29:09.440+0000 7f599daf07c0 -1 osd.4 0 log_to_monitors true 2024-04-05T23:29:09.828 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-05T23:29:09.828 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-05T23:29:09.828 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-05T23:29:09.828 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-05T23:29:09.828 INFO:teuthology.orchestra.run.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108687 s, 965 MB/s 2024-04-05T23:29:09.828 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-05T23:29:10.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:09 smithi130 ceph-mon[79417]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:10.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:09 smithi130 ceph-mon[79417]: from='osd.4 [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T23:29:10.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:09 smithi130 ceph-mon[79417]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T23:29:10.144 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:09 smithi027 ceph-mon[78597]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:10.144 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:09 smithi027 ceph-mon[78597]: from='osd.4 [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T23:29:10.144 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:09 smithi027 ceph-mon[78597]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T23:29:10.205 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi130:vg_nvme/lv_3 2024-04-05T23:29:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:09 smithi016 ceph-mon[38248]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:09 smithi016 ceph-mon[38248]: from='osd.4 [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T23:29:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:09 smithi016 ceph-mon[38248]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T23:29:10.460 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:29:11.185 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: osdmap e28: 5 total, 4 up, 5 in 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='osd.4 [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:11.187 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.217 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-4[92155]: 2024-04-05T23:29:10.908+0000 7f5992954700 -1 osd.4 0 waiting for initial osdmap 2024-04-05T23:29:11.217 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-4[92155]: 2024-04-05T23:29:10.915+0000 7f598ef4a700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:29:11.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T23:29:11.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: osdmap e28: 5 total, 4 up, 5 in 2024-04-05T23:29:11.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:11.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='osd.4 [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:29:11.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:29:11.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:11.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:11.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:11.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:11.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T23:29:11.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: osdmap e28: 5 total, 4 up, 5 in 2024-04-05T23:29:11.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:11.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='osd.4 [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:29:11.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:29:11.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:11.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:11.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:11.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:11.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:11.362 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.368+0000 7fe34e9b3700 1 -- 172.21.15.130:0/4267986320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3480ffe40 msgr2=0x7fe348100260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:11.362 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.368+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/4267986320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3480ffe40 0x7fe348100260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe330009990 tx=0x7fe330009ca0 comp rx=0 tx=0).stop 2024-04-05T23:29:11.362 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.368+0000 7fe34e9b3700 1 -- 172.21.15.130:0/4267986320 shutdown_connections 2024-04-05T23:29:11.362 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.368+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/4267986320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101960 0x7fe348108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:11.362 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.368+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/4267986320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3480ffe40 0x7fe348100260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:11.362 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.368+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/4267986320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348100fa0 0x7fe348101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:11.362 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.368+0000 7fe34e9b3700 1 -- 172.21.15.130:0/4267986320 >> 172.21.15.130:0/4267986320 conn(0x7fe3480fb400 msgr2=0x7fe3480fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:29:11.363 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.368+0000 7fe34e9b3700 1 -- 172.21.15.130:0/4267986320 shutdown_connections 2024-04-05T23:29:11.363 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.368+0000 7fe34e9b3700 1 -- 172.21.15.130:0/4267986320 wait complete. 2024-04-05T23:29:11.364 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe34e9b3700 1 Processor -- start 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe34e9b3700 1 -- start start 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe34e9b3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3480ffe40 0x7fe348069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe34e9b3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe348100fa0 0x7fe34819bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe34e9b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101960 0x7fe34819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe34e9b3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34819e8b0 con 0x7fe3480ffe40 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe34e9b3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34819ea20 con 0x7fe348101960 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe34e9b3700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34819ebe0 con 0x7fe348100fa0 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe3477fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe348100fa0 0x7fe34819bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe3477fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe348100fa0 0x7fe34819bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:49074/0 (socket says 172.21.15.130:49074) 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe3477fe700 1 -- 172.21.15.130:0/3051635950 learned_addr learned my addr 172.21.15.130:0/3051635950 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe3477fe700 1 -- 172.21.15.130:0/3051635950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101960 msgr2=0x7fe34819e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.369+0000 7fe3477fe700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101960 0x7fe34819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.370+0000 7fe3477fe700 1 -- 172.21.15.130:0/3051635950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3480ffe40 msgr2=0x7fe348069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.370+0000 7fe3477fe700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3480ffe40 0x7fe348069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:11.365 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.370+0000 7fe3477fe700 1 -- 172.21.15.130:0/3051635950 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3300095f0 con 0x7fe348100fa0 2024-04-05T23:29:11.366 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.370+0000 7fe3477fe700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe348100fa0 0x7fe34819bb40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe33800b530 tx=0x7fe33800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:29:11.366 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.370+0000 7fe3457fa700 1 -- 172.21.15.130:0/3051635950 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338011830 con 0x7fe348100fa0 2024-04-05T23:29:11.366 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.370+0000 7fe3457fa700 1 -- 172.21.15.130:0/3051635950 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe338011e50 con 0x7fe348100fa0 2024-04-05T23:29:11.366 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.370+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34819eec0 con 0x7fe348100fa0 2024-04-05T23:29:11.366 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.370+0000 7fe3457fa700 1 -- 172.21.15.130:0/3051635950 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33800f2e0 con 0x7fe348100fa0 2024-04-05T23:29:11.366 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.370+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34819f330 con 0x7fe348100fa0 2024-04-05T23:29:11.366 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.371+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe348107260 con 0x7fe348100fa0 2024-04-05T23:29:11.367 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.372+0000 7fe3457fa700 1 -- 172.21.15.130:0/3051635950 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fe338010580 con 0x7fe348100fa0 2024-04-05T23:29:11.367 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.373+0000 7fe3457fa700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe3340a1120 0x7fe3340a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:11.367 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.373+0000 7fe3457fa700 1 -- 172.21.15.130:0/3051635950 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3821+0+0 (secure 0 0 0) 0x7fe3380cb7c0 con 0x7fe348100fa0 2024-04-05T23:29:11.367 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.373+0000 7fe347fff700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe3340a1120 0x7fe3340a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:29:11.367 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.374+0000 7fe347fff700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe3340a1120 0x7fe3340a35e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe3300037f0 tx=0x7fe330005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:29:11.370 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.377+0000 7fe3457fa700 1 -- 172.21.15.130:0/3051635950 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe3380960c0 con 0x7fe348100fa0 2024-04-05T23:29:11.569 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:11.575+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe34802ce50 con 0x7fe3340a1120 2024-04-05T23:29:12.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi027 2024-04-05T23:29:12.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi027 to 8556M 2024-04-05T23:29:12.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:29:12.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: osdmap e29: 5 total, 4 up, 5 in 2024-04-05T23:29:12.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:12.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:12.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:12.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:29:12.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:29:12.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:12.242 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi027 2024-04-05T23:29:12.242 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi027 to 8556M 2024-04-05T23:29:12.243 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:29:12.243 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: osdmap e29: 5 total, 4 up, 5 in 2024-04-05T23:29:12.243 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:12.243 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:12.243 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:12.243 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:29:12.243 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:29:12.243 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi027 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi027 to 8556M 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: osdmap e29: 5 total, 4 up, 5 in 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 132 MiB used, 357 GiB / 358 GiB avail 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:29:12.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:13.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:12 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:29:13.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:12 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:29:13.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:12 smithi027 ceph-mon[78597]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:29:13.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:12 smithi027 ceph-mon[78597]: osd.4 [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] boot 2024-04-05T23:29:13.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:12 smithi027 ceph-mon[78597]: osdmap e30: 5 total, 5 up, 5 in 2024-04-05T23:29:13.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:12 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:13.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:12 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:29:13.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:12 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:29:13.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:12 smithi016 ceph-mon[38248]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:29:13.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:12 smithi016 ceph-mon[38248]: osd.4 [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] boot 2024-04-05T23:29:13.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:12 smithi016 ceph-mon[38248]: osdmap e30: 5 total, 5 up, 5 in 2024-04-05T23:29:13.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:12 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:13.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:12 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:29:13.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:12 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:29:13.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:12 smithi130 ceph-mon[79417]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:29:13.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:12 smithi130 ceph-mon[79417]: osd.4 [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] boot 2024-04-05T23:29:13.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:12 smithi130 ceph-mon[79417]: osdmap e30: 5 total, 5 up, 5 in 2024-04-05T23:29:13.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:12 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:29:14.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:13 smithi027 ceph-mon[78597]: osdmap e31: 5 total, 5 up, 5 in 2024-04-05T23:29:14.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:13 smithi027 ceph-mon[78597]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:14.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:13 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/742948704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d921f6c-abd3-47b6-a898-5f5bf0afd5d2"}]: dispatch 2024-04-05T23:29:14.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:13 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d921f6c-abd3-47b6-a898-5f5bf0afd5d2"}]: dispatch 2024-04-05T23:29:14.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:13 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d921f6c-abd3-47b6-a898-5f5bf0afd5d2"}]': finished 2024-04-05T23:29:14.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:13 smithi027 ceph-mon[78597]: osdmap e32: 6 total, 5 up, 6 in 2024-04-05T23:29:14.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:14.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:13 smithi016 ceph-mon[38248]: osdmap e31: 5 total, 5 up, 5 in 2024-04-05T23:29:14.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:13 smithi016 ceph-mon[38248]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:14.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:13 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/742948704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d921f6c-abd3-47b6-a898-5f5bf0afd5d2"}]: dispatch 2024-04-05T23:29:14.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:13 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d921f6c-abd3-47b6-a898-5f5bf0afd5d2"}]: dispatch 2024-04-05T23:29:14.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:13 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d921f6c-abd3-47b6-a898-5f5bf0afd5d2"}]': finished 2024-04-05T23:29:14.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:13 smithi016 ceph-mon[38248]: osdmap e32: 6 total, 5 up, 6 in 2024-04-05T23:29:14.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:14.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:13 smithi130 ceph-mon[79417]: osdmap e31: 5 total, 5 up, 5 in 2024-04-05T23:29:14.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:13 smithi130 ceph-mon[79417]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:14.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:13 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/742948704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d921f6c-abd3-47b6-a898-5f5bf0afd5d2"}]: dispatch 2024-04-05T23:29:14.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:13 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d921f6c-abd3-47b6-a898-5f5bf0afd5d2"}]: dispatch 2024-04-05T23:29:14.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:13 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d921f6c-abd3-47b6-a898-5f5bf0afd5d2"}]': finished 2024-04-05T23:29:14.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:13 smithi130 ceph-mon[79417]: osdmap e32: 6 total, 5 up, 6 in 2024-04-05T23:29:14.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:15.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:14 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/447011674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:29:15.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:14 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/447011674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:29:15.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:14 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/447011674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:29:16.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:15 smithi027 ceph-mon[78597]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:16.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:15 smithi016 ceph-mon[38248]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:16.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:15 smithi130 ceph-mon[79417]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:18.566 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:18 smithi130 ceph-mon[79417]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:18.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:18 smithi016 ceph-mon[38248]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:18.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:18 smithi027 ceph-mon[78597]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:20.477 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:20 smithi130 ceph-mon[79417]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:20.478 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:20 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T23:29:20.478 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:20 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:20.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:20 smithi016 ceph-mon[38248]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:20.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:20 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T23:29:20.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:20 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:20.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:20 smithi027 ceph-mon[78597]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:20.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:20 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T23:29:20.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:20 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:21.560 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:21 smithi130 ceph-mon[79417]: Deploying daemon osd.5 on smithi130 2024-04-05T23:29:21.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:21 smithi016 ceph-mon[38248]: Deploying daemon osd.5 on smithi130 2024-04-05T23:29:21.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:21 smithi027 ceph-mon[78597]: Deploying daemon osd.5 on smithi130 2024-04-05T23:29:22.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:22 smithi016 ceph-mon[38248]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:22.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:22 smithi130 ceph-mon[79417]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:22.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:22 smithi027 ceph-mon[78597]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:24.503 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:24 smithi130 ceph-mon[79417]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:24.503 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:24.503 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:24.503 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:24 smithi016 ceph-mon[38248]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:24 smithi027 ceph-mon[78597]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:25.831 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 5 on host 'smithi130' 2024-04-05T23:29:25.832 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.838+0000 7fe3457fa700 1 -- 172.21.15.130:0/3051635950 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe34802ce50 con 0x7fe3340a1120 2024-04-05T23:29:25.833 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe3340a1120 msgr2=0x7fe3340a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:25.833 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe3340a1120 0x7fe3340a35e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe3300037f0 tx=0x7fe330005fb0 comp rx=0 tx=0).stop 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe348100fa0 msgr2=0x7fe34819bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe348100fa0 0x7fe34819bb40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe33800b530 tx=0x7fe33800b8f0 comp rx=0 tx=0).stop 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 shutdown_connections 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe3340a1120 0x7fe3340a35e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3480ffe40 0x7fe348069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101960 0x7fe34819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 --2- 172.21.15.130:0/3051635950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe348100fa0 0x7fe34819bb40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 >> 172.21.15.130:0/3051635950 conn(0x7fe3480fb400 msgr2=0x7fe3481043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 shutdown_connections 2024-04-05T23:29:25.834 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:29:25.841+0000 7fe34e9b3700 1 -- 172.21.15.130:0/3051635950 wait complete. 2024-04-05T23:29:26.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:25 smithi130 ceph-mon[79417]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:26.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:26.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:26.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:26.262 DEBUG:teuthology.orchestra.run.smithi130:osd.5> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-05T23:29:26.265 INFO:tasks.cephadm:Deploying osd.6 on smithi016 with /dev/vg_nvme/lv_2... 2024-04-05T23:29:26.265 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-05T23:29:26.294 INFO:journalctl@ceph.osd.5.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:29:26.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:25 smithi016 ceph-mon[38248]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:26.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:26.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:26.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:26.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:25 smithi027 ceph-mon[78597]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:26.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:26.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:26.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:26.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:29:27.834 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:29:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:29:27.493+0000 7f6eaba1f7c0 -1 osd.5 0 log_to_monitors true 2024-04-05T23:29:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-05T23:29:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-05T23:29:28.327 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-05T23:29:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-05T23:29:28.327 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0394349 s, 266 MB/s 2024-04-05T23:29:28.327 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-05T23:29:28.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:28 smithi016 ceph-mon[38248]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:28.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:28 smithi016 ceph-mon[38248]: from='osd.5 [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T23:29:28.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:28 smithi016 ceph-mon[38248]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T23:29:28.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:28 smithi130 ceph-mon[79417]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:28.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:28 smithi130 ceph-mon[79417]: from='osd.5 [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T23:29:28.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:28 smithi130 ceph-mon[79417]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T23:29:28.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:28 smithi027 ceph-mon[78597]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:28.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:28 smithi027 ceph-mon[78597]: from='osd.5 [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T23:29:28.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:28 smithi027 ceph-mon[78597]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T23:29:28.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-04-05T23:29:29.104 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:29:29.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: osdmap e33: 6 total, 5 up, 6 in 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='osd.5 [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi130 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi130 to 8556M 2024-04-05T23:29:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:29.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:29.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.584 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:29:29.335+0000 7f6ea0883700 -1 osd.5 0 waiting for initial osdmap 2024-04-05T23:29:29.584 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:29:29.343+0000 7f6e9ce79700 -1 osd.5 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:29:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T23:29:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: osdmap e33: 6 total, 5 up, 6 in 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='osd.5 [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi130 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi130 to 8556M 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: osdmap e33: 6 total, 5 up, 6 in 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='osd.5 [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi130 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi130 to 8556M 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:29.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:29.719 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.264+0000 7ffb85c00700 1 -- 172.21.15.16:0/3660960361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 msgr2=0x7ffb80103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.264+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3660960361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 0x7ffb80103dc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009990 tx=0x7ffb70009ca0 comp rx=0 tx=0).stop 2024-04-05T23:29:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.264+0000 7ffb85c00700 1 -- 172.21.15.16:0/3660960361 shutdown_connections 2024-04-05T23:29:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.264+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3660960361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 0x7ffb80103dc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.264+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3660960361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb80106c30 0x7ffb801090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.264+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3660960361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb80104300 0x7ffb801066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.264+0000 7ffb85c00700 1 -- 172.21.15.16:0/3660960361 >> 172.21.15.16:0/3660960361 conn(0x7ffb800fb3e0 msgr2=0x7ffb800fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:29:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.264+0000 7ffb85c00700 1 -- 172.21.15.16:0/3660960361 shutdown_connections 2024-04-05T23:29:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.264+0000 7ffb85c00700 1 -- 172.21.15.16:0/3660960361 wait complete. 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.265+0000 7ffb85c00700 1 Processor -- start 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.265+0000 7ffb85c00700 1 -- start start 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb85c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 0x7ffb80199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb85c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb80104300 0x7ffb80199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb85c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb80106c30 0x7ffb801a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb85c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801a0b60 con 0x7ffb801019d0 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb85c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801a0ca0 con 0x7ffb80104300 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb85c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801a0e30 con 0x7ffb80106c30 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb7ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 0x7ffb80199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb7ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 0x7ffb80199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58788/0 (socket says 172.21.15.16:58788) 2024-04-05T23:29:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb7ea00700 1 -- 172.21.15.16:0/3876189972 learned_addr learned my addr 172.21.15.16:0/3876189972 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb7e000700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb80104300 0x7ffb80199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb7f400700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb80106c30 0x7ffb801a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb7ea00700 1 -- 172.21.15.16:0/3876189972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb80106c30 msgr2=0x7ffb801a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb7ea00700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb80106c30 0x7ffb801a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb7ea00700 1 -- 172.21.15.16:0/3876189972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb80104300 msgr2=0x7ffb80199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.266+0000 7ffb7ea00700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb80104300 0x7ffb80199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.267+0000 7ffb7ea00700 1 -- 172.21.15.16:0/3876189972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700095f0 con 0x7ffb801019d0 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.267+0000 7ffb7e000700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb80104300 0x7ffb80199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.267+0000 7ffb7ea00700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 0x7ffb80199020 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffb70000c00 tx=0x7ffb70005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.267+0000 7ffb77400700 1 -- 172.21.15.16:0/3876189972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7001d070 con 0x7ffb801019d0 2024-04-05T23:29:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.267+0000 7ffb77400700 1 -- 172.21.15.16:0/3876189972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb70022470 con 0x7ffb801019d0 2024-04-05T23:29:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.267+0000 7ffb77400700 1 -- 172.21.15.16:0/3876189972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70017650 con 0x7ffb801019d0 2024-04-05T23:29:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.267+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb801a10b0 con 0x7ffb801019d0 2024-04-05T23:29:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.267+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb801a1470 con 0x7ffb801019d0 2024-04-05T23:29:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.268+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb80108800 con 0x7ffb801019d0 2024-04-05T23:29:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.270+0000 7ffb77400700 1 -- 172.21.15.16:0/3876189972 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7ffb700225e0 con 0x7ffb801019d0 2024-04-05T23:29:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.271+0000 7ffb77400700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb640a11a0 0x7ffb640a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.272+0000 7ffb77400700 1 -- 172.21.15.16:0/3876189972 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4566+0+0 (secure 0 0 0) 0x7ffb700cd5c0 con 0x7ffb801019d0 2024-04-05T23:29:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.272+0000 7ffb7e000700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb640a11a0 0x7ffb640a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:29:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.272+0000 7ffb7e000700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb640a11a0 0x7ffb640a3660 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c011490 tx=0x7ffb6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:29:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.275+0000 7ffb77400700 1 -- 172.21.15.16:0/3876189972 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ffb70097be0 con 0x7ffb801019d0 2024-04-05T23:29:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:30.501+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb80061320 con 0x7ffb640a11a0 2024-04-05T23:29:30.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:30 smithi016 ceph-mon[38248]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:30 smithi016 ceph-mon[38248]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:29:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:30 smithi016 ceph-mon[38248]: osdmap e34: 6 total, 5 up, 6 in 2024-04-05T23:29:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:30 smithi016 ceph-mon[38248]: osd.5 [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] boot 2024-04-05T23:29:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:30 smithi016 ceph-mon[38248]: osdmap e35: 6 total, 6 up, 6 in 2024-04-05T23:29:30.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:30 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:30.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:30 smithi130 ceph-mon[79417]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:30.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:30 smithi130 ceph-mon[79417]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:29:30.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:30 smithi130 ceph-mon[79417]: osdmap e34: 6 total, 5 up, 6 in 2024-04-05T23:29:30.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:30.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:30.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:30 smithi130 ceph-mon[79417]: osd.5 [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] boot 2024-04-05T23:29:30.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:30 smithi130 ceph-mon[79417]: osdmap e35: 6 total, 6 up, 6 in 2024-04-05T23:29:30.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:30 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:30 smithi027 ceph-mon[78597]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-05T23:29:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:30 smithi027 ceph-mon[78597]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:29:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:30 smithi027 ceph-mon[78597]: osdmap e34: 6 total, 5 up, 6 in 2024-04-05T23:29:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:30 smithi027 ceph-mon[78597]: osd.5 [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] boot 2024-04-05T23:29:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:30 smithi027 ceph-mon[78597]: osdmap e35: 6 total, 6 up, 6 in 2024-04-05T23:29:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:30 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:29:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:31 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:29:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:31 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:29:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:31 smithi027 ceph-mon[78597]: osdmap e36: 6 total, 6 up, 6 in 2024-04-05T23:29:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:29:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:29:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:31.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:31 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:29:31.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:31 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:29:31.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:31 smithi016 ceph-mon[38248]: osdmap e36: 6 total, 6 up, 6 in 2024-04-05T23:29:31.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:29:31.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:29:31.828 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:31.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:31 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:29:31.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:31 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:29:31.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:31 smithi130 ceph-mon[79417]: osdmap e36: 6 total, 6 up, 6 in 2024-04-05T23:29:31.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:29:31.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:29:31.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:32.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:32 smithi027 ceph-mon[78597]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:29:32.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:32 smithi027 ceph-mon[78597]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:32.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:32 smithi027 ceph-mon[78597]: osdmap e37: 6 total, 6 up, 6 in 2024-04-05T23:29:32.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:32 smithi016 ceph-mon[38248]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:29:32.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:32 smithi016 ceph-mon[38248]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:32.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:32 smithi016 ceph-mon[38248]: osdmap e37: 6 total, 6 up, 6 in 2024-04-05T23:29:32.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:32 smithi130 ceph-mon[79417]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:29:32.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:32 smithi130 ceph-mon[79417]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:32.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:32 smithi130 ceph-mon[79417]: osdmap e37: 6 total, 6 up, 6 in 2024-04-05T23:29:33.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:33 smithi027 ceph-mon[78597]: osdmap e38: 6 total, 6 up, 6 in 2024-04-05T23:29:33.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:33 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/326389652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6"}]: dispatch 2024-04-05T23:29:33.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:33 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/326389652' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6"}]': finished 2024-04-05T23:29:33.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:33 smithi027 ceph-mon[78597]: osdmap e39: 7 total, 6 up, 7 in 2024-04-05T23:29:33.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:33.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:33 smithi016 ceph-mon[38248]: osdmap e38: 6 total, 6 up, 6 in 2024-04-05T23:29:33.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:33 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/326389652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6"}]: dispatch 2024-04-05T23:29:33.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:33 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/326389652' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6"}]': finished 2024-04-05T23:29:33.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:33 smithi016 ceph-mon[38248]: osdmap e39: 7 total, 6 up, 7 in 2024-04-05T23:29:33.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:33.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:33 smithi130 ceph-mon[79417]: osdmap e38: 6 total, 6 up, 6 in 2024-04-05T23:29:33.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:33 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/326389652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6"}]: dispatch 2024-04-05T23:29:33.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:33 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/326389652' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6"}]': finished 2024-04-05T23:29:33.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:33 smithi130 ceph-mon[79417]: osdmap e39: 7 total, 6 up, 7 in 2024-04-05T23:29:33.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:34.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:34 smithi027 ceph-mon[78597]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:34.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:34 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/122232358' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:29:34.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:34 smithi016 ceph-mon[38248]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:34.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:34 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/122232358' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:29:34.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:34 smithi130 ceph-mon[79417]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:34.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:34 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/122232358' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:29:36.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:36 smithi027 ceph-mon[78597]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:36.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:36 smithi016 ceph-mon[38248]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:36.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:36 smithi130 ceph-mon[79417]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:38.460 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:38 smithi016 ceph-mon[38248]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T23:29:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:38 smithi027 ceph-mon[78597]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T23:29:38.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:38 smithi130 ceph-mon[79417]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T23:29:40.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:40 smithi016 ceph-mon[38248]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-05T23:29:40.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-05T23:29:40.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:40.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:40 smithi027 ceph-mon[78597]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-05T23:29:40.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-05T23:29:40.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:40.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:40 smithi130 ceph-mon[79417]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-05T23:29:40.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-05T23:29:40.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:41.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:41 smithi027 ceph-mon[78597]: Deploying daemon osd.6 on smithi016 2024-04-05T23:29:41.818 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:41 smithi016 ceph-mon[38248]: Deploying daemon osd.6 on smithi016 2024-04-05T23:29:41.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:41 smithi130 ceph-mon[79417]: Deploying daemon osd.6 on smithi016 2024-04-05T23:29:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:42 smithi016 ceph-mon[38248]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-05T23:29:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:42 smithi027 ceph-mon[78597]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-05T23:29:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:42 smithi130 ceph-mon[79417]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-05T23:29:44.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:44 smithi027 ceph-mon[78597]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-05T23:29:44.812 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:44 smithi016 ceph-mon[38248]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-05T23:29:44.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:44 smithi130 ceph-mon[79417]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-05T23:29:45.999 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:45.999 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:46.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:46.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:45 smithi016 ceph-mon[38248]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-05T23:29:46.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:46.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:46.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:46.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:45 smithi027 ceph-mon[78597]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-05T23:29:46.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:46.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:46.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:46.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:45 smithi130 ceph-mon[79417]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-05T23:29:48.001 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 6 on host 'smithi016' 2024-04-05T23:29:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.007+0000 7ffb77400700 1 -- 172.21.15.16:0/3876189972 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffb80061320 con 0x7ffb640a11a0 2024-04-05T23:29:48.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.009+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb640a11a0 msgr2=0x7ffb640a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:48.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.009+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb640a11a0 0x7ffb640a3660 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c011490 tx=0x7ffb6c009260 comp rx=0 tx=0).stop 2024-04-05T23:29:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 msgr2=0x7ffb80199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 0x7ffb80199020 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffb70000c00 tx=0x7ffb70005720 comp rx=0 tx=0).stop 2024-04-05T23:29:48.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 shutdown_connections 2024-04-05T23:29:48.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb640a11a0 0x7ffb640a3660 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:48.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb801019d0 0x7ffb80199020 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:48.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb80106c30 0x7ffb801a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:48.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 --2- 172.21.15.16:0/3876189972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb80104300 0x7ffb80199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:48.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 >> 172.21.15.16:0/3876189972 conn(0x7ffb800fb3e0 msgr2=0x7ffb80101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:29:48.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 shutdown_connections 2024-04-05T23:29:48.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:29:48.010+0000 7ffb85c00700 1 -- 172.21.15.16:0/3876189972 wait complete. 2024-04-05T23:29:48.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:48.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:48.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-mon[38248]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-05T23:29:48.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:48.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:48.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:48.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:48 smithi130 ceph-mon[79417]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-05T23:29:48.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:48.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:48.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:48.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:48 smithi027 ceph-mon[78597]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-05T23:29:48.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:48.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:29:48.483 DEBUG:teuthology.orchestra.run.smithi016:osd.6> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.6.service 2024-04-05T23:29:48.487 INFO:tasks.cephadm:Deploying osd.7 on smithi027 with /dev/vg_nvme/lv_2... 2024-04-05T23:29:48.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-05T23:29:48.559 INFO:journalctl@ceph.osd.6.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:29:48.726 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:29:49.075 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:29:48 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:29:48.992+0000 7f94d46777c0 -1 osd.6 0 log_to_monitors true 2024-04-05T23:29:49.153 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:49 smithi027 ceph-mon[78597]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-05T23:29:49.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:49 smithi016 ceph-mon[38248]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-05T23:29:49.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:49 smithi130 ceph-mon[79417]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-05T23:29:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-05T23:29:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-05T23:29:50.081 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-05T23:29:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-05T23:29:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110658 s, 948 MB/s 2024-04-05T23:29:50.081 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 2024-04-05T23:29:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-05T23:29:50.390 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:50 smithi027 ceph-mon[78597]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-05T23:29:50.390 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:50 smithi027 ceph-mon[78597]: osdmap e40: 7 total, 6 up, 7 in 2024-04-05T23:29:50.390 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:50.390 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:50 smithi027 ceph-mon[78597]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:29:50.390 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:50 smithi027 ceph-mon[78597]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:50.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi027:vg_nvme/lv_2 2024-04-05T23:29:50.583 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:29:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:29:50.116+0000 7f94c8200700 -1 osd.6 0 waiting for initial osdmap 2024-04-05T23:29:50.583 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:29:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:29:50.127+0000 7f94c3a00700 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:29:50.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:50 smithi016 ceph-mon[38248]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-05T23:29:50.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:50 smithi016 ceph-mon[38248]: osdmap e40: 7 total, 6 up, 7 in 2024-04-05T23:29:50.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:50.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:50 smithi016 ceph-mon[38248]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:29:50.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:50 smithi016 ceph-mon[38248]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:50.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:50 smithi130 ceph-mon[79417]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-05T23:29:50.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:50 smithi130 ceph-mon[79417]: osdmap e40: 7 total, 6 up, 7 in 2024-04-05T23:29:50.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:50.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:50 smithi130 ceph-mon[79417]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:29:50.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:50 smithi130 ceph-mon[79417]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-05T23:29:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:51 smithi016 ceph-mon[38248]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:51 smithi016 ceph-mon[38248]: osdmap e41: 7 total, 6 up, 7 in 2024-04-05T23:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:51.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:51 smithi027 ceph-mon[78597]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:29:51.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:51 smithi027 ceph-mon[78597]: osdmap e41: 7 total, 6 up, 7 in 2024-04-05T23:29:51.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:51.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:51.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:51 smithi130 ceph-mon[79417]: from='osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:29:51.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:51 smithi130 ceph-mon[79417]: osdmap e41: 7 total, 6 up, 7 in 2024-04-05T23:29:51.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:51.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:51.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 -- 172.21.15.27:0/527663263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102a40 msgr2=0x7f1594109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 --2- 172.21.15.27:0/527663263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102a40 0x7f1594109360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f158c00b0a0 tx=0x7f158c00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:29:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 -- 172.21.15.27:0/527663263 shutdown_connections 2024-04-05T23:29:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 --2- 172.21.15.27:0/527663263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102a40 0x7f1594109360 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 --2- 172.21.15.27:0/527663263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1594100f20 0x7f1594101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 --2- 172.21.15.27:0/527663263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594102080 0x7f1594102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 -- 172.21.15.27:0/527663263 >> 172.21.15.27:0/527663263 conn(0x7f15940fc4c0 msgr2=0x7f15940fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:29:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 -- 172.21.15.27:0/527663263 shutdown_connections 2024-04-05T23:29:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 -- 172.21.15.27:0/527663263 wait complete. 2024-04-05T23:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 Processor -- start 2024-04-05T23:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.753+0000 7f159bbd9700 1 -- start start 2024-04-05T23:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f159bbd9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594100f20 0x7f1594074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f159bbd9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102080 0x7f15940732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f159bbd9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1594102a40 0x7f1594073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f159bbd9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594073fb0 con 0x7f1594100f20 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f159bbd9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594074120 con 0x7f1594102080 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f159bbd9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15941a0ff0 con 0x7f1594102a40 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f1599174700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102080 0x7f15940732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f1599174700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102080 0x7f15940732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56840/0 (socket says 172.21.15.27:56840) 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f1599174700 1 -- 172.21.15.27:0/3684647000 learned_addr learned my addr 172.21.15.27:0/3684647000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f159a176700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1594102a40 0x7f1594073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f1599174700 1 -- 172.21.15.27:0/3684647000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1594102a40 msgr2=0x7f1594073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f1599174700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1594102a40 0x7f1594073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f1599174700 1 -- 172.21.15.27:0/3684647000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594100f20 msgr2=0x7f1594074c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f1599174700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594100f20 0x7f1594074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f1599174700 1 -- 172.21.15.27:0/3684647000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c009d00 con 0x7f1594102080 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f1599174700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102080 0x7f15940732d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f158400b530 tx=0x7f158400b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f158affd700 1 -- 172.21.15.27:0/3684647000 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1584011830 con 0x7f1594102080 2024-04-05T23:29:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f158affd700 1 -- 172.21.15.27:0/3684647000 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1584011e50 con 0x7f1594102080 2024-04-05T23:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.754+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15941a1280 con 0x7f1594102080 2024-04-05T23:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.755+0000 7f158affd700 1 -- 172.21.15.27:0/3684647000 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158400f2e0 con 0x7f1594102080 2024-04-05T23:29:51.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.755+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15941a1780 con 0x7f1594102080 2024-04-05T23:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.756+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1594108320 con 0x7f1594102080 2024-04-05T23:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.757+0000 7f158affd700 1 -- 172.21.15.27:0/3684647000 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f1584010580 con 0x7f1594102080 2024-04-05T23:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.757+0000 7f158affd700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f15800a1120 0x7f15800a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.758+0000 7f158affd700 1 -- 172.21.15.27:0/3684647000 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f15840cb710 con 0x7f1594102080 2024-04-05T23:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.758+0000 7f1599975700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f15800a1120 0x7f15800a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:29:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.758+0000 7f1599975700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f15800a1120 0x7f15800a35e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1590005950 tx=0x7f15900058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:29:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.762+0000 7f158affd700 1 -- 172.21.15.27:0/3684647000 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1584095bb0 con 0x7f1594102080 2024-04-05T23:29:51.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:29:51.974+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1594061320 con 0x7f15800a1120 2024-04-05T23:29:52.403 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:29:52.403 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:29:52.403 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] boot 2024-04-05T23:29:52.403 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: osdmap e42: 7 total, 7 up, 7 in 2024-04-05T23:29:52.403 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:52.403 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:52.403 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.403 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.403 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:52.404 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:52.404 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:52.404 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.404 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:52.404 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:52.404 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.404 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:29:52.404 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:29:52.404 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:52 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:52.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] boot 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: osdmap e42: 7 total, 7 up, 7 in 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:52.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:52.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:29:52.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:29:52.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:52 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:52.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:29:52.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:29:52.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: osd.6 [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] boot 2024-04-05T23:29:52.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: osdmap e42: 7 total, 7 up, 7 in 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:29:52.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:52 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:29:53.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:53 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi016 2024-04-05T23:29:53.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:53 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi016 to 5758M 2024-04-05T23:29:53.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:53 smithi027 ceph-mon[78597]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:29:53.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:53 smithi027 ceph-mon[78597]: osdmap e43: 7 total, 7 up, 7 in 2024-04-05T23:29:53.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:53 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi016 2024-04-05T23:29:53.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:53 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi016 to 5758M 2024-04-05T23:29:53.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:53 smithi016 ceph-mon[38248]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:29:53.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:53 smithi016 ceph-mon[38248]: osdmap e43: 7 total, 7 up, 7 in 2024-04-05T23:29:53.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:53 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi016 2024-04-05T23:29:53.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:53 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi016 to 5758M 2024-04-05T23:29:53.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:53 smithi130 ceph-mon[79417]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:29:53.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:53 smithi130 ceph-mon[79417]: osdmap e43: 7 total, 7 up, 7 in 2024-04-05T23:29:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:54 smithi027 ceph-mon[78597]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:54 smithi027 ceph-mon[78597]: from='client.? 172.21.15.27:0/1749425094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc642843-7b4c-48ea-9c1c-74345e3da8df"}]: dispatch 2024-04-05T23:29:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:54 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc642843-7b4c-48ea-9c1c-74345e3da8df"}]: dispatch 2024-04-05T23:29:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:54 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc642843-7b4c-48ea-9c1c-74345e3da8df"}]': finished 2024-04-05T23:29:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:54 smithi027 ceph-mon[78597]: osdmap e44: 8 total, 7 up, 8 in 2024-04-05T23:29:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:29:54.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:54 smithi016 ceph-mon[38248]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:54.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:54 smithi016 ceph-mon[38248]: from='client.? 172.21.15.27:0/1749425094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc642843-7b4c-48ea-9c1c-74345e3da8df"}]: dispatch 2024-04-05T23:29:54.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:54 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc642843-7b4c-48ea-9c1c-74345e3da8df"}]: dispatch 2024-04-05T23:29:54.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:54 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc642843-7b4c-48ea-9c1c-74345e3da8df"}]': finished 2024-04-05T23:29:54.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:54 smithi016 ceph-mon[38248]: osdmap e44: 8 total, 7 up, 8 in 2024-04-05T23:29:54.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:29:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:54 smithi130 ceph-mon[79417]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:54 smithi130 ceph-mon[79417]: from='client.? 172.21.15.27:0/1749425094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc642843-7b4c-48ea-9c1c-74345e3da8df"}]: dispatch 2024-04-05T23:29:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:54 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc642843-7b4c-48ea-9c1c-74345e3da8df"}]: dispatch 2024-04-05T23:29:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:54 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc642843-7b4c-48ea-9c1c-74345e3da8df"}]': finished 2024-04-05T23:29:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:54 smithi130 ceph-mon[79417]: osdmap e44: 8 total, 7 up, 8 in 2024-04-05T23:29:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:29:55.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:55 smithi027 ceph-mon[78597]: from='client.? 172.21.15.27:0/1847815132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:29:55.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:55 smithi016 ceph-mon[38248]: from='client.? 172.21.15.27:0/1847815132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:29:55.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:55 smithi130 ceph-mon[79417]: from='client.? 172.21.15.27:0/1847815132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:29:56.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:56 smithi027 ceph-mon[78597]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:56.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:56 smithi016 ceph-mon[38248]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:56.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:56 smithi130 ceph-mon[79417]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:58.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:29:58 smithi027 ceph-mon[78597]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:29:58 smithi016 ceph-mon[38248]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:29:58.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:29:58 smithi130 ceph-mon[79417]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:00 smithi027 ceph-mon[78597]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:00 smithi027 ceph-mon[78597]: overall HEALTH_OK 2024-04-05T23:30:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:00 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-05T23:30:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:00 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:00.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:00 smithi016 ceph-mon[38248]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:00 smithi016 ceph-mon[38248]: overall HEALTH_OK 2024-04-05T23:30:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:00 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-05T23:30:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:00 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:00 smithi130 ceph-mon[79417]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:00 smithi130 ceph-mon[79417]: overall HEALTH_OK 2024-04-05T23:30:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:00 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-05T23:30:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:00 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:01.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:01 smithi016 ceph-mon[38248]: Deploying daemon osd.7 on smithi027 2024-04-05T23:30:01.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:01 smithi130 ceph-mon[79417]: Deploying daemon osd.7 on smithi027 2024-04-05T23:30:01.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:01 smithi027 ceph-mon[78597]: Deploying daemon osd.7 on smithi027 2024-04-05T23:30:02.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:02 smithi016 ceph-mon[38248]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:02.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:02 smithi130 ceph-mon[79417]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:02.597 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:02 smithi027 ceph-mon[78597]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:04.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:04 smithi016 ceph-mon[38248]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:04.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:04 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:04.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:04 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:04.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:04 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:04 smithi130 ceph-mon[79417]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:04 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:04 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:04 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:04.693 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:04 smithi027 ceph-mon[78597]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:04.694 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:04 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:04.694 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:04 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:04.694 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:04 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:06.121 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 7 on host 'smithi027' 2024-04-05T23:30:06.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.128+0000 7f158affd700 1 -- 172.21.15.27:0/3684647000 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1594061320 con 0x7f15800a1120 2024-04-05T23:30:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f15800a1120 msgr2=0x7f15800a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f15800a1120 0x7f15800a35e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1590005950 tx=0x7f15900058e0 comp rx=0 tx=0).stop 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102080 msgr2=0x7f15940732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102080 0x7f15940732d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f158400b530 tx=0x7f158400b8f0 comp rx=0 tx=0).stop 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 shutdown_connections 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f15800a1120 0x7f15800a35e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594100f20 0x7f1594074c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1594102a40 0x7f1594073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 --2- 172.21.15.27:0/3684647000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1594102080 0x7f15940732d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 >> 172.21.15.27:0/3684647000 conn(0x7f15940fc4c0 msgr2=0x7f1594105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 shutdown_connections 2024-04-05T23:30:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:06.131+0000 7f159bbd9700 1 -- 172.21.15.27:0/3684647000 wait complete. 2024-04-05T23:30:06.510 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.510 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.510 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:06.510 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:06.510 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:06 smithi027 ceph-mon[78597]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:06.510 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.510 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.510 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:06.510 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.512 DEBUG:teuthology.orchestra.run.smithi027:osd.7> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.7.service 2024-04-05T23:30:06.514 INFO:tasks.cephadm:Deploying osd.8 on smithi130 with /dev/vg_nvme/lv_2... 2024-04-05T23:30:06.514 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-05T23:30:06.545 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.545 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.545 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:06.546 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:06.546 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:06 smithi130 ceph-mon[79417]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:06.546 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.546 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.546 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:06.546 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.547 INFO:journalctl@ceph.osd.7.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:30:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:06 smithi016 ceph-mon[38248]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:06.759 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:30:08.102 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-05T23:30:08.102 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-05T23:30:08.102 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-05T23:30:08.102 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-05T23:30:08.102 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108526 s, 966 MB/s 2024-04-05T23:30:08.102 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-05T23:30:08.216 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:30:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:30:07.887+0000 7f1f5b9fe7c0 -1 osd.7 0 log_to_monitors true 2024-04-05T23:30:08.403 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:08 smithi130 ceph-mon[79417]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:08.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:08 smithi130 ceph-mon[79417]: from='osd.7 [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-05T23:30:08.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:08 smithi130 ceph-mon[79417]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-05T23:30:08.440 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi130:vg_nvme/lv_2 2024-04-05T23:30:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:08 smithi016 ceph-mon[38248]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:08 smithi016 ceph-mon[38248]: from='osd.7 [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-05T23:30:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:08 smithi016 ceph-mon[38248]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-05T23:30:08.609 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:08 smithi027 ceph-mon[78597]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:08.609 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:08 smithi027 ceph-mon[78597]: from='osd.7 [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-05T23:30:08.609 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:08 smithi027 ceph-mon[78597]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-05T23:30:08.698 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: osdmap e45: 8 total, 7 up, 8 in 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='osd.7 [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:09.526 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.527 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:09.527 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:09.527 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-05T23:30:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: osdmap e45: 8 total, 7 up, 8 in 2024-04-05T23:30:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='osd.7 [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:30:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:30:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:09.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:09.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:09.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:09.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:09.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.704 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.711+0000 7f1642610700 1 -- 172.21.15.130:0/3678725105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c101930 msgr2=0x7f163c108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:09.704 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.711+0000 7f1642610700 1 --2- 172.21.15.130:0/3678725105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c101930 0x7f163c108250 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f163400b0a0 tx=0x7f163400b3b0 comp rx=0 tx=0).stop 2024-04-05T23:30:09.704 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.711+0000 7f1642610700 1 -- 172.21.15.130:0/3678725105 shutdown_connections 2024-04-05T23:30:09.704 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.711+0000 7f1642610700 1 --2- 172.21.15.130:0/3678725105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c101930 0x7f163c108250 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:09.705 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.711+0000 7f1642610700 1 --2- 172.21.15.130:0/3678725105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f163c0ffe10 0x7f163c100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:09.705 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.711+0000 7f1642610700 1 --2- 172.21.15.130:0/3678725105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f163c100f70 0x7f163c1013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:09.705 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.711+0000 7f1642610700 1 -- 172.21.15.130:0/3678725105 >> 172.21.15.130:0/3678725105 conn(0x7f163c0fb3b0 msgr2=0x7f163c0fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:30:09.705 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.711+0000 7f1642610700 1 -- 172.21.15.130:0/3678725105 shutdown_connections 2024-04-05T23:30:09.705 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.711+0000 7f1642610700 1 -- 172.21.15.130:0/3678725105 wait complete. 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f1642610700 1 Processor -- start 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f1642610700 1 -- start start 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f1642610700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f163c0ffe10 0x7f163c196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f1642610700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c100f70 0x7f163c1972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f1642610700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f163c101930 0x7f163c19e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f1642610700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c19e890 con 0x7f163c0ffe10 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f1642610700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c19e9d0 con 0x7f163c101930 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f1642610700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c19eb90 con 0x7f163c100f70 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f163b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c100f70 0x7f163c1972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:09.707 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f163b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c100f70 0x7f163c1972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:54736/0 (socket says 172.21.15.130:54736) 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f163b7fe700 1 -- 172.21.15.130:0/4120642947 learned_addr learned my addr 172.21.15.130:0/4120642947 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.712+0000 7f1640bad700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f163c101930 0x7f163c19e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f163b7fe700 1 -- 172.21.15.130:0/4120642947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f163c101930 msgr2=0x7f163c19e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f163b7fe700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f163c101930 0x7f163c19e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f163b7fe700 1 -- 172.21.15.130:0/4120642947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f163c0ffe10 msgr2=0x7f163c196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f163b7fe700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f163c0ffe10 0x7f163c196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f163b7fe700 1 -- 172.21.15.130:0/4120642947 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1634009d00 con 0x7f163c100f70 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f163b7fe700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c100f70 0x7f163c1972d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f163000b530 tx=0x7f163000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f16397fa700 1 -- 172.21.15.130:0/4120642947 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1630011830 con 0x7f163c100f70 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f16397fa700 1 -- 172.21.15.130:0/4120642947 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1630011e50 con 0x7f163c100f70 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163c19ee70 con 0x7f163c100f70 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f16397fa700 1 -- 172.21.15.130:0/4120642947 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163000f2e0 con 0x7f163c100f70 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.713+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163c19f330 con 0x7f163c100f70 2024-04-05T23:30:09.708 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.714+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163c04ebb0 con 0x7f163c100f70 2024-04-05T23:30:09.709 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.715+0000 7f16397fa700 1 -- 172.21.15.130:0/4120642947 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f1630010380 con 0x7f163c100f70 2024-04-05T23:30:09.709 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.716+0000 7f16397fa700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16240a11a0 0x7f16240a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:09.709 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.716+0000 7f16397fa700 1 -- 172.21.15.130:0/4120642947 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5150+0+0 (secure 0 0 0) 0x7f16300cb770 con 0x7f163c100f70 2024-04-05T23:30:09.710 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.716+0000 7f163bfff700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16240a11a0 0x7f16240a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:09.710 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.717+0000 7f163bfff700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16240a11a0 0x7f16240a3660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f162c005d90 tx=0x7f162c005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:30:09.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.720+0000 7f16397fa700 1 -- 172.21.15.130:0/4120642947 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1630095b60 con 0x7f163c100f70 2024-04-05T23:30:09.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-05T23:30:09.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: osdmap e45: 8 total, 7 up, 8 in 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='osd.7 [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:09.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:09.911 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:09.918+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f163c0fd5b0 con 0x7f16240a11a0 2024-04-05T23:30:10.467 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:30:10.184+0000 7f1f50862700 -1 osd.7 0 waiting for initial osdmap 2024-04-05T23:30:10.467 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:30:10.197+0000 7f1f48e50700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:30:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi027 2024-04-05T23:30:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi027 to 5704M 2024-04-05T23:30:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:30:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: osdmap e46: 8 total, 7 up, 8 in 2024-04-05T23:30:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:10.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: osdmap e47: 8 total, 7 up, 8 in 2024-04-05T23:30:10.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:10.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:30:10.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:30:10.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:10.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:10 smithi027 ceph-mon[78597]: from='osd.7 ' entity='osd.7' 2024-04-05T23:30:10.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi027 2024-04-05T23:30:10.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi027 to 5704M 2024-04-05T23:30:10.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:10.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:30:10.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: osdmap e46: 8 total, 7 up, 8 in 2024-04-05T23:30:10.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:10.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:10.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: osdmap e47: 8 total, 7 up, 8 in 2024-04-05T23:30:10.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:10.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:30:10.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:30:10.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:10.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:10 smithi130 ceph-mon[79417]: from='osd.7 ' entity='osd.7' 2024-04-05T23:30:10.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi027 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi027 to 5704M 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: osdmap e46: 8 total, 7 up, 8 in 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: osdmap e47: 8 total, 7 up, 8 in 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:10 smithi016 ceph-mon[38248]: from='osd.7 ' entity='osd.7' 2024-04-05T23:30:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:11 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:30:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:11 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:30:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:11 smithi027 ceph-mon[78597]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:30:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:11 smithi027 ceph-mon[78597]: osd.7 [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] boot 2024-04-05T23:30:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:11 smithi027 ceph-mon[78597]: osdmap e48: 8 total, 8 up, 8 in 2024-04-05T23:30:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:11 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:30:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:11 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:30:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:11 smithi016 ceph-mon[38248]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:30:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:11 smithi016 ceph-mon[38248]: osd.7 [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] boot 2024-04-05T23:30:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:11 smithi016 ceph-mon[38248]: osdmap e48: 8 total, 8 up, 8 in 2024-04-05T23:30:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:11.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:11 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:30:11.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:11 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:30:11.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:11 smithi130 ceph-mon[79417]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:30:11.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:11.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:11 smithi130 ceph-mon[79417]: osd.7 [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] boot 2024-04-05T23:30:11.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:11 smithi130 ceph-mon[79417]: osdmap e48: 8 total, 8 up, 8 in 2024-04-05T23:30:11.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:30:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:12 smithi027 ceph-mon[78597]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:12 smithi027 ceph-mon[78597]: osdmap e49: 8 total, 8 up, 8 in 2024-04-05T23:30:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:12 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/3721594812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "584a112e-7c86-49ea-b1b6-30733d0f1e31"}]: dispatch 2024-04-05T23:30:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:12 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "584a112e-7c86-49ea-b1b6-30733d0f1e31"}]: dispatch 2024-04-05T23:30:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:12 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "584a112e-7c86-49ea-b1b6-30733d0f1e31"}]': finished 2024-04-05T23:30:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:12 smithi027 ceph-mon[78597]: osdmap e50: 9 total, 8 up, 9 in 2024-04-05T23:30:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:12 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:12.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:12 smithi016 ceph-mon[38248]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:12 smithi016 ceph-mon[38248]: osdmap e49: 8 total, 8 up, 8 in 2024-04-05T23:30:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:12 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/3721594812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "584a112e-7c86-49ea-b1b6-30733d0f1e31"}]: dispatch 2024-04-05T23:30:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:12 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "584a112e-7c86-49ea-b1b6-30733d0f1e31"}]: dispatch 2024-04-05T23:30:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:12 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "584a112e-7c86-49ea-b1b6-30733d0f1e31"}]': finished 2024-04-05T23:30:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:12 smithi016 ceph-mon[38248]: osdmap e50: 9 total, 8 up, 9 in 2024-04-05T23:30:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:12 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:12 smithi130 ceph-mon[79417]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-05T23:30:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:12 smithi130 ceph-mon[79417]: osdmap e49: 8 total, 8 up, 8 in 2024-04-05T23:30:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:12 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/3721594812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "584a112e-7c86-49ea-b1b6-30733d0f1e31"}]: dispatch 2024-04-05T23:30:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:12 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "584a112e-7c86-49ea-b1b6-30733d0f1e31"}]: dispatch 2024-04-05T23:30:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:12 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "584a112e-7c86-49ea-b1b6-30733d0f1e31"}]': finished 2024-04-05T23:30:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:12 smithi130 ceph-mon[79417]: osdmap e50: 9 total, 8 up, 9 in 2024-04-05T23:30:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:12 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:13.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:13 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/1240226559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:30:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:13 smithi027 ceph-mon[78597]: osdmap e51: 9 total, 8 up, 9 in 2024-04-05T23:30:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:13 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:13 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/1240226559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:30:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:13 smithi016 ceph-mon[38248]: osdmap e51: 9 total, 8 up, 9 in 2024-04-05T23:30:13.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:13 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:13.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:13 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/1240226559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:30:13.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:13 smithi130 ceph-mon[79417]: osdmap e51: 9 total, 8 up, 9 in 2024-04-05T23:30:13.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:13 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:14.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:14 smithi027 ceph-mon[78597]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:14.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:14 smithi027 ceph-mon[78597]: osdmap e52: 9 total, 8 up, 9 in 2024-04-05T23:30:14.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:14 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:14.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:14 smithi016 ceph-mon[38248]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:14 smithi016 ceph-mon[38248]: osdmap e52: 9 total, 8 up, 9 in 2024-04-05T23:30:14.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:14 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:14.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:14 smithi130 ceph-mon[79417]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:14.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:14 smithi130 ceph-mon[79417]: osdmap e52: 9 total, 8 up, 9 in 2024-04-05T23:30:14.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:14 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:16.662 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:16 smithi130 ceph-mon[79417]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:16.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:16 smithi027 ceph-mon[78597]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:16.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:16 smithi016 ceph-mon[38248]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:18.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:18 smithi130 ceph-mon[79417]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:18.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:18 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-05T23:30:18.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:18 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:18.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:18 smithi027 ceph-mon[78597]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:18.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-05T23:30:18.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:18 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:18.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:18 smithi016 ceph-mon[38248]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:18.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:18 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-05T23:30:18.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:18 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:19.661 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:19 smithi130 ceph-mon[79417]: Deploying daemon osd.8 on smithi130 2024-04-05T23:30:19.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:19 smithi027 ceph-mon[78597]: Deploying daemon osd.8 on smithi130 2024-04-05T23:30:19.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:19 smithi016 ceph-mon[38248]: Deploying daemon osd.8 on smithi130 2024-04-05T23:30:20.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:20 smithi027 ceph-mon[78597]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:20.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:20 smithi016 ceph-mon[38248]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:20.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:20 smithi130 ceph-mon[79417]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:22.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:22 smithi027 ceph-mon[78597]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:22.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:22 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:22.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:22 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:22.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:22 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:22.809 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:22 smithi130 ceph-mon[79417]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:22.809 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:22 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:22.809 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:22 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:22.809 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:22 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:22.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:22 smithi016 ceph-mon[38248]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:22.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:22 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:22.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:22 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:22.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:22 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:23.767 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.775+0000 7f16397fa700 1 -- 172.21.15.130:0/4120642947 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f163c0fd5b0 con 0x7f16240a11a0 2024-04-05T23:30:23.768 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 8 on host 'smithi130' 2024-04-05T23:30:23.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16240a11a0 msgr2=0x7f16240a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:23.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16240a11a0 0x7f16240a3660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f162c005d90 tx=0x7f162c005d00 comp rx=0 tx=0).stop 2024-04-05T23:30:23.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c100f70 msgr2=0x7f163c1972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:23.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c100f70 0x7f163c1972d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f163000b530 tx=0x7f163000b840 comp rx=0 tx=0).stop 2024-04-05T23:30:23.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 shutdown_connections 2024-04-05T23:30:23.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16240a11a0 0x7f16240a3660 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:23.771 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f163c0ffe10 0x7f163c196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:23.771 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f163c101930 0x7f163c19e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:23.771 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 --2- 172.21.15.130:0/4120642947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f163c100f70 0x7f163c1972d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:23.771 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 >> 172.21.15.130:0/4120642947 conn(0x7f163c0fb3b0 msgr2=0x7f163c0fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:30:23.771 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 shutdown_connections 2024-04-05T23:30:23.771 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:30:23.777+0000 7f1642610700 1 -- 172.21.15.130:0/4120642947 wait complete. 2024-04-05T23:30:24.156 DEBUG:teuthology.orchestra.run.smithi130:osd.8> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.8.service 2024-04-05T23:30:24.159 INFO:tasks.cephadm:Deploying osd.9 on smithi016 with /dev/vg_nvme/lv_1... 2024-04-05T23:30:24.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-05T23:30:24.190 INFO:journalctl@ceph.osd.8.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:30:24.361 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:24 smithi130 ceph-mon[79417]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:24.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:24.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:24.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:24.553 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:30:24.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:24 smithi016 ceph-mon[38248]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:24 smithi027 ceph-mon[78597]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:24.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:25.944 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:30:25 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:30:25.626+0000 7f2631a907c0 -1 osd.8 0 log_to_monitors true 2024-04-05T23:30:26.270 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-05T23:30:26.270 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-05T23:30:26.270 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-05T23:30:26.270 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-05T23:30:26.270 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0301174 s, 348 MB/s 2024-04-05T23:30:26.270 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-05T23:30:26.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:26 smithi016 ceph-mon[38248]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:26.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:26 smithi016 ceph-mon[38248]: from='osd.8 [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-05T23:30:26.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:26 smithi016 ceph-mon[38248]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-05T23:30:26.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:26 smithi130 ceph-mon[79417]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:26.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:26 smithi130 ceph-mon[79417]: from='osd.8 [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-05T23:30:26.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:26 smithi130 ceph-mon[79417]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-05T23:30:26.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-04-05T23:30:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:26 smithi027 ceph-mon[78597]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:26 smithi027 ceph-mon[78597]: from='osd.8 [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-05T23:30:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:26 smithi027 ceph-mon[78597]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-05T23:30:27.029 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:30:27.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-05T23:30:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: osdmap e53: 9 total, 8 up, 9 in 2024-04-05T23:30:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='osd.8 [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:30:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:30:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi130 2024-04-05T23:30:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:27.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:27.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:27.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi130 to 5704M 2024-04-05T23:30:27.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:27.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:27.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:27 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.584 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:30:27.336+0000 7f26268f4700 -1 osd.8 0 waiting for initial osdmap 2024-04-05T23:30:27.584 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:30:27.352+0000 7f2622eea700 -1 osd.8 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: osdmap e53: 9 total, 8 up, 9 in 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='osd.8 [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi130 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:27.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi130 to 5704M 2024-04-05T23:30:27.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:27.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:27.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:27 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: osdmap e53: 9 total, 8 up, 9 in 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='osd.8 [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi130 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi130 to 5704M 2024-04-05T23:30:27.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:27.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:27.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:27.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:27 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.207+0000 7fd9c1e00700 1 -- 172.21.15.16:0/2989645986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc073940 msgr2=0x7fd9bc073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.207+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/2989645986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc073940 0x7fd9bc073dc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009990 tx=0x7fd9a4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:30:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.207+0000 7fd9c1e00700 1 -- 172.21.15.16:0/2989645986 shutdown_connections 2024-04-05T23:30:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.207+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/2989645986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9bc108e10 0x7fd9bc10b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.207+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/2989645986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc074f10 0x7fd9bc073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.207+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/2989645986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc073940 0x7fd9bc073dc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.207+0000 7fd9c1e00700 1 -- 172.21.15.16:0/2989645986 >> 172.21.15.16:0/2989645986 conn(0x7fd9bc0fc4e0 msgr2=0x7fd9bc0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:30:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.207+0000 7fd9c1e00700 1 -- 172.21.15.16:0/2989645986 shutdown_connections 2024-04-05T23:30:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.207+0000 7fd9c1e00700 1 -- 172.21.15.16:0/2989645986 wait complete. 2024-04-05T23:30:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.208+0000 7fd9c1e00700 1 Processor -- start 2024-04-05T23:30:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.208+0000 7fd9c1e00700 1 -- start start 2024-04-05T23:30:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.208+0000 7fd9c1e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9bc073940 0x7fd9bc199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.208+0000 7fd9c1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc074f10 0x7fd9bc199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.208+0000 7fd9c1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc108e10 0x7fd9bc1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.208+0000 7fd9c1e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc1a0c60 con 0x7fd9bc074f10 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.208+0000 7fd9c1e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc1a0da0 con 0x7fd9bc108e10 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.208+0000 7fd9c1e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc1a0f30 con 0x7fd9bc073940 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9b3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc074f10 0x7fd9bc199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9b3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc074f10 0x7fd9bc199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46762/0 (socket says 172.21.15.16:46762) 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9b3e00700 1 -- 172.21.15.16:0/453936265 learned_addr learned my addr 172.21.15.16:0/453936265 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9baa00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9bc073940 0x7fd9bc199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9bb400700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc108e10 0x7fd9bc1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9b3e00700 1 -- 172.21.15.16:0/453936265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9bc073940 msgr2=0x7fd9bc199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9b3e00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9bc073940 0x7fd9bc199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9b3e00700 1 -- 172.21.15.16:0/453936265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc108e10 msgr2=0x7fd9bc1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9b3e00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc108e10 0x7fd9bc1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9b3e00700 1 -- 172.21.15.16:0/453936265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a40095f0 con 0x7fd9bc074f10 2024-04-05T23:30:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9baa00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9bc073940 0x7fd9bc199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:30:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.209+0000 7fd9b3e00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc074f10 0x7fd9bc199660 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40095c0 tx=0x7fd9a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:30:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.210+0000 7fd9c0a00700 1 -- 172.21.15.16:0/453936265 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a401d070 con 0x7fd9bc074f10 2024-04-05T23:30:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.210+0000 7fd9c0a00700 1 -- 172.21.15.16:0/453936265 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9a4021410 con 0x7fd9bc074f10 2024-04-05T23:30:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.210+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc1a11b0 con 0x7fd9bc074f10 2024-04-05T23:30:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.210+0000 7fd9c0a00700 1 -- 172.21.15.16:0/453936265 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4017650 con 0x7fd9bc074f10 2024-04-05T23:30:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.210+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc1a1620 con 0x7fd9bc074f10 2024-04-05T23:30:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.211+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc107bd0 con 0x7fd9bc074f10 2024-04-05T23:30:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.213+0000 7fd9c0a00700 1 -- 172.21.15.16:0/453936265 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fd9a4021810 con 0x7fd9bc074f10 2024-04-05T23:30:28.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.214+0000 7fd9c0a00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd9a00a11a0 0x7fd9a00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:28.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.214+0000 7fd9c0a00700 1 -- 172.21.15.16:0/453936265 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5549+0+0 (secure 0 0 0) 0x7fd9a40cd250 con 0x7fd9bc074f10 2024-04-05T23:30:28.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.214+0000 7fd9baa00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd9a00a11a0 0x7fd9a00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:28.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.215+0000 7fd9baa00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd9a00a11a0 0x7fd9a00a3660 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0752f0 tx=0x7fd9ac0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:30:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.218+0000 7fd9c0a00700 1 -- 172.21.15.16:0/453936265 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd9a40974a0 con 0x7fd9bc074f10 2024-04-05T23:30:28.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:28.447+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9bc061320 con 0x7fd9a00a11a0 2024-04-05T23:30:28.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:28 smithi016 ceph-mon[38248]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:28 smithi016 ceph-mon[38248]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:30:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:28 smithi016 ceph-mon[38248]: osdmap e54: 9 total, 8 up, 9 in 2024-04-05T23:30:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:28.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:28 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:28.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:28 smithi130 ceph-mon[79417]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:28.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:28 smithi130 ceph-mon[79417]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:30:28.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:28 smithi130 ceph-mon[79417]: osdmap e54: 9 total, 8 up, 9 in 2024-04-05T23:30:28.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:28.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:28 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:28.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:28 smithi027 ceph-mon[78597]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:28.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:28 smithi027 ceph-mon[78597]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:30:28.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:28 smithi027 ceph-mon[78597]: osdmap e54: 9 total, 8 up, 9 in 2024-04-05T23:30:28.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:28.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:28 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:29.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:29 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:30:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:29 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:30:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:29 smithi016 ceph-mon[38248]: osd.8 [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] boot 2024-04-05T23:30:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:29 smithi016 ceph-mon[38248]: osdmap e55: 9 total, 9 up, 9 in 2024-04-05T23:30:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:30:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:30:29.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:29 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:29 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:30:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:29 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:30:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:29 smithi130 ceph-mon[79417]: osd.8 [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] boot 2024-04-05T23:30:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:29 smithi130 ceph-mon[79417]: osdmap e55: 9 total, 9 up, 9 in 2024-04-05T23:30:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:30:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:30:29.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:29 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:29.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:29 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:30:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:29 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:30:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:29 smithi027 ceph-mon[78597]: osd.8 [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] boot 2024-04-05T23:30:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:29 smithi027 ceph-mon[78597]: osdmap e55: 9 total, 9 up, 9 in 2024-04-05T23:30:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:30:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:30:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:30:29.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:29 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:30.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:30 smithi027 ceph-mon[78597]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:30:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:30 smithi027 ceph-mon[78597]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:30.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:30 smithi027 ceph-mon[78597]: osdmap e56: 9 total, 9 up, 9 in 2024-04-05T23:30:30.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:30 smithi016 ceph-mon[38248]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:30:30.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:30 smithi016 ceph-mon[38248]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:30.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:30 smithi016 ceph-mon[38248]: osdmap e56: 9 total, 9 up, 9 in 2024-04-05T23:30:30.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:30 smithi130 ceph-mon[79417]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:30:30.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:30 smithi130 ceph-mon[79417]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-05T23:30:30.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:30 smithi130 ceph-mon[79417]: osdmap e56: 9 total, 9 up, 9 in 2024-04-05T23:30:31.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:31 smithi027 ceph-mon[78597]: osdmap e57: 9 total, 9 up, 9 in 2024-04-05T23:30:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:31 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3276382461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14bd3f6d-201c-45c2-b82a-aa358d1a0f28"}]: dispatch 2024-04-05T23:30:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:31 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14bd3f6d-201c-45c2-b82a-aa358d1a0f28"}]: dispatch 2024-04-05T23:30:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:31 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14bd3f6d-201c-45c2-b82a-aa358d1a0f28"}]': finished 2024-04-05T23:30:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:31 smithi027 ceph-mon[78597]: osdmap e58: 10 total, 9 up, 10 in 2024-04-05T23:30:31.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:31 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:31.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:31 smithi016 ceph-mon[38248]: osdmap e57: 9 total, 9 up, 9 in 2024-04-05T23:30:31.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:31 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3276382461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14bd3f6d-201c-45c2-b82a-aa358d1a0f28"}]: dispatch 2024-04-05T23:30:31.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:31 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14bd3f6d-201c-45c2-b82a-aa358d1a0f28"}]: dispatch 2024-04-05T23:30:31.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:31 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14bd3f6d-201c-45c2-b82a-aa358d1a0f28"}]': finished 2024-04-05T23:30:31.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:31 smithi016 ceph-mon[38248]: osdmap e58: 10 total, 9 up, 10 in 2024-04-05T23:30:31.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:31 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:31.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:31 smithi130 ceph-mon[79417]: osdmap e57: 9 total, 9 up, 9 in 2024-04-05T23:30:31.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:31 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3276382461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14bd3f6d-201c-45c2-b82a-aa358d1a0f28"}]: dispatch 2024-04-05T23:30:31.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:31 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14bd3f6d-201c-45c2-b82a-aa358d1a0f28"}]: dispatch 2024-04-05T23:30:31.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:31 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14bd3f6d-201c-45c2-b82a-aa358d1a0f28"}]': finished 2024-04-05T23:30:31.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:31 smithi130 ceph-mon[79417]: osdmap e58: 10 total, 9 up, 10 in 2024-04-05T23:30:31.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:31 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:32.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:32 smithi027 ceph-mon[78597]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:32.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:32 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1419300321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:30:32.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:32 smithi027 ceph-mon[78597]: osdmap e59: 10 total, 9 up, 10 in 2024-04-05T23:30:32.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:32 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:32.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:32 smithi016 ceph-mon[38248]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:32.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:32 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1419300321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:30:32.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:32 smithi016 ceph-mon[38248]: osdmap e59: 10 total, 9 up, 10 in 2024-04-05T23:30:32.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:32 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:32.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:32 smithi130 ceph-mon[79417]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:32.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:32 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1419300321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:30:32.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:32 smithi130 ceph-mon[79417]: osdmap e59: 10 total, 9 up, 10 in 2024-04-05T23:30:32.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:32 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:34.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:34 smithi027 ceph-mon[78597]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-05T23:30:34.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:34 smithi016 ceph-mon[38248]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-05T23:30:34.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:34 smithi130 ceph-mon[79417]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-05T23:30:36.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:36 smithi027 ceph-mon[78597]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-05T23:30:36.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:36 smithi016 ceph-mon[38248]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-05T23:30:36.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:36 smithi130 ceph-mon[79417]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-05T23:30:37.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-05T23:30:37.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:37.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-05T23:30:37.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:37.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-05T23:30:37.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:38.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:38 smithi016 ceph-mon[38248]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-05T23:30:38.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:38 smithi016 ceph-mon[38248]: Deploying daemon osd.9 on smithi016 2024-04-05T23:30:38.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:38 smithi027 ceph-mon[78597]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-05T23:30:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:38 smithi027 ceph-mon[78597]: Deploying daemon osd.9 on smithi016 2024-04-05T23:30:38.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:38 smithi130 ceph-mon[79417]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-05T23:30:38.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:38 smithi130 ceph-mon[79417]: Deploying daemon osd.9 on smithi016 2024-04-05T23:30:40.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:40 smithi016 ceph-mon[38248]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-05T23:30:40.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:40 smithi027 ceph-mon[78597]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-05T23:30:40.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:40 smithi130 ceph-mon[79417]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-05T23:30:42.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:42 smithi027 ceph-mon[78597]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-05T23:30:42.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:42 smithi016 ceph-mon[38248]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-05T23:30:42.838 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:42 smithi130 ceph-mon[79417]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-05T23:30:43.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:43 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:43.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:43 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:43.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:43 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:43.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:43 smithi016 ceph-mon[38248]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:43.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:43 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:43.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:43 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:43.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:43 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:43.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:43 smithi027 ceph-mon[78597]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:43 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:43 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:43 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:43 smithi130 ceph-mon[79417]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:45.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.357+0000 7fd9c0a00700 1 -- 172.21.15.16:0/453936265 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd9bc061320 con 0x7fd9a00a11a0 2024-04-05T23:30:45.350 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 9 on host 'smithi016' 2024-04-05T23:30:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd9a00a11a0 msgr2=0x7fd9a00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd9a00a11a0 0x7fd9a00a3660 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0752f0 tx=0x7fd9ac0073d0 comp rx=0 tx=0).stop 2024-04-05T23:30:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc074f10 msgr2=0x7fd9bc199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc074f10 0x7fd9bc199660 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40095c0 tx=0x7fd9a4004730 comp rx=0 tx=0).stop 2024-04-05T23:30:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 shutdown_connections 2024-04-05T23:30:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd9a00a11a0 0x7fd9a00a3660 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9bc108e10 0x7fd9bc1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9bc073940 0x7fd9bc199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 --2- 172.21.15.16:0/453936265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9bc074f10 0x7fd9bc199660 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.359+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 >> 172.21.15.16:0/453936265 conn(0x7fd9bc0fc4e0 msgr2=0x7fd9bc104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:30:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.360+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 shutdown_connections 2024-04-05T23:30:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:30:45.360+0000 7fd9c1e00700 1 -- 172.21.15.16:0/453936265 wait complete. 2024-04-05T23:30:45.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:45.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:45.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:45.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:45.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:45.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:45.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:45.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:45.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:45.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:45.791 DEBUG:teuthology.orchestra.run.smithi016:osd.9> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.9.service 2024-04-05T23:30:45.795 INFO:tasks.cephadm:Deploying osd.10 on smithi027 with /dev/vg_nvme/lv_1... 2024-04-05T23:30:45.795 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-05T23:30:45.874 INFO:journalctl@ceph.osd.9.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:30:46.036 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:30:46.454 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:46 smithi027 ceph-mon[78597]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:46.454 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:46 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:46.455 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:46 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:46.455 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:46 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:46.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:46 smithi016 ceph-mon[38248]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:46.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:46 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:46.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:46 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:46.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:46 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:46.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:46 smithi130 ceph-mon[79417]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:46.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:46 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:46.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:46 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:30:46.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:46 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:46.832 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:30:46 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:30:46.571+0000 7ff42084a7c0 -1 osd.9 0 log_to_monitors true 2024-04-05T23:30:47.384 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-05T23:30:47.384 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-05T23:30:47.384 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-05T23:30:47.384 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-05T23:30:47.384 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110956 s, 945 MB/s 2024-04-05T23:30:47.384 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 2024-04-05T23:30:47.384 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-05T23:30:47.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:47 smithi016 ceph-mon[38248]: from='osd.9 [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-05T23:30:47.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:47 smithi016 ceph-mon[38248]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-05T23:30:47.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:47 smithi130 ceph-mon[79417]: from='osd.9 [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-05T23:30:47.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:47 smithi130 ceph-mon[79417]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-05T23:30:47.674 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:47 smithi027 ceph-mon[78597]: from='osd.9 [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-05T23:30:47.674 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:47 smithi027 ceph-mon[78597]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-05T23:30:47.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi027:vg_nvme/lv_1 2024-04-05T23:30:47.935 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:30:48.463 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:48 smithi027 ceph-mon[78597]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:48.463 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:48 smithi027 ceph-mon[78597]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-05T23:30:48.463 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:48 smithi027 ceph-mon[78597]: osdmap e60: 10 total, 9 up, 10 in 2024-04-05T23:30:48.463 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:48.463 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:48 smithi027 ceph-mon[78597]: from='osd.9 [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:30:48.463 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:48 smithi027 ceph-mon[78597]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:30:48.582 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:30:48 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:30:48.354+0000 7ff414400700 -1 osd.9 0 waiting for initial osdmap 2024-04-05T23:30:48.583 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:30:48 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:30:48.375+0000 7ff40fc00700 -1 osd.9 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:30:48.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:48 smithi016 ceph-mon[38248]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:48.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:48 smithi016 ceph-mon[38248]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-05T23:30:48.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:48 smithi016 ceph-mon[38248]: osdmap e60: 10 total, 9 up, 10 in 2024-04-05T23:30:48.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:48.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:48 smithi016 ceph-mon[38248]: from='osd.9 [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:30:48.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:48 smithi016 ceph-mon[38248]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:30:48.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:48 smithi130 ceph-mon[79417]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:48.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:48 smithi130 ceph-mon[79417]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-05T23:30:48.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:48 smithi130 ceph-mon[79417]: osdmap e60: 10 total, 9 up, 10 in 2024-04-05T23:30:48.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:48.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:48 smithi130 ceph-mon[79417]: from='osd.9 [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:30:48.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:48 smithi130 ceph-mon[79417]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-05T23:30:49.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.035+0000 7f306e9e7700 1 -- 172.21.15.27:0/2943500006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3068100ef0 msgr2=0x7f3068101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:49.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.035+0000 7f306e9e7700 1 --2- 172.21.15.27:0/2943500006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3068100ef0 0x7f3068101330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3058009990 tx=0x7f3058009ca0 comp rx=0 tx=0).stop 2024-04-05T23:30:49.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.035+0000 7f306e9e7700 1 -- 172.21.15.27:0/2943500006 shutdown_connections 2024-04-05T23:30:49.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.035+0000 7f306e9e7700 1 --2- 172.21.15.27:0/2943500006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068102a90 0x7f30681093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:49.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.035+0000 7f306e9e7700 1 --2- 172.21.15.27:0/2943500006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3068100ef0 0x7f3068101330 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:49.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.035+0000 7f306e9e7700 1 --2- 172.21.15.27:0/2943500006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30681020d0 0x7f3068102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:49.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.035+0000 7f306e9e7700 1 -- 172.21.15.27:0/2943500006 >> 172.21.15.27:0/2943500006 conn(0x7f30680fc4e0 msgr2=0x7f30680fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:30:49.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.035+0000 7f306e9e7700 1 -- 172.21.15.27:0/2943500006 shutdown_connections 2024-04-05T23:30:49.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.035+0000 7f306e9e7700 1 -- 172.21.15.27:0/2943500006 wait complete. 2024-04-05T23:30:49.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.036+0000 7f306e9e7700 1 Processor -- start 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.036+0000 7f306e9e7700 1 -- start start 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.036+0000 7f306e9e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068100ef0 0x7f3068196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.036+0000 7f306e9e7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30681020d0 0x7f3068197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.036+0000 7f306e9e7700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3068102a90 0x7f306819e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.036+0000 7f306e9e7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306819e990 con 0x7f3068100ef0 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.036+0000 7f306e9e7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306819ead0 con 0x7f30681020d0 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.036+0000 7f306e9e7700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306819ec60 con 0x7f3068102a90 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30681020d0 0x7f3068197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30681020d0 0x7f3068197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45604/0 (socket says 172.21.15.27:45604) 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30677fe700 1 -- 172.21.15.27:0/1843956208 learned_addr learned my addr 172.21.15.27:0/1843956208 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f306cf84700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3068102a90 0x7f306819e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f3067fff700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068100ef0 0x7f3068196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30677fe700 1 -- 172.21.15.27:0/1843956208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3068102a90 msgr2=0x7f306819e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:49.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30677fe700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3068102a90 0x7f306819e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:49.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30677fe700 1 -- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068100ef0 msgr2=0x7f3068196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:30:49.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30677fe700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068100ef0 0x7f3068196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:30:49.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30677fe700 1 -- 172.21.15.27:0/1843956208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30580095f0 con 0x7f30681020d0 2024-04-05T23:30:49.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f3067fff700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068100ef0 0x7f3068196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:30:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30677fe700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30681020d0 0x7f3068197390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f305000e9a0 tx=0x7f305000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:30:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30657fa700 1 -- 172.21.15.27:0/1843956208 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050009790 con 0x7f30681020d0 2024-04-05T23:30:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306819ef40 con 0x7f30681020d0 2024-04-05T23:30:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30657fa700 1 -- 172.21.15.27:0/1843956208 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f305000cbd0 con 0x7f30681020d0 2024-04-05T23:30:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f30657fa700 1 -- 172.21.15.27:0/1843956208 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050018680 con 0x7f30681020d0 2024-04-05T23:30:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.037+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306819f400 con 0x7f30681020d0 2024-04-05T23:30:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.039+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30681083e0 con 0x7f30681020d0 2024-04-05T23:30:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.039+0000 7f30657fa700 1 -- 172.21.15.27:0/1843956208 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f305000cd40 con 0x7f30681020d0 2024-04-05T23:30:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.040+0000 7f30657fa700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f30540a1120 0x7f30540a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:30:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.040+0000 7f30657fa700 1 -- 172.21.15.27:0/1843956208 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5981+0+0 (secure 0 0 0) 0x7f3050014070 con 0x7f30681020d0 2024-04-05T23:30:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.040+0000 7f3067fff700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f30540a1120 0x7f30540a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:30:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.041+0000 7f3067fff700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f30540a1120 0x7f30540a35e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f305800b7b0 tx=0x7f3058005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:30:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.044+0000 7f30657fa700 1 -- 172.21.15.27:0/1843956208 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3050052ab0 con 0x7f30681020d0 2024-04-05T23:30:49.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:30:49.258+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3068061320 con 0x7f30540a1120 2024-04-05T23:30:49.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:30:49.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: osdmap e61: 10 total, 9 up, 10 in 2024-04-05T23:30:49.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:49.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:49.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.628 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.629 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.629 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.629 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:49.629 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:49.629 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.629 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:30:49.629 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:30:49.629 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:49 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:30:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: osdmap e61: 10 total, 9 up, 10 in 2024-04-05T23:30:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:49.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:49.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:30:49.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:30:49.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:49 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:49.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: osdmap e61: 10 total, 9 up, 10 in 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:30:49.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:30:49.836 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:30:49.836 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:49 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:50.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:50 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:30:50.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:50 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:30:50.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:50 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi016 2024-04-05T23:30:50.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:50 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi016 to 4318M 2024-04-05T23:30:50.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:50 smithi027 ceph-mon[78597]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:30:50.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:50 smithi027 ceph-mon[78597]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:50.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:50 smithi027 ceph-mon[78597]: osd.9 [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] boot 2024-04-05T23:30:50.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:50 smithi027 ceph-mon[78597]: osdmap e62: 10 total, 10 up, 10 in 2024-04-05T23:30:50.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:50.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:50 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:30:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:50 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:30:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:50 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi016 2024-04-05T23:30:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:50 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi016 to 4318M 2024-04-05T23:30:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:50 smithi016 ceph-mon[38248]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:30:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:50 smithi016 ceph-mon[38248]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:50 smithi016 ceph-mon[38248]: osd.9 [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] boot 2024-04-05T23:30:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:50 smithi016 ceph-mon[38248]: osdmap e62: 10 total, 10 up, 10 in 2024-04-05T23:30:50.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:50 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:30:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:50 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:30:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:50 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi016 2024-04-05T23:30:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:50 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi016 to 4318M 2024-04-05T23:30:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:50 smithi130 ceph-mon[79417]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:30:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:50 smithi130 ceph-mon[79417]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-05T23:30:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:50 smithi130 ceph-mon[79417]: osd.9 [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] boot 2024-04-05T23:30:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:50 smithi130 ceph-mon[79417]: osdmap e62: 10 total, 10 up, 10 in 2024-04-05T23:30:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:30:51.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:51 smithi027 ceph-mon[78597]: osdmap e63: 10 total, 10 up, 10 in 2024-04-05T23:30:51.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:51 smithi027 ceph-mon[78597]: from='client.? 172.21.15.27:0/1149054059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366e5c3c-48b4-4718-9128-021c930445ee"}]: dispatch 2024-04-05T23:30:51.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:51 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366e5c3c-48b4-4718-9128-021c930445ee"}]: dispatch 2024-04-05T23:30:51.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:51 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "366e5c3c-48b4-4718-9128-021c930445ee"}]': finished 2024-04-05T23:30:51.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:51 smithi027 ceph-mon[78597]: osdmap e64: 11 total, 10 up, 11 in 2024-04-05T23:30:51.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:30:51.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:51 smithi016 ceph-mon[38248]: osdmap e63: 10 total, 10 up, 10 in 2024-04-05T23:30:51.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:51 smithi016 ceph-mon[38248]: from='client.? 172.21.15.27:0/1149054059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366e5c3c-48b4-4718-9128-021c930445ee"}]: dispatch 2024-04-05T23:30:51.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:51 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366e5c3c-48b4-4718-9128-021c930445ee"}]: dispatch 2024-04-05T23:30:51.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:51 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "366e5c3c-48b4-4718-9128-021c930445ee"}]': finished 2024-04-05T23:30:51.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:51 smithi016 ceph-mon[38248]: osdmap e64: 11 total, 10 up, 11 in 2024-04-05T23:30:51.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:30:51.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:51 smithi130 ceph-mon[79417]: osdmap e63: 10 total, 10 up, 10 in 2024-04-05T23:30:51.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:51 smithi130 ceph-mon[79417]: from='client.? 172.21.15.27:0/1149054059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366e5c3c-48b4-4718-9128-021c930445ee"}]: dispatch 2024-04-05T23:30:51.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:51 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366e5c3c-48b4-4718-9128-021c930445ee"}]: dispatch 2024-04-05T23:30:51.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:51 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "366e5c3c-48b4-4718-9128-021c930445ee"}]': finished 2024-04-05T23:30:51.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:51 smithi130 ceph-mon[79417]: osdmap e64: 11 total, 10 up, 11 in 2024-04-05T23:30:51.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:30:52.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:52 smithi027 ceph-mon[78597]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:52.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:52 smithi027 ceph-mon[78597]: from='client.? 172.21.15.27:0/2900435581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:30:52.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:52 smithi016 ceph-mon[38248]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:52.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:52 smithi016 ceph-mon[38248]: from='client.? 172.21.15.27:0/2900435581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:30:52.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:52 smithi130 ceph-mon[79417]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:52.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:52 smithi130 ceph-mon[79417]: from='client.? 172.21.15.27:0/2900435581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:30:54.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:54 smithi027 ceph-mon[78597]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:54.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:54 smithi016 ceph-mon[38248]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:54.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:54 smithi130 ceph-mon[79417]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:56.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:56 smithi027 ceph-mon[78597]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:56 smithi016 ceph-mon[38248]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:56.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:56 smithi130 ceph-mon[79417]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:57.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:57 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-05T23:30:57.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:57 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:57.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:57 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-05T23:30:57.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:57 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:57.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:57 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-05T23:30:57.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:57 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:30:58.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:58 smithi027 ceph-mon[78597]: Deploying daemon osd.10 on smithi027 2024-04-05T23:30:58.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:30:58 smithi027 ceph-mon[78597]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:58.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:58 smithi016 ceph-mon[38248]: Deploying daemon osd.10 on smithi027 2024-04-05T23:30:58.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:30:58 smithi016 ceph-mon[38248]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:30:58.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:58 smithi130 ceph-mon[79417]: Deploying daemon osd.10 on smithi027 2024-04-05T23:30:58.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:30:58 smithi130 ceph-mon[79417]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:00.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:00 smithi027 ceph-mon[78597]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:00.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:00 smithi016 ceph-mon[38248]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:00.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:00 smithi130 ceph-mon[79417]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:02.194 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:02.194 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:02.195 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:02.195 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:02 smithi027 ceph-mon[78597]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:02.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:02.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:02.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:02.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:02 smithi016 ceph-mon[38248]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:02.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:02.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:02.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:02.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:02 smithi130 ceph-mon[79417]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:03.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.368+0000 7f30657fa700 1 -- 172.21.15.27:0/1843956208 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f3068061320 con 0x7f30540a1120 2024-04-05T23:31:03.361 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 10 on host 'smithi027' 2024-04-05T23:31:03.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f30540a1120 msgr2=0x7f30540a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:03.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f30540a1120 0x7f30540a35e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f305800b7b0 tx=0x7f3058005fb0 comp rx=0 tx=0).stop 2024-04-05T23:31:03.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30681020d0 msgr2=0x7f3068197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:03.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30681020d0 0x7f3068197390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f305000e9a0 tx=0x7f305000ecb0 comp rx=0 tx=0).stop 2024-04-05T23:31:03.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 shutdown_connections 2024-04-05T23:31:03.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f30540a1120 0x7f30540a35e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:03.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068100ef0 0x7f3068196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:03.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3068102a90 0x7f306819e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:03.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 --2- 172.21.15.27:0/1843956208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30681020d0 0x7f3068197390 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:03.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 >> 172.21.15.27:0/1843956208 conn(0x7f30680fc4e0 msgr2=0x7f3068105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:03.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 shutdown_connections 2024-04-05T23:31:03.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-05T23:31:03.370+0000 7f306e9e7700 1 -- 172.21.15.27:0/1843956208 wait complete. 2024-04-05T23:31:03.737 DEBUG:teuthology.orchestra.run.smithi027:osd.10> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.10.service 2024-04-05T23:31:03.740 INFO:tasks.cephadm:Deploying osd.11 on smithi130 with /dev/vg_nvme/lv_1... 2024-04-05T23:31:03.740 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-05T23:31:03.774 INFO:journalctl@ceph.osd.10.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:31:03.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:03 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:03.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:03 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:03.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:03 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:03.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:03 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:03.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:03 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:03.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:03 smithi027 ceph-mon[78597]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:03.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:03 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:03.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:03 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:03.969 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:03 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:03.969 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:31:04.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:04.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:04.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:03 smithi016 ceph-mon[38248]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:04.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:04.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:03 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:03 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:03 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:04.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:03 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:04.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:03 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:03 smithi130 ceph-mon[79417]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:04.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:03 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:03 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:04.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:03 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:05.315 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-05T23:31:05.315 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-05T23:31:05.315 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-05T23:31:05.315 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-05T23:31:05.316 INFO:teuthology.orchestra.run.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118113 s, 888 MB/s 2024-04-05T23:31:05.316 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-05T23:31:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:05 smithi027 ceph-mon[78597]: from='osd.10 [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-05T23:31:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:05 smithi027 ceph-mon[78597]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-05T23:31:05.467 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:31:05 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:31:05.072+0000 7f578239f7c0 -1 osd.10 0 log_to_monitors true 2024-04-05T23:31:05.499 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:05 smithi130 ceph-mon[79417]: from='osd.10 [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-05T23:31:05.500 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:05 smithi130 ceph-mon[79417]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-05T23:31:05.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:05 smithi016 ceph-mon[38248]: from='osd.10 [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-05T23:31:05.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:05 smithi016 ceph-mon[38248]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-05T23:31:05.653 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi130:vg_nvme/lv_1 2024-04-05T23:31:05.872 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:31:06.131 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:31:06.137+0000 7f5777203700 -1 osd.10 0 waiting for initial osdmap 2024-04-05T23:31:06.299 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-05T23:31:06.299 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: osdmap e65: 11 total, 10 up, 11 in 2024-04-05T23:31:06.299 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:06.299 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='osd.10 [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:31:06.299 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: osdmap e66: 11 total, 10 up, 11 in 2024-04-05T23:31:06.300 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:06.467 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:31:06.154+0000 7f57737f9700 -1 osd.10 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:31:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-05T23:31:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: osdmap e65: 11 total, 10 up, 11 in 2024-04-05T23:31:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='osd.10 [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:31:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:31:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: osdmap e66: 11 total, 10 up, 11 in 2024-04-05T23:31:06.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-05T23:31:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: osdmap e65: 11 total, 10 up, 11 in 2024-04-05T23:31:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='osd.10 [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:31:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-05T23:31:06.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-05T23:31:06.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:06.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:06.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:06.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:06.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-05T23:31:06.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: osdmap e66: 11 total, 10 up, 11 in 2024-04-05T23:31:06.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 -- 172.21.15.130:0/1594463455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd864100fa0 msgr2=0x7fd864101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 --2- 172.21.15.130:0/1594463455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd864100fa0 0x7fd864101420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd854009990 tx=0x7fd854009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 -- 172.21.15.130:0/1594463455 shutdown_connections 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 --2- 172.21.15.130:0/1594463455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8640ffe40 0x7fd864100260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 --2- 172.21.15.130:0/1594463455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd864101960 0x7fd864108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 --2- 172.21.15.130:0/1594463455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd864100fa0 0x7fd864101420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 -- 172.21.15.130:0/1594463455 >> 172.21.15.130:0/1594463455 conn(0x7fd8640fb400 msgr2=0x7fd8640fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 -- 172.21.15.130:0/1594463455 shutdown_connections 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 -- 172.21.15.130:0/1594463455 wait complete. 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 Processor -- start 2024-04-05T23:31:06.828 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.835+0000 7fd86a49b700 1 -- start start 2024-04-05T23:31:06.830 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd86a49b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8640ffe40 0x7fd8641990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:06.830 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd86a49b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd864100fa0 0x7fd864199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:06.830 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd86a49b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864101960 0x7fd8641a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:06.830 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd86a49b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8641a0c20 con 0x7fd864101960 2024-04-05T23:31:06.830 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd86a49b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8641a0d60 con 0x7fd864100fa0 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd86a49b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8641a0ef0 con 0x7fd8640ffe40 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd863fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8640ffe40 0x7fd8641990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd863fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8640ffe40 0x7fd8641990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:37050/0 (socket says 172.21.15.130:37050) 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd863fff700 1 -- 172.21.15.130:0/2383382439 learned_addr learned my addr 172.21.15.130:0/2383382439 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd8637fe700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd864100fa0 0x7fd864199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd868a38700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864101960 0x7fd8641a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd863fff700 1 -- 172.21.15.130:0/2383382439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd864100fa0 msgr2=0x7fd864199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd863fff700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd864100fa0 0x7fd864199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd863fff700 1 -- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864101960 msgr2=0x7fd8641a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd863fff700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864101960 0x7fd8641a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd863fff700 1 -- 172.21.15.130:0/2383382439 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8540095f0 con 0x7fd8640ffe40 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd868a38700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864101960 0x7fd8641a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd863fff700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8640ffe40 0x7fd8641990e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd85800b560 tx=0x7fd85800b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:06.831 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd8617fa700 1 -- 172.21.15.130:0/2383382439 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858011830 con 0x7fd8640ffe40 2024-04-05T23:31:06.832 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd8617fa700 1 -- 172.21.15.130:0/2383382439 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd858011e50 con 0x7fd8640ffe40 2024-04-05T23:31:06.832 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8641a11d0 con 0x7fd8640ffe40 2024-04-05T23:31:06.832 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.836+0000 7fd8617fa700 1 -- 172.21.15.130:0/2383382439 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85800f2e0 con 0x7fd8640ffe40 2024-04-05T23:31:06.832 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.837+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8641a16d0 con 0x7fd8640ffe40 2024-04-05T23:31:06.832 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.838+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86404ebb0 con 0x7fd8640ffe40 2024-04-05T23:31:06.832 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.839+0000 7fd8617fa700 1 -- 172.21.15.130:0/2383382439 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fd858011990 con 0x7fd8640ffe40 2024-04-05T23:31:06.833 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.839+0000 7fd8617fa700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd84c0a1120 0x7fd84c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:06.833 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.839+0000 7fd8617fa700 1 -- 172.21.15.130:0/2383382439 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fd8580cc200 con 0x7fd8640ffe40 2024-04-05T23:31:06.833 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.840+0000 7fd8637fe700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd84c0a1120 0x7fd84c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:06.833 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.840+0000 7fd8637fe700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd84c0a1120 0x7fd84c0a35e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd854006010 tx=0x7fd8540037b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:06.835 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:06.843+0000 7fd8617fa700 1 -- 172.21.15.130:0/2383382439 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd858096100 con 0x7fd8640ffe40 2024-04-05T23:31:07.036 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:07.044+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8640fd620 con 0x7fd84c0a1120 2024-04-05T23:31:07.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:07 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi027 2024-04-05T23:31:07.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:07 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi027 to 4278M 2024-04-05T23:31:07.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:07.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:31:07.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:31:07.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:07.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:07 smithi130 ceph-mon[79417]: osd.10 [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] boot 2024-04-05T23:31:07.398 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:07 smithi130 ceph-mon[79417]: osdmap e67: 11 total, 11 up, 11 in 2024-04-05T23:31:07.398 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:07 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:07.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:07 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi027 2024-04-05T23:31:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:07 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi027 to 4278M 2024-04-05T23:31:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:31:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:31:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:07 smithi027 ceph-mon[78597]: osd.10 [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] boot 2024-04-05T23:31:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:07 smithi027 ceph-mon[78597]: osdmap e67: 11 total, 11 up, 11 in 2024-04-05T23:31:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:07 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:07.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:07 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi027 2024-04-05T23:31:07.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:07 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi027 to 4278M 2024-04-05T23:31:07.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:07.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T23:31:07.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T23:31:07.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:07.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:07 smithi016 ceph-mon[38248]: osd.10 [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] boot 2024-04-05T23:31:07.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:07 smithi016 ceph-mon[38248]: osdmap e67: 11 total, 11 up, 11 in 2024-04-05T23:31:07.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:07 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:31:08.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:08 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:31:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:08 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:31:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:08 smithi027 ceph-mon[78597]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:31:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:08 smithi027 ceph-mon[78597]: pgmap v192: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:08 smithi027 ceph-mon[78597]: osdmap e68: 11 total, 11 up, 11 in 2024-04-05T23:31:08.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:08 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:31:08.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:08 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:31:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:08 smithi016 ceph-mon[38248]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:31:08.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:08 smithi016 ceph-mon[38248]: pgmap v192: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:08.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:08 smithi016 ceph-mon[38248]: osdmap e68: 11 total, 11 up, 11 in 2024-04-05T23:31:08.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:08 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:31:08.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:08 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:31:08.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:08 smithi130 ceph-mon[79417]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:31:08.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:08 smithi130 ceph-mon[79417]: pgmap v192: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:08.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:08 smithi130 ceph-mon[79417]: osdmap e68: 11 total, 11 up, 11 in 2024-04-05T23:31:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:09 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/2744648332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f8660f8-f57e-4f24-9b2e-bab7a440fc2b"}]: dispatch 2024-04-05T23:31:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:09 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f8660f8-f57e-4f24-9b2e-bab7a440fc2b"}]: dispatch 2024-04-05T23:31:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:09 smithi027 ceph-mon[78597]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f8660f8-f57e-4f24-9b2e-bab7a440fc2b"}]': finished 2024-04-05T23:31:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:09 smithi027 ceph-mon[78597]: osdmap e69: 12 total, 11 up, 12 in 2024-04-05T23:31:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:09 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:09.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:09 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/2744648332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f8660f8-f57e-4f24-9b2e-bab7a440fc2b"}]: dispatch 2024-04-05T23:31:09.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:09 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f8660f8-f57e-4f24-9b2e-bab7a440fc2b"}]: dispatch 2024-04-05T23:31:09.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:09 smithi016 ceph-mon[38248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f8660f8-f57e-4f24-9b2e-bab7a440fc2b"}]': finished 2024-04-05T23:31:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:09 smithi016 ceph-mon[38248]: osdmap e69: 12 total, 11 up, 12 in 2024-04-05T23:31:09.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:09 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:09.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:09 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/2744648332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f8660f8-f57e-4f24-9b2e-bab7a440fc2b"}]: dispatch 2024-04-05T23:31:09.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:09 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f8660f8-f57e-4f24-9b2e-bab7a440fc2b"}]: dispatch 2024-04-05T23:31:09.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:09 smithi130 ceph-mon[79417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f8660f8-f57e-4f24-9b2e-bab7a440fc2b"}]': finished 2024-04-05T23:31:09.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:09 smithi130 ceph-mon[79417]: osdmap e69: 12 total, 11 up, 12 in 2024-04-05T23:31:09.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:09 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:10.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:10 smithi027 ceph-mon[78597]: pgmap v195: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:10.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:10 smithi027 ceph-mon[78597]: osdmap e70: 12 total, 11 up, 12 in 2024-04-05T23:31:10.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:10 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:10.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:10 smithi027 ceph-mon[78597]: from='client.? 172.21.15.130:0/4052036142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:31:10.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:10 smithi016 ceph-mon[38248]: pgmap v195: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:10.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:10 smithi016 ceph-mon[38248]: osdmap e70: 12 total, 11 up, 12 in 2024-04-05T23:31:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:10 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:10.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:10 smithi016 ceph-mon[38248]: from='client.? 172.21.15.130:0/4052036142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:31:10.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:10 smithi130 ceph-mon[79417]: pgmap v195: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:10.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:10 smithi130 ceph-mon[79417]: osdmap e70: 12 total, 11 up, 12 in 2024-04-05T23:31:10.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:10 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:10.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:10 smithi130 ceph-mon[79417]: from='client.? 172.21.15.130:0/4052036142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T23:31:11.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:11 smithi027 ceph-mon[78597]: osdmap e71: 12 total, 11 up, 12 in 2024-04-05T23:31:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:11 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:11.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:11 smithi016 ceph-mon[38248]: osdmap e71: 12 total, 11 up, 12 in 2024-04-05T23:31:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:11 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:11 smithi130 ceph-mon[79417]: osdmap e71: 12 total, 11 up, 12 in 2024-04-05T23:31:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:11 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:12.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:12 smithi027 ceph-mon[78597]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:12.808 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:12 smithi130 ceph-mon[79417]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:12 smithi016 ceph-mon[38248]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:14.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:14 smithi130 ceph-mon[79417]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-05T23:31:14.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:14 smithi027 ceph-mon[78597]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-05T23:31:14.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:14 smithi016 ceph-mon[38248]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-05T23:31:15.665 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-05T23:31:15.666 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:15 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:15.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:15 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-05T23:31:15.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:15 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:15.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-05T23:31:15.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:15 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:16.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:16 smithi027 ceph-mon[78597]: Deploying daemon osd.11 on smithi130 2024-04-05T23:31:16.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:16 smithi027 ceph-mon[78597]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-05T23:31:16.739 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:16 smithi130 ceph-mon[79417]: Deploying daemon osd.11 on smithi130 2024-04-05T23:31:16.739 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:16 smithi130 ceph-mon[79417]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-05T23:31:16.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:16 smithi016 ceph-mon[38248]: Deploying daemon osd.11 on smithi130 2024-04-05T23:31:16.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:16 smithi016 ceph-mon[38248]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-05T23:31:18.675 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:18 smithi130 ceph-mon[79417]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-05T23:31:18.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:18 smithi027 ceph-mon[78597]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-05T23:31:18.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:18 smithi016 ceph-mon[38248]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-05T23:31:19.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:19 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:19.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:19 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:19.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:19 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:19.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:19 smithi130 ceph-mon[79417]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-05T23:31:20.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:19 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:20.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:19 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:20.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:19 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:20.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:19 smithi016 ceph-mon[38248]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-05T23:31:20.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:19 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:20.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:19 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:20.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:19 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:20.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:19 smithi027 ceph-mon[78597]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-05T23:31:21.194 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.202+0000 7fd8617fa700 1 -- 172.21.15.130:0/2383382439 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fd8640fd620 con 0x7fd84c0a1120 2024-04-05T23:31:21.195 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 11 on host 'smithi130' 2024-04-05T23:31:21.197 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd84c0a1120 msgr2=0x7fd84c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:21.197 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd84c0a1120 0x7fd84c0a35e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd854006010 tx=0x7fd8540037b0 comp rx=0 tx=0).stop 2024-04-05T23:31:21.197 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8640ffe40 msgr2=0x7fd8641990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:21.197 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8640ffe40 0x7fd8641990e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd85800b560 tx=0x7fd85800b920 comp rx=0 tx=0).stop 2024-04-05T23:31:21.197 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 shutdown_connections 2024-04-05T23:31:21.197 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd84c0a1120 0x7fd84c0a35e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:21.197 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864101960 0x7fd8641a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:21.197 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8640ffe40 0x7fd8641990e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:21.198 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 --2- 172.21.15.130:0/2383382439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd864100fa0 0x7fd864199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:21.198 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 >> 172.21.15.130:0/2383382439 conn(0x7fd8640fb400 msgr2=0x7fd8640fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:21.198 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 shutdown_connections 2024-04-05T23:31:21.198 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:21.204+0000 7fd86a49b700 1 -- 172.21.15.130:0/2383382439 wait complete. 2024-04-05T23:31:21.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:21 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:21 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:21 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:21.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:21 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:21.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:21 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:21 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:21 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:21 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:21.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:21 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:21 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:21 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:21.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:21 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:21.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:21 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:21 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:21 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:21 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:21.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:21 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:21 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:21 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:21.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:21 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:21.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:21 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:21 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:21 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:21.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:21 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:21.598 DEBUG:teuthology.orchestra.run.smithi130:osd.11> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.11.service 2024-04-05T23:31:21.600 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-05T23:31:21.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-05T23:31:21.633 INFO:journalctl@ceph.osd.11.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:31:21.979 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:31:22.441 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:22 smithi130 ceph-mon[79417]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-05T23:31:22.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:22 smithi027 ceph-mon[78597]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-05T23:31:22.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:22 smithi016 ceph-mon[38248]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-05T23:31:23.084 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:31:22 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:31:22.792+0000 7f366cc9c7c0 -1 osd.11 0 log_to_monitors true 2024-04-05T23:31:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.190+0000 7fc75cc00700 1 -- 172.21.15.16:0/1138174028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc758100ef0 msgr2=0x7fc758101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.190+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1138174028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc758100ef0 0x7fc758101330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc748009990 tx=0x7fc748009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.190+0000 7fc75cc00700 1 -- 172.21.15.16:0/1138174028 shutdown_connections 2024-04-05T23:31:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.190+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1138174028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc758100ef0 0x7fc758101330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.190+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1138174028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc758102a90 0x7fc7581093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.190+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1138174028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7581020d0 0x7fc758102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.190+0000 7fc75cc00700 1 -- 172.21.15.16:0/1138174028 >> 172.21.15.16:0/1138174028 conn(0x7fc7580fc4e0 msgr2=0x7fc7580fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.190+0000 7fc75cc00700 1 -- 172.21.15.16:0/1138174028 shutdown_connections 2024-04-05T23:31:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.190+0000 7fc75cc00700 1 -- 172.21.15.16:0/1138174028 wait complete. 2024-04-05T23:31:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.191+0000 7fc75cc00700 1 Processor -- start 2024-04-05T23:31:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc75cc00700 1 -- start start 2024-04-05T23:31:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc75cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc758100ef0 0x7fc758074be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc75cc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7581020d0 0x7fc7580732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc75cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc758102a90 0x7fc758073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc75cc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc758073ef0 con 0x7fc758102a90 2024-04-05T23:31:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc75cc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc758074060 con 0x7fc758100ef0 2024-04-05T23:31:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc75cc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7581a0f50 con 0x7fc7581020d0 2024-04-05T23:31:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc755e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc758100ef0 0x7fc758074be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc755400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7581020d0 0x7fc7580732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc755e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc758100ef0 0x7fc758074be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35800/0 (socket says 172.21.15.16:35800) 2024-04-05T23:31:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc755e00700 1 -- 172.21.15.16:0/1256075113 learned_addr learned my addr 172.21.15.16:0/1256075113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.192+0000 7fc756800700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc758102a90 0x7fc758073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc755e00700 1 -- 172.21.15.16:0/1256075113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7581020d0 msgr2=0x7fc7580732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc755e00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7581020d0 0x7fc7580732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc755e00700 1 -- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc758102a90 msgr2=0x7fc758073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc755e00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc758102a90 0x7fc758073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc755e00700 1 -- 172.21.15.16:0/1256075113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7480095f0 con 0x7fc758100ef0 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc755e00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc758100ef0 0x7fc758074be0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc748000c00 tx=0x7fc748005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc74ea00700 1 -- 172.21.15.16:0/1256075113 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74801d070 con 0x7fc758100ef0 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7581a1150 con 0x7fc758100ef0 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc74ea00700 1 -- 172.21.15.16:0/1256075113 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc748021410 con 0x7fc758100ef0 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.193+0000 7fc74ea00700 1 -- 172.21.15.16:0/1256075113 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748017650 con 0x7fc758100ef0 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.194+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7581a15c0 con 0x7fc758100ef0 2024-04-05T23:31:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.195+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc758108480 con 0x7fc758100ef0 2024-04-05T23:31:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.197+0000 7fc74ea00700 1 -- 172.21.15.16:0/1256075113 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fc748021580 con 0x7fc758100ef0 2024-04-05T23:31:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.199+0000 7fc74ea00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc73c0a11a0 0x7fc73c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.199+0000 7fc74ea00700 1 -- 172.21.15.16:0/1256075113 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fc7480cdd10 con 0x7fc758100ef0 2024-04-05T23:31:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.199+0000 7fc755400700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc73c0a11a0 0x7fc73c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.200+0000 7fc755400700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc73c0a11a0 0x7fc73c0a3660 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc744007c60 tx=0x7fc7440073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.203+0000 7fc74ea00700 1 -- 172.21.15.16:0/1256075113 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc748097b10 con 0x7fc758100ef0 2024-04-05T23:31:23.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:23 smithi016 ceph-mon[38248]: from='osd.11 [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-05T23:31:23.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:23 smithi016 ceph-mon[38248]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-05T23:31:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.420+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fc75804ebb0 con 0x7fc758100ef0 2024-04-05T23:31:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.421+0000 7fc74ea00700 1 -- 172.21.15.16:0/1256075113 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) v1 ==== 74+0+133 (secure 0 0 0) 0x7fc74809b130 con 0x7fc758100ef0 2024-04-05T23:31:23.414 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:23.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc73c0a11a0 msgr2=0x7fc73c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:23.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc73c0a11a0 0x7fc73c0a3660 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc744007c60 tx=0x7fc7440073d0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc758100ef0 msgr2=0x7fc758074be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc758100ef0 0x7fc758074be0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc748000c00 tx=0x7fc748005740 comp rx=0 tx=0).stop 2024-04-05T23:31:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 shutdown_connections 2024-04-05T23:31:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc73c0a11a0 0x7fc73c0a3660 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc758102a90 0x7fc758073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc758100ef0 0x7fc758074be0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 --2- 172.21.15.16:0/1256075113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7581020d0 0x7fc7580732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 >> 172.21.15.16:0/1256075113 conn(0x7fc7580fc4e0 msgr2=0x7fc7581055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 shutdown_connections 2024-04-05T23:31:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:23.423+0000 7fc75cc00700 1 -- 172.21.15.16:0/1256075113 wait complete. 2024-04-05T23:31:23.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:23 smithi027 ceph-mon[78597]: from='osd.11 [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-05T23:31:23.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:23 smithi027 ceph-mon[78597]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-05T23:31:23.538 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:23 smithi130 ceph-mon[79417]: from='osd.11 [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-05T23:31:23.538 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:23 smithi130 ceph-mon[79417]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-05T23:31:23.787 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"num_osds":12,"num_up_osds":11,"osd_up_since":1712359867,"num_in_osds":12,"osd_in_since":1712359868,"num_remapped_pgs":0} 2024-04-05T23:31:24.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: osdmap e72: 12 total, 11 up, 12 in 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='osd.11 [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1256075113' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:24.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:24.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:24 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: osdmap e72: 12 total, 11 up, 12 in 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='osd.11 [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1256075113' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:24.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:24.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:24 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.584 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:31:24.218+0000 7f3661b00700 -1 osd.11 0 waiting for initial osdmap 2024-04-05T23:31:24.584 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:31:24.236+0000 7f365e0f6700 -1 osd.11 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: osdmap e72: 12 total, 11 up, 12 in 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='osd.11 [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1256075113' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:31:24.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:24.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:24.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:24 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:24.788 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-05T23:31:25.161 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:31:25.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:25 smithi027 ceph-mon[78597]: Detected new or changed devices on smithi130 2024-04-05T23:31:25.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:25 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi130 to 4278M 2024-04-05T23:31:25.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:25 smithi027 ceph-mon[78597]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:31:25.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:25 smithi027 ceph-mon[78597]: osdmap e73: 12 total, 11 up, 12 in 2024-04-05T23:31:25.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:25.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:25 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:25.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:25 smithi016 ceph-mon[38248]: Detected new or changed devices on smithi130 2024-04-05T23:31:25.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:25 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi130 to 4278M 2024-04-05T23:31:25.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:25 smithi016 ceph-mon[38248]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:31:25.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:25 smithi016 ceph-mon[38248]: osdmap e73: 12 total, 11 up, 12 in 2024-04-05T23:31:25.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:25.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:25 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:25.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:25 smithi130 ceph-mon[79417]: Detected new or changed devices on smithi130 2024-04-05T23:31:25.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:25 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi130 to 4278M 2024-04-05T23:31:25.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:25 smithi130 ceph-mon[79417]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-05T23:31:25.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:25 smithi130 ceph-mon[79417]: osdmap e73: 12 total, 11 up, 12 in 2024-04-05T23:31:25.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:25.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:25 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.358+0000 7f6587a00700 1 -- 172.21.15.16:0/291097787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580102a90 msgr2=0x7f65801093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.358+0000 7f6587a00700 1 --2- 172.21.15.16:0/291097787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580102a90 0x7f65801093b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f657800b0a0 tx=0x7f657800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.358+0000 7f6587a00700 1 -- 172.21.15.16:0/291097787 shutdown_connections 2024-04-05T23:31:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.358+0000 7f6587a00700 1 --2- 172.21.15.16:0/291097787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580102a90 0x7f65801093b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.358+0000 7f6587a00700 1 --2- 172.21.15.16:0/291097787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6580100ef0 0x7f6580101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.358+0000 7f6587a00700 1 --2- 172.21.15.16:0/291097787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65801020d0 0x7f6580102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.358+0000 7f6587a00700 1 -- 172.21.15.16:0/291097787 >> 172.21.15.16:0/291097787 conn(0x7f65800fc4e0 msgr2=0x7f65800fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:26.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.358+0000 7f6587a00700 1 -- 172.21.15.16:0/291097787 shutdown_connections 2024-04-05T23:31:26.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.358+0000 7f6587a00700 1 -- 172.21.15.16:0/291097787 wait complete. 2024-04-05T23:31:26.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.359+0000 7f6587a00700 1 Processor -- start 2024-04-05T23:31:26.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6587a00700 1 -- start start 2024-04-05T23:31:26.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6587a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6580100ef0 0x7f6580199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:26.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6587a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65801020d0 0x7f65801996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6587a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580102a90 0x7f65801a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6587a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65801a0cb0 con 0x7f6580102a90 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6587a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65801a0df0 con 0x7f65801020d0 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6587a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65801a0f80 con 0x7f6580100ef0 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6585000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6580100ef0 0x7f6580199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6585a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580102a90 0x7f65801a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6585000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6580100ef0 0x7f6580199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:38798/0 (socket says 172.21.15.16:38798) 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f6585000700 1 -- 172.21.15.16:0/3345611269 learned_addr learned my addr 172.21.15.16:0/3345611269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.360+0000 7f657fe00700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65801020d0 0x7f65801996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f6585000700 1 -- 172.21.15.16:0/3345611269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65801020d0 msgr2=0x7f65801996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f6585000700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65801020d0 0x7f65801996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f6585000700 1 -- 172.21.15.16:0/3345611269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580102a90 msgr2=0x7f65801a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f6585000700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580102a90 0x7f65801a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f6585000700 1 -- 172.21.15.16:0/3345611269 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6578009d00 con 0x7f6580100ef0 2024-04-05T23:31:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f6585000700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6580100ef0 0x7f6580199170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f657400e9a0 tx=0x7f657400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f657d600700 1 -- 172.21.15.16:0/3345611269 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574009790 con 0x7f6580100ef0 2024-04-05T23:31:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65801a1260 con 0x7f6580100ef0 2024-04-05T23:31:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f657d600700 1 -- 172.21.15.16:0/3345611269 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f657400c810 con 0x7f6580100ef0 2024-04-05T23:31:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f657d600700 1 -- 172.21.15.16:0/3345611269 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574003c70 con 0x7f6580100ef0 2024-04-05T23:31:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.361+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65801a1760 con 0x7f6580100ef0 2024-04-05T23:31:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.362+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658004ebb0 con 0x7f6580100ef0 2024-04-05T23:31:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.364+0000 7f657d600700 1 -- 172.21.15.16:0/3345611269 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f657400cc80 con 0x7f6580100ef0 2024-04-05T23:31:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.365+0000 7f657d600700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f65680a1120 0x7f65680a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.366+0000 7f657d600700 1 -- 172.21.15.16:0/3345611269 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6574014070 con 0x7f6580100ef0 2024-04-05T23:31:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.366+0000 7f657fe00700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f65680a1120 0x7f65680a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.366+0000 7f657fe00700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f65680a1120 0x7f65680a35e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f656c009c60 tx=0x7f656c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.368+0000 7f657d600700 1 -- 172.21.15.16:0/3345611269 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f65740965d0 con 0x7f6580100ef0 2024-04-05T23:31:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.580+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f658010c7a0 con 0x7f6580100ef0 2024-04-05T23:31:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.581+0000 7f657d600700 1 -- 172.21.15.16:0/3345611269 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) v1 ==== 74+0+133 (secure 0 0 0) 0x7f6574099bf0 con 0x7f6580100ef0 2024-04-05T23:31:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f65680a1120 msgr2=0x7f65680a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f65680a1120 0x7f65680a35e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f656c009c60 tx=0x7f656c009260 comp rx=0 tx=0).stop 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6580100ef0 msgr2=0x7f6580199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6580100ef0 0x7f6580199170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f657400e9a0 tx=0x7f657400ecb0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 shutdown_connections 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f65680a1120 0x7f65680a35e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6580102a90 0x7f65801a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6580100ef0 0x7f6580199170 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 --2- 172.21.15.16:0/3345611269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65801020d0 0x7f65801996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 >> 172.21.15.16:0/3345611269 conn(0x7f65800fc4e0 msgr2=0x7f6580073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.583+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 shutdown_connections 2024-04-05T23:31:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:26.584+0000 7f6587a00700 1 -- 172.21.15.16:0/3345611269 wait complete. 2024-04-05T23:31:26.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:26 smithi016 ceph-mon[38248]: purged_snaps scrub starts 2024-04-05T23:31:26.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:26 smithi016 ceph-mon[38248]: purged_snaps scrub ok 2024-04-05T23:31:26.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:26 smithi016 ceph-mon[38248]: osd.11 [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] boot 2024-04-05T23:31:26.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:26 smithi016 ceph-mon[38248]: osdmap e74: 12 total, 12 up, 12 in 2024-04-05T23:31:26.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:26 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:26.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:26 smithi016 ceph-mon[38248]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:26.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:26 smithi130 ceph-mon[79417]: purged_snaps scrub starts 2024-04-05T23:31:26.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:26 smithi130 ceph-mon[79417]: purged_snaps scrub ok 2024-04-05T23:31:26.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:26 smithi130 ceph-mon[79417]: osd.11 [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] boot 2024-04-05T23:31:26.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:26 smithi130 ceph-mon[79417]: osdmap e74: 12 total, 12 up, 12 in 2024-04-05T23:31:26.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:26 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:26.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:26 smithi130 ceph-mon[79417]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:26.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:26 smithi027 ceph-mon[78597]: purged_snaps scrub starts 2024-04-05T23:31:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:26 smithi027 ceph-mon[78597]: purged_snaps scrub ok 2024-04-05T23:31:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:26 smithi027 ceph-mon[78597]: osd.11 [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] boot 2024-04-05T23:31:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:26 smithi027 ceph-mon[78597]: osdmap e74: 12 total, 12 up, 12 in 2024-04-05T23:31:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:26 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:31:26.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:26 smithi027 ceph-mon[78597]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-05T23:31:26.975 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":75,"num_osds":12,"num_up_osds":12,"osd_up_since":1712359885,"num_in_osds":12,"osd_in_since":1712359868,"num_remapped_pgs":0} 2024-04-05T23:31:26.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-05T23:31:27.338 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:31:27.483 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:27 smithi016 ceph-mon[38248]: osdmap e75: 12 total, 12 up, 12 in 2024-04-05T23:31:27.484 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:27 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3345611269' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T23:31:27.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:27 smithi130 ceph-mon[79417]: osdmap e75: 12 total, 12 up, 12 in 2024-04-05T23:31:27.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:27 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3345611269' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T23:31:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:27 smithi027 ceph-mon[78597]: osdmap e75: 12 total, 12 up, 12 in 2024-04-05T23:31:27.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:27 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3345611269' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T23:31:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.453+0000 7f1a83600700 1 -- 172.21.15.16:0/1243010577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c100f30 msgr2=0x7f1a7c101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.453+0000 7f1a83600700 1 --2- 172.21.15.16:0/1243010577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c100f30 0x7f1a7c101390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009990 tx=0x7f1a68009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.453+0000 7f1a83600700 1 -- 172.21.15.16:0/1243010577 shutdown_connections 2024-04-05T23:31:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.453+0000 7f1a83600700 1 --2- 172.21.15.16:0/1243010577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a7c1018d0 0x7f1a7c10ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.453+0000 7f1a83600700 1 --2- 172.21.15.16:0/1243010577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a7c106f70 0x7f1a7c107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.453+0000 7f1a83600700 1 --2- 172.21.15.16:0/1243010577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c100f30 0x7f1a7c101390 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.453+0000 7f1a83600700 1 -- 172.21.15.16:0/1243010577 >> 172.21.15.16:0/1243010577 conn(0x7f1a7c0780d0 msgr2=0x7f1a7c0784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.453+0000 7f1a83600700 1 -- 172.21.15.16:0/1243010577 shutdown_connections 2024-04-05T23:31:28.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.453+0000 7f1a83600700 1 -- 172.21.15.16:0/1243010577 wait complete. 2024-04-05T23:31:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.455+0000 7f1a83600700 1 Processor -- start 2024-04-05T23:31:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.455+0000 7f1a83600700 1 -- start start 2024-04-05T23:31:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.455+0000 7f1a83600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a7c100f30 0x7f1a7c199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.455+0000 7f1a83600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c1018d0 0x7f1a7c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a83600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a7c106f70 0x7f1a7c19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a83600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c19e520 con 0x7f1a7c1018d0 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a83600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c19e690 con 0x7f1a7c100f30 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a83600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c19e850 con 0x7f1a7c106f70 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a7be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c1018d0 0x7f1a7c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a81600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a7c106f70 0x7f1a7c19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a7be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c1018d0 0x7f1a7c199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52824/0 (socket says 172.21.15.16:52824) 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a7be00700 1 -- 172.21.15.16:0/1606087025 learned_addr learned my addr 172.21.15.16:0/1606087025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a81600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a7c106f70 0x7f1a7c19df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:38806/0 (socket says 172.21.15.16:38806) 2024-04-05T23:31:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a80c00700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a7c100f30 0x7f1a7c199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a7be00700 1 -- 172.21.15.16:0/1606087025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a7c106f70 msgr2=0x7f1a7c19df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a7be00700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a7c106f70 0x7f1a7c19df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a7be00700 1 -- 172.21.15.16:0/1606087025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a7c100f30 msgr2=0x7f1a7c199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a7be00700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a7c100f30 0x7f1a7c199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a7be00700 1 -- 172.21.15.16:0/1606087025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a680095f0 con 0x7f1a7c1018d0 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a81600700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a7c106f70 0x7f1a7c19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.456+0000 7f1a80c00700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a7c100f30 0x7f1a7c199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.457+0000 7f1a7be00700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c1018d0 0x7f1a7c199b60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1a68005910 tx=0x7f1a68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.457+0000 7f1a79600700 1 -- 172.21.15.16:0/1606087025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6801d070 con 0x7f1a7c1018d0 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.457+0000 7f1a79600700 1 -- 172.21.15.16:0/1606087025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a68022470 con 0x7f1a7c1018d0 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.457+0000 7f1a79600700 1 -- 172.21.15.16:0/1606087025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a68017650 con 0x7f1a7c1018d0 2024-04-05T23:31:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.457+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7c19ead0 con 0x7f1a7c1018d0 2024-04-05T23:31:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.457+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c1a61c0 con 0x7f1a7c1018d0 2024-04-05T23:31:28.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.458+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a7c1042b0 con 0x7f1a7c1018d0 2024-04-05T23:31:28.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.460+0000 7f1a79600700 1 -- 172.21.15.16:0/1606087025 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f1a680225e0 con 0x7f1a7c1018d0 2024-04-05T23:31:28.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.462+0000 7f1a79600700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1a640a11a0 0x7f1a640a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:28.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.462+0000 7f1a79600700 1 -- 172.21.15.16:0/1606087025 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1a680cd660 con 0x7f1a7c1018d0 2024-04-05T23:31:28.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.462+0000 7f1a80c00700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1a640a11a0 0x7f1a640a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:28.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.463+0000 7f1a80c00700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1a640a11a0 0x7f1a640a3660 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009fd0 tx=0x7f1a70009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:28.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.466+0000 7f1a79600700 1 -- 172.21.15.16:0/1606087025 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1a680973a0 con 0x7f1a7c1018d0 2024-04-05T23:31:28.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:28 smithi016 ceph-mon[38248]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:28.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:28 smithi130 ceph-mon[79417]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.677+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a7c102620 con 0x7f1a7c1018d0 2024-04-05T23:31:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.678+0000 7f1a79600700 1 -- 172.21.15.16:0/1606087025 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+18387 (secure 0 0 0) 0x7f1a6809a9c0 con 0x7f1a7c1018d0 2024-04-05T23:31:28.672 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:28.672 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":75,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:31:26.220652+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:31:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1a640a11a0 msgr2=0x7f1a640a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1a640a11a0 0x7f1a640a3660 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009fd0 tx=0x7f1a70009260 comp rx=0 tx=0).stop 2024-04-05T23:31:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c1018d0 msgr2=0x7f1a7c199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c1018d0 0x7f1a7c199b60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1a68005910 tx=0x7f1a68004730 comp rx=0 tx=0).stop 2024-04-05T23:31:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 shutdown_connections 2024-04-05T23:31:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1a640a11a0 0x7f1a640a3660 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a7c106f70 0x7f1a7c19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a7c100f30 0x7f1a7c199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 --2- 172.21.15.16:0/1606087025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a7c1018d0 0x7f1a7c199b60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 >> 172.21.15.16:0/1606087025 conn(0x7f1a7c0780d0 msgr2=0x7f1a7c10c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 shutdown_connections 2024-04-05T23:31:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:28.681+0000 7f1a83600700 1 -- 172.21.15.16:0/1606087025 wait complete. 2024-04-05T23:31:28.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:28 smithi027 ceph-mon[78597]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:29.110 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:31:29.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-05T23:31:29.472 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:31:29.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:29 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1606087025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:31:29.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:29 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1606087025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:31:29.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:29 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1606087025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:31:30.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:30 smithi016 ceph-mon[38248]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:30.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:30 smithi130 ceph-mon[79417]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.645+0000 7efd0d000700 1 -- 172.21.15.16:0/1532795330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd08106f70 msgr2=0x7efd08107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.645+0000 7efd0d000700 1 --2- 172.21.15.16:0/1532795330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd08106f70 0x7efd08107350 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009990 tx=0x7efcf8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.645+0000 7efd0d000700 1 -- 172.21.15.16:0/1532795330 shutdown_connections 2024-04-05T23:31:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.645+0000 7efd0d000700 1 --2- 172.21.15.16:0/1532795330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd08106f70 0x7efd08107350 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.645+0000 7efd0d000700 1 --2- 172.21.15.16:0/1532795330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd081018d0 0x7efd0810ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.645+0000 7efd0d000700 1 --2- 172.21.15.16:0/1532795330 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd08100f30 0x7efd08101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.645+0000 7efd0d000700 1 -- 172.21.15.16:0/1532795330 >> 172.21.15.16:0/1532795330 conn(0x7efd080780d0 msgr2=0x7efd080784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.645+0000 7efd0d000700 1 -- 172.21.15.16:0/1532795330 shutdown_connections 2024-04-05T23:31:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.646+0000 7efd0d000700 1 -- 172.21.15.16:0/1532795330 wait complete. 2024-04-05T23:31:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.646+0000 7efd0d000700 1 Processor -- start 2024-04-05T23:31:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd0d000700 1 -- start start 2024-04-05T23:31:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd0d000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd08100f30 0x7efd08199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd0d000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd081018d0 0x7efd08199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd0d000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd08106f70 0x7efd0819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd0d000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0819e470 con 0x7efd08106f70 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd0d000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0819e5e0 con 0x7efd081018d0 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd0d000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0819e7a0 con 0x7efd08100f30 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd05e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd08100f30 0x7efd08199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd05e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd08100f30 0x7efd08199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:38830/0 (socket says 172.21.15.16:38830) 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd05e00700 1 -- 172.21.15.16:0/2761171863 learned_addr learned my addr 172.21.15.16:0/2761171863 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efd06800700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd08106f70 0x7efd0819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.647+0000 7efcffe00700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd081018d0 0x7efd08199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efd05e00700 1 -- 172.21.15.16:0/2761171863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd081018d0 msgr2=0x7efd08199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efd05e00700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd081018d0 0x7efd08199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efd05e00700 1 -- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd08106f70 msgr2=0x7efd0819de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efd05e00700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd08106f70 0x7efd0819de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efd05e00700 1 -- 172.21.15.16:0/2761171863 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf80095f0 con 0x7efd08100f30 2024-04-05T23:31:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efd05e00700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd08100f30 0x7efd08199570 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efcf8005910 tx=0x7efcf8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efcfea00700 1 -- 172.21.15.16:0/2761171863 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf801d070 con 0x7efd08100f30 2024-04-05T23:31:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efcfea00700 1 -- 172.21.15.16:0/2761171863 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcf8022470 con 0x7efd08100f30 2024-04-05T23:31:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efcfea00700 1 -- 172.21.15.16:0/2761171863 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8017650 con 0x7efd08100f30 2024-04-05T23:31:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0819ea20 con 0x7efd08100f30 2024-04-05T23:31:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.648+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd081a62e0 con 0x7efd08100f30 2024-04-05T23:31:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.649+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd081042b0 con 0x7efd08100f30 2024-04-05T23:31:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.651+0000 7efcfea00700 1 -- 172.21.15.16:0/2761171863 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7efcf80225e0 con 0x7efd08100f30 2024-04-05T23:31:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.653+0000 7efcfea00700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efcec0a11a0 0x7efcec0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.653+0000 7efcfea00700 1 -- 172.21.15.16:0/2761171863 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efcf80cd650 con 0x7efd08100f30 2024-04-05T23:31:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.653+0000 7efcffe00700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efcec0a11a0 0x7efcec0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.653+0000 7efcffe00700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efcec0a11a0 0x7efcec0a3660 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efd0819aad0 tx=0x7efcf0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.655+0000 7efcfea00700 1 -- 172.21.15.16:0/2761171863 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efcf8097270 con 0x7efd08100f30 2024-04-05T23:31:30.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:30 smithi027 ceph-mon[78597]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:30.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.869+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd0819a880 con 0x7efd08100f30 2024-04-05T23:31:30.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.870+0000 7efcfea00700 1 -- 172.21.15.16:0/2761171863 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7efcf809a890 con 0x7efd08100f30 2024-04-05T23:31:30.864 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:31:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efcec0a11a0 msgr2=0x7efcec0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efcec0a11a0 0x7efcec0a3660 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efd0819aad0 tx=0x7efcf0009260 comp rx=0 tx=0).stop 2024-04-05T23:31:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd08100f30 msgr2=0x7efd08199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd08100f30 0x7efd08199570 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efcf8005910 tx=0x7efcf8004730 comp rx=0 tx=0).stop 2024-04-05T23:31:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 shutdown_connections 2024-04-05T23:31:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efcec0a11a0 0x7efcec0a3660 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd08106f70 0x7efd0819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd08100f30 0x7efd08199570 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 --2- 172.21.15.16:0/2761171863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd081018d0 0x7efd08199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 >> 172.21.15.16:0/2761171863 conn(0x7efd080780d0 msgr2=0x7efd0810c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 shutdown_connections 2024-04-05T23:31:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:30.873+0000 7efd0d000700 1 -- 172.21.15.16:0/2761171863 wait complete. 2024-04-05T23:31:31.239 INFO:tasks.cephadm:Adding mds.a on smithi016 2024-04-05T23:31:31.239 INFO:tasks.cephadm:Adding mds.d on smithi016 2024-04-05T23:31:31.239 INFO:tasks.cephadm:Adding mds.g on smithi016 2024-04-05T23:31:31.240 INFO:tasks.cephadm:Adding mds.j on smithi016 2024-04-05T23:31:31.240 INFO:tasks.cephadm:Adding mds.b on smithi027 2024-04-05T23:31:31.240 INFO:tasks.cephadm:Adding mds.e on smithi027 2024-04-05T23:31:31.240 INFO:tasks.cephadm:Adding mds.h on smithi027 2024-04-05T23:31:31.240 INFO:tasks.cephadm:Adding mds.k on smithi027 2024-04-05T23:31:31.240 INFO:tasks.cephadm:Adding mds.c on smithi130 2024-04-05T23:31:31.240 INFO:tasks.cephadm:Adding mds.f on smithi130 2024-04-05T23:31:31.240 INFO:tasks.cephadm:Adding mds.i on smithi130 2024-04-05T23:31:31.240 INFO:tasks.cephadm:Adding mds.l on smithi130 2024-04-05T23:31:31.240 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph orch apply mds all '12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi027=b;smithi027=e;smithi027=h;smithi027=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l' 2024-04-05T23:31:31.474 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:31:31.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:31 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2761171863' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:31:31.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:31 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2761171863' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:31:31.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:31 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2761171863' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:31:32.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 -- 172.21.15.130:0/311202830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102270 msgr2=0x7f8d9c1026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:32.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 --2- 172.21.15.130:0/311202830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102270 0x7f8d9c1026d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009990 tx=0x7f8d8c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 -- 172.21.15.130:0/311202830 shutdown_connections 2024-04-05T23:31:32.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 --2- 172.21.15.130:0/311202830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d9c0ff290 0x7f8d9c0ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 --2- 172.21.15.130:0/311202830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c102c10 0x7f8d9c1094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 --2- 172.21.15.130:0/311202830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102270 0x7f8d9c1026d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 -- 172.21.15.130:0/311202830 >> 172.21.15.130:0/311202830 conn(0x7f8d9c0fabf0 msgr2=0x7f8d9c0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:32.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 -- 172.21.15.130:0/311202830 shutdown_connections 2024-04-05T23:31:32.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 -- 172.21.15.130:0/311202830 wait complete. 2024-04-05T23:31:32.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 Processor -- start 2024-04-05T23:31:32.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.508+0000 7f8da2514700 1 -- start start 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da2514700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d9c0ff290 0x7f8d9c101b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da2514700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c102270 0x7f8d9c100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da2514700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102c10 0x7f8d9c100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da2514700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c100f00 con 0x7f8d9c0ff290 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da2514700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c101070 con 0x7f8d9c102270 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da2514700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c1a0f20 con 0x7f8d9c102c10 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da0ab1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102c10 0x7f8d9c100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da0ab1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102c10 0x7f8d9c100760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:53450/0 (socket says 172.21.15.130:53450) 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da0ab1700 1 -- 172.21.15.130:0/2354767836 learned_addr learned my addr 172.21.15.130:0/2354767836 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8d9b7fe700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c102270 0x7f8d9c100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da0ab1700 1 -- 172.21.15.130:0/2354767836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c102270 msgr2=0x7f8d9c100220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:32.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da0ab1700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c102270 0x7f8d9c100220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da0ab1700 1 -- 172.21.15.130:0/2354767836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d9c0ff290 msgr2=0x7f8d9c101b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da0ab1700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d9c0ff290 0x7f8d9c101b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da0ab1700 1 -- 172.21.15.130:0/2354767836 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c0095f0 con 0x7f8d9c102c10 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8d9b7fe700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c102270 0x7f8d9c100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8da0ab1700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102c10 0x7f8d9c100760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400b970 tx=0x7f8d9400bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.509+0000 7f8d997fa700 1 -- 172.21.15.130:0/2354767836 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9400c9b0 con 0x7f8d9c102c10 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.510+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c1a11b0 con 0x7f8d9c102c10 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.510+0000 7f8d997fa700 1 -- 172.21.15.130:0/2354767836 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d94014440 con 0x7f8d9c102c10 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.510+0000 7f8d997fa700 1 -- 172.21.15.130:0/2354767836 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d940075d0 con 0x7f8d9c102c10 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.510+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c1a16b0 con 0x7f8d9c102c10 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.511+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9c108450 con 0x7f8d9c102c10 2024-04-05T23:31:32.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.511+0000 7f8d997fa700 1 -- 172.21.15.130:0/2354767836 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f8d940145b0 con 0x7f8d9c102c10 2024-04-05T23:31:32.506 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.512+0000 7f8d997fa700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8d880a1120 0x7f8d880a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:32.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.512+0000 7f8d997fa700 1 -- 172.21.15.130:0/2354767836 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8d94006e40 con 0x7f8d9c102c10 2024-04-05T23:31:32.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.512+0000 7f8d9bfff700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8d880a1120 0x7f8d880a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:32.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.513+0000 7f8d9bfff700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8d880a1120 0x7f8d880a35e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0693a0 tx=0x7f8d84005ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:32.509 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.516+0000 7f8d997fa700 1 -- 172.21.15.130:0/2354767836 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8d94095810 con 0x7f8d9c102c10 2024-04-05T23:31:32.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:32 smithi016 ceph-mon[38248]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:32.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:32 smithi130 ceph-mon[79417]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:32.711 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.719+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi027=b;smithi027=e;smithi027=h;smithi027=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d9c10cb20 con 0x7f8d880a1120 2024-04-05T23:31:32.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:32 smithi027 ceph-mon[78597]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:32.720 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.728+0000 7f8d997fa700 1 -- 172.21.15.130:0/2354767836 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8d9c10cb20 con 0x7f8d880a1120 2024-04-05T23:31:32.720 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mds.all update... 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8d880a1120 msgr2=0x7f8d880a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8d880a1120 0x7f8d880a35e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0693a0 tx=0x7f8d84005ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102c10 msgr2=0x7f8d9c100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102c10 0x7f8d9c100760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400b970 tx=0x7f8d9400bd30 comp rx=0 tx=0).stop 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 shutdown_connections 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f8d880a1120 0x7f8d880a35e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d9c0ff290 0x7f8d9c101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c102c10 0x7f8d9c100760 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 --2- 172.21.15.130:0/2354767836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d9c102270 0x7f8d9c100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 >> 172.21.15.130:0/2354767836 conn(0x7f8d9c0fabf0 msgr2=0x7f8d9c105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 shutdown_connections 2024-04-05T23:31:32.723 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-05T23:31:32.730+0000 7f8da2514700 1 -- 172.21.15.130:0/2354767836 wait complete. 2024-04-05T23:31:33.045 DEBUG:teuthology.orchestra.run.smithi016:mds.a> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.a.service 2024-04-05T23:31:33.051 DEBUG:teuthology.orchestra.run.smithi016:mds.d> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.d.service 2024-04-05T23:31:33.095 DEBUG:teuthology.orchestra.run.smithi016:mds.g> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.g.service 2024-04-05T23:31:33.126 INFO:journalctl@ceph.mds.a.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:31:33.129 DEBUG:teuthology.orchestra.run.smithi016:mds.j> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.j.service 2024-04-05T23:31:33.166 INFO:journalctl@ceph.mds.d.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:31:33.168 DEBUG:teuthology.orchestra.run.smithi027:mds.b> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.b.service 2024-04-05T23:31:33.171 DEBUG:teuthology.orchestra.run.smithi027:mds.e> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.e.service 2024-04-05T23:31:33.201 INFO:journalctl@ceph.mds.g.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:31:33.210 INFO:journalctl@ceph.mds.b.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:31:33.211 DEBUG:teuthology.orchestra.run.smithi027:mds.h> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.h.service 2024-04-05T23:31:33.235 INFO:journalctl@ceph.mds.j.smithi016.stdout:-- Logs begin at Fri 2024-04-05 23:08:52 UTC. -- 2024-04-05T23:31:33.242 INFO:journalctl@ceph.mds.e.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:31:33.244 DEBUG:teuthology.orchestra.run.smithi027:mds.k> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.k.service 2024-04-05T23:31:33.273 INFO:journalctl@ceph.mds.h.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:31:33.274 DEBUG:teuthology.orchestra.run.smithi130:mds.c> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.c.service 2024-04-05T23:31:33.276 DEBUG:teuthology.orchestra.run.smithi130:mds.f> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.f.service 2024-04-05T23:31:33.302 INFO:journalctl@ceph.mds.k.smithi027.stdout:-- Logs begin at Fri 2024-04-05 23:02:26 UTC. -- 2024-04-05T23:31:33.306 INFO:journalctl@ceph.mds.c.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:31:33.307 DEBUG:teuthology.orchestra.run.smithi130:mds.i> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.i.service 2024-04-05T23:31:33.334 INFO:journalctl@ceph.mds.f.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:31:33.335 DEBUG:teuthology.orchestra.run.smithi130:mds.l> sudo journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.l.service 2024-04-05T23:31:33.361 INFO:journalctl@ceph.mds.i.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:31:33.361 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-05T23:31:33.362 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:31:33.387 INFO:journalctl@ceph.mds.l.smithi130.stdout:-- Logs begin at Fri 2024-04-05 23:01:16 UTC. -- 2024-04-05T23:31:33.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.724+0000 7fa36ec00700 1 -- 172.21.15.16:0/273599798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa368102940 msgr2=0x7fa3681091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:33.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.724+0000 7fa36ec00700 1 --2- 172.21.15.16:0/273599798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa368102940 0x7fa3681091f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa36000b0d0 tx=0x7fa36000b3e0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.724+0000 7fa36ec00700 1 -- 172.21.15.16:0/273599798 shutdown_connections 2024-04-05T23:31:33.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.724+0000 7fa36ec00700 1 --2- 172.21.15.16:0/273599798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa368102940 0x7fa3681091f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.724+0000 7fa36ec00700 1 --2- 172.21.15.16:0/273599798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa368100db0 0x7fa3681011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.724+0000 7fa36ec00700 1 --2- 172.21.15.16:0/273599798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa368101fa0 0x7fa368102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.724+0000 7fa36ec00700 1 -- 172.21.15.16:0/273599798 >> 172.21.15.16:0/273599798 conn(0x7fa3680fc320 msgr2=0x7fa3680fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:33.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.724+0000 7fa36ec00700 1 -- 172.21.15.16:0/273599798 shutdown_connections 2024-04-05T23:31:33.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.725+0000 7fa36ec00700 1 -- 172.21.15.16:0/273599798 wait complete. 2024-04-05T23:31:33.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.726+0000 7fa36ec00700 1 Processor -- start 2024-04-05T23:31:33.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.726+0000 7fa36ec00700 1 -- start start 2024-04-05T23:31:33.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa36ec00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa368100db0 0x7fa368196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:33.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa36ec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa368101fa0 0x7fa368197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:33.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa36ec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa368102940 0x7fa36819e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:33.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa36ec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36819e950 con 0x7fa368101fa0 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa36ec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36819ea90 con 0x7fa368102940 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa36ec00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36819ec20 con 0x7fa368100db0 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa367400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa368101fa0 0x7fa368197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa367e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa368100db0 0x7fa368196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa367e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa368100db0 0x7fa368196e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55888/0 (socket says 172.21.15.16:55888) 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa367e00700 1 -- 172.21.15.16:0/4170008439 learned_addr learned my addr 172.21.15.16:0/4170008439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa36cc00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa368102940 0x7fa36819e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa367e00700 1 -- 172.21.15.16:0/4170008439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa368102940 msgr2=0x7fa36819e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa367e00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa368102940 0x7fa36819e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa367e00700 1 -- 172.21.15.16:0/4170008439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa368101fa0 msgr2=0x7fa368197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa367e00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa368101fa0 0x7fa368197360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.727+0000 7fa367e00700 1 -- 172.21.15.16:0/4170008439 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa360009d00 con 0x7fa368100db0 2024-04-05T23:31:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.728+0000 7fa367e00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa368100db0 0x7fa368196e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa35800e970 tx=0x7fa35800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.728+0000 7fa364c00700 1 -- 172.21.15.16:0/4170008439 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358009790 con 0x7fa368100db0 2024-04-05T23:31:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.728+0000 7fa364c00700 1 -- 172.21.15.16:0/4170008439 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa35800c7e0 con 0x7fa368100db0 2024-04-05T23:31:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.728+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36819ef00 con 0x7fa368100db0 2024-04-05T23:31:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.728+0000 7fa364c00700 1 -- 172.21.15.16:0/4170008439 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358003c70 con 0x7fa368100db0 2024-04-05T23:31:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.728+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36819f330 con 0x7fa368100db0 2024-04-05T23:31:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.729+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3681081e0 con 0x7fa368100db0 2024-04-05T23:31:33.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.731+0000 7fa364c00700 1 -- 172.21.15.16:0/4170008439 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fa358004220 con 0x7fa368100db0 2024-04-05T23:31:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.732+0000 7fa364c00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa34c0a11a0 0x7fa34c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.733+0000 7fa364c00700 1 -- 172.21.15.16:0/4170008439 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa358014070 con 0x7fa368100db0 2024-04-05T23:31:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.733+0000 7fa367400700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa34c0a11a0 0x7fa34c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.733+0000 7fa367400700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa34c0a11a0 0x7fa34c0a3660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa354005950 tx=0x7fa35400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:33.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.739+0000 7fa364c00700 1 -- 172.21.15.16:0/4170008439 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa3580961e0 con 0x7fa368100db0 2024-04-05T23:31:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.960+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa36810c620 con 0x7fa368100db0 2024-04-05T23:31:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.961+0000 7fa364c00700 1 -- 172.21.15.16:0/4170008439 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+18387 (secure 0 0 0) 0x7fa358099800 con 0x7fa368100db0 2024-04-05T23:31:33.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:33.955 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":75,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:31:26.220652+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:31:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa34c0a11a0 msgr2=0x7fa34c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa34c0a11a0 0x7fa34c0a3660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa354005950 tx=0x7fa35400b410 comp rx=0 tx=0).stop 2024-04-05T23:31:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa368100db0 msgr2=0x7fa368196e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa368100db0 0x7fa368196e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa35800e970 tx=0x7fa35800ec80 comp rx=0 tx=0).stop 2024-04-05T23:31:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 shutdown_connections 2024-04-05T23:31:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa34c0a11a0 0x7fa34c0a3660 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa368102940 0x7fa36819e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa368100db0 0x7fa368196e20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 --2- 172.21.15.16:0/4170008439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa368101fa0 0x7fa368197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 >> 172.21.15.16:0/4170008439 conn(0x7fa3680fc320 msgr2=0x7fa368105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 shutdown_connections 2024-04-05T23:31:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:33.964+0000 7fa36ec00700 1 -- 172.21.15.16:0/4170008439 wait complete. 2024-04-05T23:31:33.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi027=b;smithi027=e;smithi027=h;smithi027=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:31:33.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi027=b;smithi027=e;smithi027=h;smithi027=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-05T23:31:33.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:33.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:33.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:33.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:33.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:33.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:33.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:33.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:33.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:33 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:33.978 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:31:33.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:31:33.998 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi027=b;smithi027=e;smithi027=h;smithi027=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:31:33.998 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi027=b;smithi027=e;smithi027=h;smithi027=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-05T23:31:33.998 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:33.998 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:33.998 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:33.998 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:33.999 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:33.999 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:33.999 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:33.999 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:33.999 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:33 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:34.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.187+0000 7f0572c00700 1 -- 172.21.15.16:0/348200974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 msgr2=0x7f056c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.187+0000 7f0572c00700 1 --2- 172.21.15.16:0/348200974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 0x7f056c101280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f055c009990 tx=0x7f055c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.187+0000 7f0572c00700 1 -- 172.21.15.16:0/348200974 shutdown_connections 2024-04-05T23:31:34.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.187+0000 7f0572c00700 1 --2- 172.21.15.16:0/348200974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 0x7f056c101280 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.187+0000 7f0572c00700 1 --2- 172.21.15.16:0/348200974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f056c102a00 0x7f056c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.187+0000 7f0572c00700 1 --2- 172.21.15.16:0/348200974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c102040 0x7f056c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.187+0000 7f0572c00700 1 -- 172.21.15.16:0/348200974 >> 172.21.15.16:0/348200974 conn(0x7f056c0fc440 msgr2=0x7f056c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:34.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.187+0000 7f0572c00700 1 -- 172.21.15.16:0/348200974 shutdown_connections 2024-04-05T23:31:34.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.188+0000 7f0572c00700 1 -- 172.21.15.16:0/348200974 wait complete. 2024-04-05T23:31:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.188+0000 7f0572c00700 1 Processor -- start 2024-04-05T23:31:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0572c00700 1 -- start start 2024-04-05T23:31:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0572c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 0x7f056c074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0572c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f056c102040 0x7f056c0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0572c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c102a00 0x7f056c073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0572c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c073fb0 con 0x7f056c100e60 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0572c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c074120 con 0x7f056c102a00 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0572c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c1a0ee0 con 0x7f056c102040 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0563e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 0x7f056c074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0563e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 0x7f056c074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54844/0 (socket says 172.21.15.16:54844) 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0563e00700 1 -- 172.21.15.16:0/1222819876 learned_addr learned my addr 172.21.15.16:0/1222819876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f056be00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f056c102040 0x7f056c0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.189+0000 7f0570c00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c102a00 0x7f056c073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0563e00700 1 -- 172.21.15.16:0/1222819876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f056c102040 msgr2=0x7f056c0732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0563e00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f056c102040 0x7f056c0732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0563e00700 1 -- 172.21.15.16:0/1222819876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c102a00 msgr2=0x7f056c073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0563e00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c102a00 0x7f056c073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0563e00700 1 -- 172.21.15.16:0/1222819876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055c0095f0 con 0x7f056c100e60 2024-04-05T23:31:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0570c00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c102a00 0x7f056c073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:31:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0563e00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 0x7f056c074c40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f055c000c00 tx=0x7f055c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0569600700 1 -- 172.21.15.16:0/1222819876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c01d070 con 0x7f056c100e60 2024-04-05T23:31:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0569600700 1 -- 172.21.15.16:0/1222819876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f055c021410 con 0x7f056c100e60 2024-04-05T23:31:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0569600700 1 -- 172.21.15.16:0/1222819876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c017650 con 0x7f056c100e60 2024-04-05T23:31:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c1a1110 con 0x7f056c100e60 2024-04-05T23:31:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.190+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c1a14d0 con 0x7f056c100e60 2024-04-05T23:31:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.191+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c1083b0 con 0x7f056c100e60 2024-04-05T23:31:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.193+0000 7f0569600700 1 -- 172.21.15.16:0/1222819876 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f055c021580 con 0x7f056c100e60 2024-04-05T23:31:34.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.196+0000 7f0569600700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f05500a11a0 0x7f05500a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.196+0000 7f0569600700 1 -- 172.21.15.16:0/1222819876 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f055c0cded0 con 0x7f056c100e60 2024-04-05T23:31:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.196+0000 7f056be00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f05500a11a0 0x7f05500a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.196+0000 7f056be00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f05500a11a0 0x7f05500a3660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f056c100b80 tx=0x7f0554009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:34.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.199+0000 7f0569600700 1 -- 172.21.15.16:0/1222819876 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f055c097af0 con 0x7f056c100e60 2024-04-05T23:31:34.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi027=b;smithi027=e;smithi027=h;smithi027=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:31:34.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi027=b;smithi027=e;smithi027=h;smithi027=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-05T23:31:34.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:34.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:34.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:34.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:34.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:31:34.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:34.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:34.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:34.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:33 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:34.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.415+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f056c107d90 con 0x7f056c100e60 2024-04-05T23:31:34.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.415+0000 7f0569600700 1 -- 172.21.15.16:0/1222819876 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7f055c09b110 con 0x7f056c100e60 2024-04-05T23:31:34.408 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.417+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f05500a11a0 msgr2=0x7f05500a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.417+0000 7f0572c00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f05500a11a0 0x7f05500a3660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f056c100b80 tx=0x7f0554009260 comp rx=0 tx=0).stop 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.417+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 msgr2=0x7f056c074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.417+0000 7f0572c00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 0x7f056c074c40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f055c000c00 tx=0x7f055c009580 comp rx=0 tx=0).stop 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.418+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 shutdown_connections 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.418+0000 7f0572c00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f05500a11a0 0x7f05500a3660 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.418+0000 7f0572c00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f056c100e60 0x7f056c074c40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.418+0000 7f0572c00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c102a00 0x7f056c073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.418+0000 7f0572c00700 1 --2- 172.21.15.16:0/1222819876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f056c102040 0x7f056c0732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.418+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 >> 172.21.15.16:0/1222819876 conn(0x7f056c0fc440 msgr2=0x7f056c1054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.418+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 shutdown_connections 2024-04-05T23:31:34.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.418+0000 7f0572c00700 1 -- 172.21.15.16:0/1222819876 wait complete. 2024-04-05T23:31:34.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:34.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.634+0000 7fcd99200700 1 -- 172.21.15.16:0/3867015228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 msgr2=0x7fcd941011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.634+0000 7fcd99200700 1 --2- 172.21.15.16:0/3867015228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 0x7fcd941011d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcd840099c0 tx=0x7fcd84009cd0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.634+0000 7fcd99200700 1 -- 172.21.15.16:0/3867015228 shutdown_connections 2024-04-05T23:31:34.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.634+0000 7fcd99200700 1 --2- 172.21.15.16:0/3867015228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 0x7fcd941011d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.634+0000 7fcd99200700 1 --2- 172.21.15.16:0/3867015228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd94102940 0x7fcd941091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.634+0000 7fcd99200700 1 --2- 172.21.15.16:0/3867015228 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd94101fa0 0x7fcd94102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.634+0000 7fcd99200700 1 -- 172.21.15.16:0/3867015228 >> 172.21.15.16:0/3867015228 conn(0x7fcd940fc320 msgr2=0x7fcd940fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:34.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.634+0000 7fcd99200700 1 -- 172.21.15.16:0/3867015228 shutdown_connections 2024-04-05T23:31:34.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.634+0000 7fcd99200700 1 -- 172.21.15.16:0/3867015228 wait complete. 2024-04-05T23:31:34.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.635+0000 7fcd99200700 1 Processor -- start 2024-04-05T23:31:34.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.635+0000 7fcd99200700 1 -- start start 2024-04-05T23:31:34.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd99200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 0x7fcd94198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:34.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd99200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd94101fa0 0x7fcd941994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:34.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd99200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd94102940 0x7fcd941a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:34.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd99200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd941a0ac0 con 0x7fcd94100db0 2024-04-05T23:31:34.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd99200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd941a0c00 con 0x7fcd94102940 2024-04-05T23:31:34.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd99200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd941a0d90 con 0x7fcd94101fa0 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 0x7fcd94198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd94101fa0 0x7fcd941994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 0x7fcd94198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54866/0 (socket says 172.21.15.16:54866) 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91e00700 1 -- 172.21.15.16:0/421668561 learned_addr learned my addr 172.21.15.16:0/421668561 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd94101fa0 0x7fcd941994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55918/0 (socket says 172.21.15.16:55918) 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd92800700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd94102940 0x7fcd941a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91400700 1 -- 172.21.15.16:0/421668561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd94102940 msgr2=0x7fcd941a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91400700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd94102940 0x7fcd941a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91400700 1 -- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 msgr2=0x7fcd94198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91400700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 0x7fcd94198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91400700 1 -- 172.21.15.16:0/421668561 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd840095f0 con 0x7fcd94101fa0 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd92800700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd94102940 0x7fcd941a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.636+0000 7fcd91e00700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 0x7fcd94198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.637+0000 7fcd91400700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd94101fa0 0x7fcd941994d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd8000d950 tx=0x7fcd8000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:34.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.637+0000 7fcd8aa00700 1 -- 172.21.15.16:0/421668561 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd80009930 con 0x7fcd94101fa0 2024-04-05T23:31:34.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.637+0000 7fcd8aa00700 1 -- 172.21.15.16:0/421668561 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd80010460 con 0x7fcd94101fa0 2024-04-05T23:31:34.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.637+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd941a1070 con 0x7fcd94101fa0 2024-04-05T23:31:34.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.637+0000 7fcd8aa00700 1 -- 172.21.15.16:0/421668561 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8000b5d0 con 0x7fcd94101fa0 2024-04-05T23:31:34.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.637+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd941a1580 con 0x7fcd94101fa0 2024-04-05T23:31:34.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.638+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd94067b50 con 0x7fcd94101fa0 2024-04-05T23:31:34.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.640+0000 7fcd8aa00700 1 -- 172.21.15.16:0/421668561 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fcd800107d0 con 0x7fcd94101fa0 2024-04-05T23:31:34.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.641+0000 7fcd8aa00700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fcd780a1120 0x7fcd780a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:34.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.641+0000 7fcd8aa00700 1 -- 172.21.15.16:0/421668561 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcd800cc090 con 0x7fcd94101fa0 2024-04-05T23:31:34.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.641+0000 7fcd91e00700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fcd780a1120 0x7fcd780a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:34.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.642+0000 7fcd91e00700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fcd780a1120 0x7fcd780a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcd84003820 tx=0x7fcd840046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:34.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.645+0000 7fcd8aa00700 1 -- 172.21.15.16:0/421668561 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcd80095c20 con 0x7fcd94101fa0 2024-04-05T23:31:34.834 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:31:34 smithi130 systemd[1]: Starting Ceph mds.l for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:34.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:34 smithi130 ceph-mon[79417]: Deploying daemon mds.l on smithi130 2024-04-05T23:31:34.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:34 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4170008439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:31:34.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:34 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1222819876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:31:34.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.912+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd9410c7e0 con 0x7fcd94101fa0 2024-04-05T23:31:34.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.913+0000 7fcd8aa00700 1 -- 172.21.15.16:0/421668561 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fcd80099240 con 0x7fcd94101fa0 2024-04-05T23:31:34.906 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:34.906 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-05T23:31:34.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fcd780a1120 msgr2=0x7fcd780a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fcd780a1120 0x7fcd780a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcd84003820 tx=0x7fcd840046c0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd94101fa0 msgr2=0x7fcd941994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:34.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd94101fa0 0x7fcd941994d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd8000d950 tx=0x7fcd8000dc60 comp rx=0 tx=0).stop 2024-04-05T23:31:34.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 shutdown_connections 2024-04-05T23:31:34.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fcd780a1120 0x7fcd780a35e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd94100db0 0x7fcd94198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd94102940 0x7fcd941a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 --2- 172.21.15.16:0/421668561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd94101fa0 0x7fcd941994d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:34.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.915+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 >> 172.21.15.16:0/421668561 conn(0x7fcd940fc320 msgr2=0x7fcd94107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:34.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.916+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 shutdown_connections 2024-04-05T23:31:34.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:34.916+0000 7fcd99200700 1 -- 172.21.15.16:0/421668561 wait complete. 2024-04-05T23:31:34.911 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 1 2024-04-05T23:31:35.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:34 smithi016 ceph-mon[38248]: Deploying daemon mds.l on smithi130 2024-04-05T23:31:35.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:34 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4170008439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:31:35.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:34 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1222819876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:31:35.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:34 smithi027 ceph-mon[78597]: Deploying daemon mds.l on smithi130 2024-04-05T23:31:35.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:34 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4170008439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:31:35.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:34 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1222819876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:31:35.334 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mds[116936]: -- 172.21.15.130:0/2825357211 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5574cc3581a0 con 0x5574cc32f000 2024-04-05T23:31:35.334 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-l[116913]: starting mds.l at 2024-04-05T23:31:35.334 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:31:35 smithi130 bash[116898]: 63d7f1c2f274edbbcce5ab2084ad12acaf41b8984d07b8c9a5a4c551f0e0173a 2024-04-05T23:31:35.334 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:31:35 smithi130 systemd[1]: Started Ceph mds.l for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:35.856 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/421668561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:35.856 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] up:boot 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: fsmap 1 up:standby 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: Deploying daemon mds.k on smithi027 2024-04-05T23:31:35.857 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:35 smithi027 ceph-mon[78597]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:36.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/421668561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:36.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] up:boot 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: fsmap 1 up:standby 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: Deploying daemon mds.k on smithi027 2024-04-05T23:31:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:35 smithi016 ceph-mon[38248]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:36.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/421668561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:36.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] up:boot 2024-04-05T23:31:36.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: fsmap 1 up:standby 2024-04-05T23:31:36.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-05T23:31:36.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:36.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:36.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:36.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:36.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:36.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:36.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: Deploying daemon mds.k on smithi027 2024-04-05T23:31:36.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:35 smithi130 ceph-mon[79417]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:36.381 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:31:36 smithi027 systemd[1]: Starting Ceph mds.k for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:36.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:37.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.136+0000 7fbe47800700 1 -- 172.21.15.16:0/1258459141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe40073940 msgr2=0x7fbe40073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:37.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.136+0000 7fbe47800700 1 --2- 172.21.15.16:0/1258459141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe40073940 0x7fbe40073dc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009990 tx=0x7fbe2c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.136+0000 7fbe47800700 1 -- 172.21.15.16:0/1258459141 shutdown_connections 2024-04-05T23:31:37.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.136+0000 7fbe47800700 1 --2- 172.21.15.16:0/1258459141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe40108c70 0x7fbe4010b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.136+0000 7fbe47800700 1 --2- 172.21.15.16:0/1258459141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe40074f20 0x7fbe40073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.136+0000 7fbe47800700 1 --2- 172.21.15.16:0/1258459141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe40073940 0x7fbe40073dc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.136+0000 7fbe47800700 1 -- 172.21.15.16:0/1258459141 >> 172.21.15.16:0/1258459141 conn(0x7fbe400fc320 msgr2=0x7fbe400fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:37.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.137+0000 7fbe47800700 1 -- 172.21.15.16:0/1258459141 shutdown_connections 2024-04-05T23:31:37.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.137+0000 7fbe47800700 1 -- 172.21.15.16:0/1258459141 wait complete. 2024-04-05T23:31:37.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe47800700 1 Processor -- start 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe47800700 1 -- start start 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe47800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe40073940 0x7fbe40198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe47800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe40074f20 0x7fbe401994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe47800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe40108c70 0x7fbe401a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe47800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe401a0ad0 con 0x7fbe40073940 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe47800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe401a0c10 con 0x7fbe40074f20 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe47800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe401a0da0 con 0x7fbe40108c70 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe44e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe40073940 0x7fbe40198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.138+0000 7fbe44e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe40073940 0x7fbe40198fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54888/0 (socket says 172.21.15.16:54888) 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe44e00700 1 -- 172.21.15.16:0/318979767 learned_addr learned my addr 172.21.15.16:0/318979767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3fe00700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe40074f20 0x7fbe401994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe45800700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe40108c70 0x7fbe401a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:37.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3fe00700 1 -- 172.21.15.16:0/318979767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe40108c70 msgr2=0x7fbe401a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3fe00700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe40108c70 0x7fbe401a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3fe00700 1 -- 172.21.15.16:0/318979767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe40073940 msgr2=0x7fbe40198fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3fe00700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe40073940 0x7fbe40198fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3fe00700 1 -- 172.21.15.16:0/318979767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c0095f0 con 0x7fbe40074f20 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3fe00700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe40074f20 0x7fbe401994e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00b680 tx=0x7fbe2c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3d600700 1 -- 172.21.15.16:0/318979767 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c01d070 con 0x7fbe40074f20 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe401a1020 con 0x7fbe40074f20 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3d600700 1 -- 172.21.15.16:0/318979767 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe2c021410 con 0x7fbe40074f20 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.139+0000 7fbe3d600700 1 -- 172.21.15.16:0/318979767 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c017650 con 0x7fbe40074f20 2024-04-05T23:31:37.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.140+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe401a1450 con 0x7fbe40074f20 2024-04-05T23:31:37.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.141+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe40107a30 con 0x7fbe40074f20 2024-04-05T23:31:37.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.142+0000 7fbe3d600700 1 -- 172.21.15.16:0/318979767 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fbe2c021580 con 0x7fbe40074f20 2024-04-05T23:31:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.144+0000 7fbe3d600700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe280a1120 0x7fbe280a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.144+0000 7fbe3d600700 1 -- 172.21.15.16:0/318979767 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbe2c0cd8b0 con 0x7fbe40074f20 2024-04-05T23:31:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.144+0000 7fbe44e00700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe280a1120 0x7fbe280a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.145+0000 7fbe44e00700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe280a1120 0x7fbe280a35e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009d20 tx=0x7fbe34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.148+0000 7fbe3d600700 1 -- 172.21.15.16:0/318979767 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbe2c0974d0 con 0x7fbe40074f20 2024-04-05T23:31:37.216 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:31:36 smithi027 bash[115571]: 71dcf5522f8e0dde834f88374bd1a6d0ffa3807a484a84677cf33de2f609ac1f 2024-04-05T23:31:37.216 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:31:36 smithi027 systemd[1]: Started Ceph mds.k for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:37.217 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:31:36 smithi027 ceph-mds[115609]: -- 172.21.15.27:0/1541489881 <== mon.2 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f0685fc1a0 con 0x55f0685d2c00 2024-04-05T23:31:37.217 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:31:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-k[115586]: starting mds.k at 2024-04-05T23:31:37.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.434+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe40067b50 con 0x7fbe40074f20 2024-04-05T23:31:37.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.434+0000 7fbe3d600700 1 -- 172.21.15.16:0/318979767 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2014 (secure 0 0 0) 0x7fbe2c09aaf0 con 0x7fbe40074f20 2024-04-05T23:31:37.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:37.427 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-04-05T23:31:37.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe280a1120 msgr2=0x7fbe280a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:37.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe280a1120 0x7fbe280a35e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009d20 tx=0x7fbe34009260 comp rx=0 tx=0).stop 2024-04-05T23:31:37.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe40074f20 msgr2=0x7fbe401994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:37.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe40074f20 0x7fbe401994e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00b680 tx=0x7fbe2c004730 comp rx=0 tx=0).stop 2024-04-05T23:31:37.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 shutdown_connections 2024-04-05T23:31:37.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe280a1120 0x7fbe280a35e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe40073940 0x7fbe40198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe40108c70 0x7fbe401a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 --2- 172.21.15.16:0/318979767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe40074f20 0x7fbe401994e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:37.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 >> 172.21.15.16:0/318979767 conn(0x7fbe400fc320 msgr2=0x7fbe40104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:37.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 shutdown_connections 2024-04-05T23:31:37.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:37.437+0000 7fbe47800700 1 -- 172.21.15.16:0/318979767 wait complete. 2024-04-05T23:31:37.431 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2024-04-05T23:31:38.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:38.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:38.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:38.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:38.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:38.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:38.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: Deploying daemon mds.j on smithi016 2024-04-05T23:31:38.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.27:6832/1368666940,v1:172.21.15.27:6833/1368666940] up:boot 2024-04-05T23:31:38.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: fsmap 2 up:standby 2024-04-05T23:31:38.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-05T23:31:38.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:38.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:37 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/318979767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: Deploying daemon mds.j on smithi016 2024-04-05T23:31:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.27:6832/1368666940,v1:172.21.15.27:6833/1368666940] up:boot 2024-04-05T23:31:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: fsmap 2 up:standby 2024-04-05T23:31:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-05T23:31:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:37 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/318979767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:38.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: Deploying daemon mds.j on smithi016 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.27:6832/1368666940,v1:172.21.15.27:6833/1368666940] up:boot 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: fsmap 2 up:standby 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:38.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:37 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/318979767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:38.622 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 05 23:31:38 smithi016 systemd[1]: Starting Ceph mds.j for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:39.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:39.512 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 05 23:31:39 smithi016 bash[89044]: 63f238f7a5c936d070c4031c5a96769f38fe738da6979e56feec128eee5f03e0 2024-04-05T23:31:39.512 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 05 23:31:39 smithi016 systemd[1]: Started Ceph mds.j for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:39.512 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 05 23:31:39 smithi016 ceph-mds[89082]: -- 172.21.15.16:0/1858940716 <== mon.2 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5584097701a0 con 0x558409748c00 2024-04-05T23:31:39.513 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 05 23:31:39 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-j[89059]: starting mds.j at 2024-04-05T23:31:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.684+0000 7fba4cc00700 1 -- 172.21.15.16:0/3703450281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba481041b0 msgr2=0x7fba481065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.684+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3703450281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba481041b0 0x7fba481065a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fba34009990 tx=0x7fba34009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.684+0000 7fba4cc00700 1 -- 172.21.15.16:0/3703450281 shutdown_connections 2024-04-05T23:31:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.684+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3703450281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba48106ae0 0x7fba48108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.684+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3703450281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba48101880 0x7fba48103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.684+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3703450281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba481041b0 0x7fba481065a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.684+0000 7fba4cc00700 1 -- 172.21.15.16:0/3703450281 >> 172.21.15.16:0/3703450281 conn(0x7fba480fb230 msgr2=0x7fba480fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.684+0000 7fba4cc00700 1 -- 172.21.15.16:0/3703450281 shutdown_connections 2024-04-05T23:31:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.684+0000 7fba4cc00700 1 -- 172.21.15.16:0/3703450281 wait complete. 2024-04-05T23:31:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.685+0000 7fba4cc00700 1 Processor -- start 2024-04-05T23:31:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba4cc00700 1 -- start start 2024-04-05T23:31:39.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba4cc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba48101880 0x7fba48198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:39.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba4cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba481041b0 0x7fba481994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba4cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba48106ae0 0x7fba481a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba4cc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba481a0ad0 con 0x7fba48106ae0 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba4cc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba481a0c10 con 0x7fba481041b0 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba4cc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba481a0da0 con 0x7fba48101880 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba45e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba48101880 0x7fba48198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba45e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba48101880 0x7fba48198fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55968/0 (socket says 172.21.15.16:55968) 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba45e00700 1 -- 172.21.15.16:0/3310293169 learned_addr learned my addr 172.21.15.16:0/3310293169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba45400700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba481041b0 0x7fba481994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.686+0000 7fba46800700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba48106ae0 0x7fba481a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba45400700 1 -- 172.21.15.16:0/3310293169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba48101880 msgr2=0x7fba48198fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba45400700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba48101880 0x7fba48198fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba45400700 1 -- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba48106ae0 msgr2=0x7fba481a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba45400700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba48106ae0 0x7fba481a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba45400700 1 -- 172.21.15.16:0/3310293169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba340095f0 con 0x7fba481041b0 2024-04-05T23:31:39.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba45e00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba48101880 0x7fba48198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:31:39.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba46800700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba48106ae0 0x7fba481a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:31:39.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba45400700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba481041b0 0x7fba481994e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba34000c00 tx=0x7fba34005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:39.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba3ea00700 1 -- 172.21.15.16:0/3310293169 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3401d070 con 0x7fba481041b0 2024-04-05T23:31:39.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba481a1020 con 0x7fba481041b0 2024-04-05T23:31:39.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba3ea00700 1 -- 172.21.15.16:0/3310293169 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba34021410 con 0x7fba481041b0 2024-04-05T23:31:39.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.687+0000 7fba3ea00700 1 -- 172.21.15.16:0/3310293169 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34017650 con 0x7fba481041b0 2024-04-05T23:31:39.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.688+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba481a1450 con 0x7fba481041b0 2024-04-05T23:31:39.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.689+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba48108680 con 0x7fba481041b0 2024-04-05T23:31:39.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.691+0000 7fba3ea00700 1 -- 172.21.15.16:0/3310293169 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fba34021580 con 0x7fba481041b0 2024-04-05T23:31:39.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.692+0000 7fba3ea00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fba2c0a1120 0x7fba2c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:39.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.692+0000 7fba3ea00700 1 -- 172.21.15.16:0/3310293169 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fba340cd750 con 0x7fba481041b0 2024-04-05T23:31:39.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.693+0000 7fba45e00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fba2c0a1120 0x7fba2c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:39.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.693+0000 7fba45e00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fba2c0a1120 0x7fba2c0a35e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba380104e0 tx=0x7fba38007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:39.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.696+0000 7fba3ea00700 1 -- 172.21.15.16:0/3310293169 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fba34097370 con 0x7fba481041b0 2024-04-05T23:31:39.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.994+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba48067b50 con 0x7fba481041b0 2024-04-05T23:31:39.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.995+0000 7fba3ea00700 1 -- 172.21.15.16:0/3310293169 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2771 (secure 0 0 0) 0x7fba3409a990 con 0x7fba481041b0 2024-04-05T23:31:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:39.988 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fba2c0a1120 msgr2=0x7fba2c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fba2c0a1120 0x7fba2c0a35e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba380104e0 tx=0x7fba38007480 comp rx=0 tx=0).stop 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba481041b0 msgr2=0x7fba481994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba481041b0 0x7fba481994e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba34000c00 tx=0x7fba34005740 comp rx=0 tx=0).stop 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 shutdown_connections 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fba2c0a1120 0x7fba2c0a35e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba48106ae0 0x7fba481a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba48101880 0x7fba48198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 --2- 172.21.15.16:0/3310293169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba481041b0 0x7fba481994e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 >> 172.21.15.16:0/3310293169 conn(0x7fba480fb230 msgr2=0x7fba48101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 shutdown_connections 2024-04-05T23:31:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:39.998+0000 7fba4cc00700 1 -- 172.21.15.16:0/3310293169 wait complete. 2024-04-05T23:31:39.992 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 4 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.16:6834/1930674650,v1:172.21.15.16:6835/1930674650] up:boot 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: fsmap 3 up:standby 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: Deploying daemon mds.d on smithi016 2024-04-05T23:31:40.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:40.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:40 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3310293169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:40.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:40.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:40.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.16:6834/1930674650,v1:172.21.15.16:6835/1930674650] up:boot 2024-04-05T23:31:40.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: fsmap 3 up:standby 2024-04-05T23:31:40.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-05T23:31:40.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:40.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:40.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:40.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:40.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: Deploying daemon mds.d on smithi016 2024-04-05T23:31:40.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:40.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:40 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3310293169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:40.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:40.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:40.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.16:6834/1930674650,v1:172.21.15.16:6835/1930674650] up:boot 2024-04-05T23:31:40.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: fsmap 3 up:standby 2024-04-05T23:31:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-05T23:31:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: Deploying daemon mds.d on smithi016 2024-04-05T23:31:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:40.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:40 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3310293169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:41.333 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:31:41 smithi016 systemd[1]: Starting Ceph mds.d for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:42.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:42.074 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:31:41 smithi016 bash[89432]: d979cec7310279fd38ac39cc66e9786fb3accbc49332c8becb94b57edc03e2cc 2024-04-05T23:31:42.074 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:31:41 smithi016 systemd[1]: Started Ceph mds.d for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:42.074 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:31:41 smithi016 ceph-mds[89470]: -- 172.21.15.16:0/2047682179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557fa02901a0 con 0x557fa0268800 2024-04-05T23:31:42.074 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:31:41 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-d[89447]: starting mds.d at 2024-04-05T23:31:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.226+0000 7f6149e00700 1 -- 172.21.15.16:0/4070923097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6144102960 msgr2=0x7f6144109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.226+0000 7f6149e00700 1 --2- 172.21.15.16:0/4070923097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6144102960 0x7f6144109210 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f613c00b0d0 tx=0x7f613c00b3e0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.226+0000 7f6149e00700 1 -- 172.21.15.16:0/4070923097 shutdown_connections 2024-04-05T23:31:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.226+0000 7f6149e00700 1 --2- 172.21.15.16:0/4070923097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6144102960 0x7f6144109210 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.226+0000 7f6149e00700 1 --2- 172.21.15.16:0/4070923097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6144100dd0 0x7f61441011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.226+0000 7f6149e00700 1 --2- 172.21.15.16:0/4070923097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6144101fc0 0x7f6144102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.226+0000 7f6149e00700 1 -- 172.21.15.16:0/4070923097 >> 172.21.15.16:0/4070923097 conn(0x7f61440fc340 msgr2=0x7f61440fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.226+0000 7f6149e00700 1 -- 172.21.15.16:0/4070923097 shutdown_connections 2024-04-05T23:31:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.226+0000 7f6149e00700 1 -- 172.21.15.16:0/4070923097 wait complete. 2024-04-05T23:31:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.227+0000 7f6149e00700 1 Processor -- start 2024-04-05T23:31:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.227+0000 7f6149e00700 1 -- start start 2024-04-05T23:31:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.227+0000 7f6149e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6144100dd0 0x7f6144199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.227+0000 7f6149e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6144101fc0 0x7f6144199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.227+0000 7f6149e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6144102960 0x7f61441a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.227+0000 7f6149e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61441a0b80 con 0x7f6144101fc0 2024-04-05T23:31:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.227+0000 7f6149e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61441a0cc0 con 0x7f6144102960 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.227+0000 7f6149e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61441a0e50 con 0x7f6144100dd0 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6142000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6144101fc0 0x7f6144199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6143400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6144102960 0x7f61441a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6142000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6144101fc0 0x7f6144199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45318/0 (socket says 172.21.15.16:45318) 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6142000700 1 -- 172.21.15.16:0/2208104877 learned_addr learned my addr 172.21.15.16:0/2208104877 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6142a00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6144100dd0 0x7f6144199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6143400700 1 -- 172.21.15.16:0/2208104877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6144100dd0 msgr2=0x7f6144199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6143400700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6144100dd0 0x7f6144199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6143400700 1 -- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6144101fc0 msgr2=0x7f6144199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6143400700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6144101fc0 0x7f6144199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6143400700 1 -- 172.21.15.16:0/2208104877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c009d00 con 0x7f6144102960 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6142a00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6144100dd0 0x7f6144199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:31:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6142000700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6144101fc0 0x7f6144199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:31:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6143400700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6144102960 0x7f61441a05b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f613c007b60 tx=0x7f613c008e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.228+0000 7f6148a00700 1 -- 172.21.15.16:0/2208104877 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c00e040 con 0x7f6144102960 2024-04-05T23:31:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.229+0000 7f6148a00700 1 -- 172.21.15.16:0/2208104877 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f613c005070 con 0x7f6144102960 2024-04-05T23:31:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.229+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61441a10d0 con 0x7f6144102960 2024-04-05T23:31:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.229+0000 7f6148a00700 1 -- 172.21.15.16:0/2208104877 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c01d6d0 con 0x7f6144102960 2024-04-05T23:31:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.229+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61441a14f0 con 0x7f6144102960 2024-04-05T23:31:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.230+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6144067b50 con 0x7f6144102960 2024-04-05T23:31:42.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.231+0000 7f6148a00700 1 -- 172.21.15.16:0/2208104877 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f613c0051e0 con 0x7f6144102960 2024-04-05T23:31:42.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.233+0000 7f6148a00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f61280a11a0 0x7f61280a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:42.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.233+0000 7f6148a00700 1 -- 172.21.15.16:0/2208104877 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f613c0cd350 con 0x7f6144102960 2024-04-05T23:31:42.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.233+0000 7f6142a00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f61280a11a0 0x7f61280a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:42.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.233+0000 7f6142a00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f61280a11a0 0x7f61280a3660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6144101e20 tx=0x7f6134009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.235+0000 7f6148a00700 1 -- 172.21.15.16:0/2208104877 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f613c096f70 con 0x7f6144102960 2024-04-05T23:31:42.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.500+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f614410c640 con 0x7f6144102960 2024-04-05T23:31:42.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.500+0000 7f6148a00700 1 -- 172.21.15.16:0/2208104877 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3528 (secure 0 0 0) 0x7f613c09a590 con 0x7f6144102960 2024-04-05T23:31:42.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:42.493 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[]} 2024-04-05T23:31:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f61280a11a0 msgr2=0x7f61280a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f61280a11a0 0x7f61280a3660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6144101e20 tx=0x7f6134009260 comp rx=0 tx=0).stop 2024-04-05T23:31:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6144102960 msgr2=0x7f61441a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6144102960 0x7f61441a05b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f613c007b60 tx=0x7f613c008e40 comp rx=0 tx=0).stop 2024-04-05T23:31:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 shutdown_connections 2024-04-05T23:31:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f61280a11a0 0x7f61280a3660 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6144102960 0x7f61441a05b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6144100dd0 0x7f6144199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 --2- 172.21.15.16:0/2208104877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6144101fc0 0x7f6144199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 >> 172.21.15.16:0/2208104877 conn(0x7f61440fc340 msgr2=0x7f6144073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 shutdown_connections 2024-04-05T23:31:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:42.503+0000 7f6149e00700 1 -- 172.21.15.16:0/2208104877 wait complete. 2024-04-05T23:31:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 5 2024-04-05T23:31:42.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:42 smithi027 ceph-mon[78597]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:42 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:42 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:42 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:42 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:42 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:42 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:42.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:42 smithi016 ceph-mon[38248]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:42.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:42 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:42.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:42 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:42.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:42 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:42.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:42 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:42.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:42 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:42.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:42 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:42 smithi130 ceph-mon[79417]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:42 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:42 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:42 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:42 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:42 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:42 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:43.639 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:31:43 smithi027 systemd[1]: Starting Ceph mds.e for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:43.639 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:43 smithi027 ceph-mon[78597]: Deploying daemon mds.e on smithi027 2024-04-05T23:31:43.640 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:43 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] up:boot 2024-04-05T23:31:43.640 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:43 smithi027 ceph-mon[78597]: fsmap 4 up:standby 2024-04-05T23:31:43.640 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:43 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-05T23:31:43.640 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:43 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2208104877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:43 smithi016 ceph-mon[38248]: Deploying daemon mds.e on smithi027 2024-04-05T23:31:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:43 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] up:boot 2024-04-05T23:31:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:43 smithi016 ceph-mon[38248]: fsmap 4 up:standby 2024-04-05T23:31:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:43 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-05T23:31:43.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2208104877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:43 smithi130 ceph-mon[79417]: Deploying daemon mds.e on smithi027 2024-04-05T23:31:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:43 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] up:boot 2024-04-05T23:31:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:43 smithi130 ceph-mon[79417]: fsmap 4 up:standby 2024-04-05T23:31:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:43 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-05T23:31:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:43 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2208104877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:44.216 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:31:43 smithi027 bash[115914]: a624edd6e0b50f85910c3b2973042886e7f9582e6e233c2bb3f8870c3c5148ff 2024-04-05T23:31:44.217 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:31:43 smithi027 systemd[1]: Started Ceph mds.e for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:44.217 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:31:43 smithi027 ceph-mds[115952]: -- 172.21.15.27:0/1394902047 <== mon.2 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559de20781a0 con 0x559de204ec00 2024-04-05T23:31:44.217 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:31:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-e[115929]: starting mds.e at 2024-04-05T23:31:44.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:44.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:44 smithi130 ceph-mon[79417]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:44.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:44 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:44.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:44 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:44.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:44 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:44.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:44 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:44.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:44 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:44.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:44 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:44.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:44 smithi027 ceph-mon[78597]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:44.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:44 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:44.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:44 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:44.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:44 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:44.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:44 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:44.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:44 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:44.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:44 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.735+0000 7f7c61c00700 1 -- 172.21.15.16:0/3178672163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1041b0 msgr2=0x7f7c5c1065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.735+0000 7f7c61c00700 1 --2- 172.21.15.16:0/3178672163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1041b0 0x7f7c5c1065a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009990 tx=0x7f7c48009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.735+0000 7f7c61c00700 1 -- 172.21.15.16:0/3178672163 shutdown_connections 2024-04-05T23:31:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.735+0000 7f7c61c00700 1 --2- 172.21.15.16:0/3178672163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c5c106ae0 0x7f7c5c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.735+0000 7f7c61c00700 1 --2- 172.21.15.16:0/3178672163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c5c101880 0x7f7c5c103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.735+0000 7f7c61c00700 1 --2- 172.21.15.16:0/3178672163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1041b0 0x7f7c5c1065a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.735+0000 7f7c61c00700 1 -- 172.21.15.16:0/3178672163 >> 172.21.15.16:0/3178672163 conn(0x7f7c5c0fb230 msgr2=0x7f7c5c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.735+0000 7f7c61c00700 1 -- 172.21.15.16:0/3178672163 shutdown_connections 2024-04-05T23:31:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.735+0000 7f7c61c00700 1 -- 172.21.15.16:0/3178672163 wait complete. 2024-04-05T23:31:44.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.736+0000 7f7c61c00700 1 Processor -- start 2024-04-05T23:31:44.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c61c00700 1 -- start start 2024-04-05T23:31:44.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c61c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c101880 0x7f7c5c1011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:44.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c61c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c5c1041b0 0x7f7c5c0ff830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:44.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c61c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c5c106ae0 0x7f7c5c0ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:44.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c61c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c100510 con 0x7f7c5c1041b0 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c61c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c100680 con 0x7f7c5c101880 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c61c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c1a0d90 con 0x7f7c5c106ae0 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c5b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c5c106ae0 0x7f7c5c0ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c5c1041b0 0x7f7c5c0ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c5b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c5c106ae0 0x7f7c5c0ffd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:50488/0 (socket says 172.21.15.16:50488) 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c5aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c101880 0x7f7c5c1011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.737+0000 7f7c5b400700 1 -- 172.21.15.16:0/1784893062 learned_addr learned my addr 172.21.15.16:0/1784893062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c5b400700 1 -- 172.21.15.16:0/1784893062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c101880 msgr2=0x7f7c5c1011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c5b400700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c101880 0x7f7c5c1011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c5b400700 1 -- 172.21.15.16:0/1784893062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c5c1041b0 msgr2=0x7f7c5c0ff830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c5b400700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c5c1041b0 0x7f7c5c0ff830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c5b400700 1 -- 172.21.15.16:0/1784893062 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c480095f0 con 0x7f7c5c106ae0 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c5aa00700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c101880 0x7f7c5c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:31:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c5b400700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c5c106ae0 0x7f7c5c0ffd70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400ef30 tx=0x7f7c5400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:44.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c53400700 1 -- 172.21.15.16:0/1784893062 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54018070 con 0x7f7c5c106ae0 2024-04-05T23:31:44.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c53400700 1 -- 172.21.15.16:0/1784893062 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c5400f040 con 0x7f7c5c106ae0 2024-04-05T23:31:44.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c1a0fc0 con 0x7f7c5c106ae0 2024-04-05T23:31:44.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c53400700 1 -- 172.21.15.16:0/1784893062 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5400cd00 con 0x7f7c5c106ae0 2024-04-05T23:31:44.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.738+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c1a13e0 con 0x7f7c5c106ae0 2024-04-05T23:31:44.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.739+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c0fcdb0 con 0x7f7c5c106ae0 2024-04-05T23:31:44.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.741+0000 7f7c53400700 1 -- 172.21.15.16:0/1784893062 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f7c54003a00 con 0x7f7c5c106ae0 2024-04-05T23:31:44.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.743+0000 7f7c53400700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7c400a1120 0x7f7c400a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:44.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.743+0000 7f7c53400700 1 -- 172.21.15.16:0/1784893062 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7c540cc130 con 0x7f7c5c106ae0 2024-04-05T23:31:44.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.743+0000 7f7c5aa00700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7c400a1120 0x7f7c400a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:44.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.743+0000 7f7c5aa00700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7c400a1120 0x7f7c400a35e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c011490 tx=0x7f7c4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:44.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:44.744+0000 7f7c53400700 1 -- 172.21.15.16:0/1784893062 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7c54095d50 con 0x7f7c5c106ae0 2024-04-05T23:31:44.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:44 smithi016 ceph-mon[38248]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:44.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:44 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:44.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:44 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:44.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:44 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:44.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:44 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:44.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:44 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:44.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:44 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:45.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.008+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c5c067b50 con 0x7f7c5c106ae0 2024-04-05T23:31:45.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.009+0000 7f7c53400700 1 -- 172.21.15.16:0/1784893062 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4282 (secure 0 0 0) 0x7f7c54099370 con 0x7f7c5c106ae0 2024-04-05T23:31:45.002 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:45.002 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2024-04-05T23:31:45.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7c400a1120 msgr2=0x7f7c400a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:45.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7c400a1120 0x7f7c400a35e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c011490 tx=0x7f7c4c009260 comp rx=0 tx=0).stop 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c5c106ae0 msgr2=0x7f7c5c0ffd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c5c106ae0 0x7f7c5c0ffd70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400ef30 tx=0x7f7c5400c3a0 comp rx=0 tx=0).stop 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 shutdown_connections 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7c400a1120 0x7f7c400a35e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c5c106ae0 0x7f7c5c0ffd70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c101880 0x7f7c5c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 --2- 172.21.15.16:0/1784893062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c5c1041b0 0x7f7c5c0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.011+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 >> 172.21.15.16:0/1784893062 conn(0x7f7c5c0fb230 msgr2=0x7f7c5c102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.012+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 shutdown_connections 2024-04-05T23:31:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:45.012+0000 7f7c61c00700 1 -- 172.21.15.16:0/1784893062 wait complete. 2024-04-05T23:31:45.006 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2024-04-05T23:31:45.152 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:31:45 smithi130 systemd[1]: Starting Ceph mds.f for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:45.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:45 smithi027 ceph-mon[78597]: Deploying daemon mds.f on smithi130 2024-04-05T23:31:45.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:45 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.27:6834/686908192,v1:172.21.15.27:6835/686908192] up:boot 2024-04-05T23:31:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:45 smithi027 ceph-mon[78597]: fsmap 5 up:standby 2024-04-05T23:31:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:45 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-05T23:31:45.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:45 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1784893062' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:45.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:45 smithi016 ceph-mon[38248]: Deploying daemon mds.f on smithi130 2024-04-05T23:31:45.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:45 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.27:6834/686908192,v1:172.21.15.27:6835/686908192] up:boot 2024-04-05T23:31:45.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:45 smithi016 ceph-mon[38248]: fsmap 5 up:standby 2024-04-05T23:31:45.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:45 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-05T23:31:45.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:45 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1784893062' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:45.834 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:31:45 smithi130 ceph-mds[117286]: -- 172.21.15.130:0/1329261495 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563ec45521a0 con 0x563ec4529000 2024-04-05T23:31:45.834 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:31:45 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: starting mds.f at 2024-04-05T23:31:45.834 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:31:45 smithi130 bash[117248]: c09010468ca6a52fc91d9bff8e31d5044383d59c5dcff4d0774bde0dbc4783fb 2024-04-05T23:31:45.834 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:31:45 smithi130 systemd[1]: Started Ceph mds.f for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:45.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:45 smithi130 ceph-mon[79417]: Deploying daemon mds.f on smithi130 2024-04-05T23:31:45.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:45 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.27:6834/686908192,v1:172.21.15.27:6835/686908192] up:boot 2024-04-05T23:31:45.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:45 smithi130 ceph-mon[79417]: fsmap 5 up:standby 2024-04-05T23:31:45.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:45 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-05T23:31:45.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:45 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1784893062' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:46.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:46 smithi130 ceph-mon[79417]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:46.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:46 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:46.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:46 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:46.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:46 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:46.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:46 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:46.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:46 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:46.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:46 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:46 smithi027 ceph-mon[78597]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:46 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:46 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:46 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:46 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:46 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:46 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:46.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:46 smithi016 ceph-mon[38248]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:46.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:46 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:46.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:46 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:46.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:46 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:46.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:46 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:46.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:46 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:46.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:46 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:47.025 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:47.071 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:31:46 smithi130 systemd[1]: Starting Ceph mds.c for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:47.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.242+0000 7fd32f600700 1 -- 172.21.15.16:0/3407285547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328073940 msgr2=0x7fd328073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:47.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.242+0000 7fd32f600700 1 --2- 172.21.15.16:0/3407285547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328073940 0x7fd328073dc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd314009990 tx=0x7fd314009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.242+0000 7fd32f600700 1 -- 172.21.15.16:0/3407285547 shutdown_connections 2024-04-05T23:31:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.242+0000 7fd32f600700 1 --2- 172.21.15.16:0/3407285547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd328108c70 0x7fd32810b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.242+0000 7fd32f600700 1 --2- 172.21.15.16:0/3407285547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328074f20 0x7fd328073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.242+0000 7fd32f600700 1 --2- 172.21.15.16:0/3407285547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328073940 0x7fd328073dc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.242+0000 7fd32f600700 1 -- 172.21.15.16:0/3407285547 >> 172.21.15.16:0/3407285547 conn(0x7fd3280fc320 msgr2=0x7fd3280fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.242+0000 7fd32f600700 1 -- 172.21.15.16:0/3407285547 shutdown_connections 2024-04-05T23:31:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.242+0000 7fd32f600700 1 -- 172.21.15.16:0/3407285547 wait complete. 2024-04-05T23:31:47.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.243+0000 7fd32f600700 1 Processor -- start 2024-04-05T23:31:47.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.243+0000 7fd32f600700 1 -- start start 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32f600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd328073940 0x7fd328198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32f600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328074f20 0x7fd3281994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32f600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328108c70 0x7fd3281a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32f600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3281a0a90 con 0x7fd328073940 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32f600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3281a0bd0 con 0x7fd328074f20 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32f600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3281a0d60 con 0x7fd328108c70 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd328073940 0x7fd328198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32d600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328108c70 0x7fd3281a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd328073940 0x7fd328198f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45350/0 (socket says 172.21.15.16:45350) 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32d600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328108c70 0x7fd3281a04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:50506/0 (socket says 172.21.15.16:50506) 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32cc00700 1 -- 172.21.15.16:0/2710506137 learned_addr learned my addr 172.21.15.16:0/2710506137 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd327e00700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328074f20 0x7fd3281994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32cc00700 1 -- 172.21.15.16:0/2710506137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328108c70 msgr2=0x7fd3281a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32cc00700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328108c70 0x7fd3281a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32cc00700 1 -- 172.21.15.16:0/2710506137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328074f20 msgr2=0x7fd3281994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32cc00700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328074f20 0x7fd3281994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32cc00700 1 -- 172.21.15.16:0/2710506137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3140095f0 con 0x7fd328073940 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.244+0000 7fd32d600700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328108c70 0x7fd3281a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.245+0000 7fd32cc00700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd328073940 0x7fd328198f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00e9a0 tx=0x7fd31c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.245+0000 7fd325600700 1 -- 172.21.15.16:0/2710506137 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c009790 con 0x7fd328073940 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.245+0000 7fd325600700 1 -- 172.21.15.16:0/2710506137 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd31c010430 con 0x7fd328073940 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.245+0000 7fd325600700 1 -- 172.21.15.16:0/2710506137 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c018680 con 0x7fd328073940 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.245+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3281a0fe0 con 0x7fd328073940 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.245+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3281a13b0 con 0x7fd328073940 2024-04-05T23:31:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.246+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd328107a30 con 0x7fd328073940 2024-04-05T23:31:47.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.249+0000 7fd325600700 1 -- 172.21.15.16:0/2710506137 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fd31c0105a0 con 0x7fd328073940 2024-04-05T23:31:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.250+0000 7fd325600700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd3100a1120 0x7fd3100a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.250+0000 7fd325600700 1 -- 172.21.15.16:0/2710506137 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd31c014070 con 0x7fd328073940 2024-04-05T23:31:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.250+0000 7fd327e00700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd3100a1120 0x7fd3100a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:47.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.251+0000 7fd327e00700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd3100a1120 0x7fd3100a35e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd314000c00 tx=0x7fd314005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:47.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.254+0000 7fd325600700 1 -- 172.21.15.16:0/2710506137 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd31c052ab0 con 0x7fd328073940 2024-04-05T23:31:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.525+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd328067b50 con 0x7fd328073940 2024-04-05T23:31:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.526+0000 7fd325600700 1 -- 172.21.15.16:0/2710506137 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5802 (secure 0 0 0) 0x7fd31c096240 con 0x7fd328073940 2024-04-05T23:31:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:47.519 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2024-04-05T23:31:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd3100a1120 msgr2=0x7fd3100a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd3100a1120 0x7fd3100a35e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd314000c00 tx=0x7fd314005fb0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd328073940 msgr2=0x7fd328198f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd328073940 0x7fd328198f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00e9a0 tx=0x7fd31c00ecb0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 shutdown_connections 2024-04-05T23:31:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd3100a1120 0x7fd3100a35e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd328073940 0x7fd328198f60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd328108c70 0x7fd3281a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 --2- 172.21.15.16:0/2710506137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328074f20 0x7fd3281994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 >> 172.21.15.16:0/2710506137 conn(0x7fd3280fc320 msgr2=0x7fd328104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 shutdown_connections 2024-04-05T23:31:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:47.528+0000 7fd32f600700 1 -- 172.21.15.16:0/2710506137 wait complete. 2024-04-05T23:31:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-05T23:31:47.584 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mds[117631]: -- 172.21.15.130:0/3555019666 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e6878a41a0 con 0x55e68787ac00 2024-04-05T23:31:47.584 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: starting mds.c at 2024-04-05T23:31:47.584 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:31:47 smithi130 bash[117593]: 8f86326b12d694df8f71f2581237c82d04931d1b9d798e0cc55adc15576cc4e9 2024-04-05T23:31:47.584 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:31:47 smithi130 systemd[1]: Started Ceph mds.c for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:47.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: Deploying daemon mds.c on smithi130 2024-04-05T23:31:47.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6834/3125278540,v1:172.21.15.130:6835/3125278540] up:boot 2024-04-05T23:31:47.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: fsmap 6 up:standby 2024-04-05T23:31:47.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:31:47.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:47.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:47.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:47.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:47.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:47.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:47 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:47.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: Deploying daemon mds.c on smithi130 2024-04-05T23:31:47.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6834/3125278540,v1:172.21.15.130:6835/3125278540] up:boot 2024-04-05T23:31:47.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: fsmap 6 up:standby 2024-04-05T23:31:47.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:31:47.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:47.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:47.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:47.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:47.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:47.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:47 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:47.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: Deploying daemon mds.c on smithi130 2024-04-05T23:31:47.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6834/3125278540,v1:172.21.15.130:6835/3125278540] up:boot 2024-04-05T23:31:47.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: fsmap 6 up:standby 2024-04-05T23:31:47.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:31:47.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:47.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:47.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:47.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:47.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:47.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:47 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:48.662 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:48 smithi027 ceph-mon[78597]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:48.662 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:48 smithi027 ceph-mon[78597]: Deploying daemon mds.h on smithi027 2024-04-05T23:31:48.663 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:48 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] up:boot 2024-04-05T23:31:48.663 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:48 smithi027 ceph-mon[78597]: fsmap 7 up:standby 2024-04-05T23:31:48.663 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:48 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-05T23:31:48.663 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:48 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2710506137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:48.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:48 smithi016 ceph-mon[38248]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:48.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:48 smithi016 ceph-mon[38248]: Deploying daemon mds.h on smithi027 2024-04-05T23:31:48.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:48 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] up:boot 2024-04-05T23:31:48.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:48 smithi016 ceph-mon[38248]: fsmap 7 up:standby 2024-04-05T23:31:48.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:48 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-05T23:31:48.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:48 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2710506137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:48.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:48 smithi130 ceph-mon[79417]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:48.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:48 smithi130 ceph-mon[79417]: Deploying daemon mds.h on smithi027 2024-04-05T23:31:48.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:48 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] up:boot 2024-04-05T23:31:48.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:48 smithi130 ceph-mon[79417]: fsmap 7 up:standby 2024-04-05T23:31:48.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:48 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-05T23:31:48.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:48 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2710506137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:48.966 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:31:48 smithi027 systemd[1]: Starting Ceph mds.h for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:49.466 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:31:49 smithi027 ceph-mds[116292]: -- 172.21.15.27:0/2267571845 <== mon.2 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bd5e2921a0 con 0x55bd5e268800 2024-04-05T23:31:49.467 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:31:49 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-h[116269]: starting mds.h at 2024-04-05T23:31:49.467 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:31:49 smithi027 bash[116254]: 5225dfab15ea1e45fd319cc2cc5044d4d70a2b6f5dfea786cf0e9332fa27bd42 2024-04-05T23:31:49.467 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:31:49 smithi027 systemd[1]: Started Ceph mds.h for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:49.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.758+0000 7f90ad000700 1 -- 172.21.15.16:0/400606618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8104150 msgr2=0x7f90a8106540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.758+0000 7f90ad000700 1 --2- 172.21.15.16:0/400606618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8104150 0x7f90a8106540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9094009990 tx=0x7f9094009ca0 comp rx=0 tx=0).stop 2024-04-05T23:31:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.758+0000 7f90ad000700 1 -- 172.21.15.16:0/400606618 shutdown_connections 2024-04-05T23:31:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.758+0000 7f90ad000700 1 --2- 172.21.15.16:0/400606618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90a8106a80 0x7f90a8108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.758+0000 7f90ad000700 1 --2- 172.21.15.16:0/400606618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90a8101820 0x7f90a8103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.758+0000 7f90ad000700 1 --2- 172.21.15.16:0/400606618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8104150 0x7f90a8106540 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.758+0000 7f90ad000700 1 -- 172.21.15.16:0/400606618 >> 172.21.15.16:0/400606618 conn(0x7f90a80fb230 msgr2=0x7f90a80fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.758+0000 7f90ad000700 1 -- 172.21.15.16:0/400606618 shutdown_connections 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.758+0000 7f90ad000700 1 -- 172.21.15.16:0/400606618 wait complete. 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.759+0000 7f90ad000700 1 Processor -- start 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.759+0000 7f90ad000700 1 -- start start 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90ad000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8101820 0x7f90a8196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90ad000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90a8104150 0x7f90a81972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90ad000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90a8106a80 0x7f90a819e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90ad000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a819e850 con 0x7f90a8101820 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90ad000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a819e9c0 con 0x7f90a8106a80 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90ad000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a819eb80 con 0x7f90a8104150 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90a5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8101820 0x7f90a8196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90a5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90a8104150 0x7f90a81972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90a5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90a8104150 0x7f90a81972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:50516/0 (socket says 172.21.15.16:50516) 2024-04-05T23:31:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90a5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8101820 0x7f90a8196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45364/0 (socket says 172.21.15.16:45364) 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90a5400700 1 -- 172.21.15.16:0/1813514314 learned_addr learned my addr 172.21.15.16:0/1813514314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.760+0000 7f90a6800700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90a8106a80 0x7f90a819e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f90a5e00700 1 -- 172.21.15.16:0/1813514314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90a8104150 msgr2=0x7f90a81972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f90a5e00700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90a8104150 0x7f90a81972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f90a5e00700 1 -- 172.21.15.16:0/1813514314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90a8106a80 msgr2=0x7f90a819e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f90a5e00700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90a8106a80 0x7f90a819e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f90a5e00700 1 -- 172.21.15.16:0/1813514314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940095f0 con 0x7f90a8101820 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f90a5e00700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8101820 0x7f90a8196d60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f909800e9f0 tx=0x7f909800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f909ea00700 1 -- 172.21.15.16:0/1813514314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909800c9c0 con 0x7f90a8101820 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f909ea00700 1 -- 172.21.15.16:0/1813514314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9098010430 con 0x7f90a8101820 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f909ea00700 1 -- 172.21.15.16:0/1813514314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9098018680 con 0x7f90a8101820 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a819ee60 con 0x7f90a8101820 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.761+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a819f280 con 0x7f90a8101820 2024-04-05T23:31:49.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.762+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a80fcdb0 con 0x7f90a8101820 2024-04-05T23:31:49.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.764+0000 7f909ea00700 1 -- 172.21.15.16:0/1813514314 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f9098010860 con 0x7f90a8101820 2024-04-05T23:31:49.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.766+0000 7f909ea00700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f908c0a11a0 0x7f908c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:49.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.767+0000 7f909ea00700 1 -- 172.21.15.16:0/1813514314 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9098014070 con 0x7f90a8101820 2024-04-05T23:31:49.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.767+0000 7f90a5400700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f908c0a11a0 0x7f908c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:49.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.768+0000 7f90a5400700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f908c0a11a0 0x7f908c0a3660 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f909400b2e0 tx=0x7f909400b750 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:49.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:49.769+0000 7f909ea00700 1 -- 172.21.15.16:0/1813514314 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f90980962b0 con 0x7f90a8101820 2024-04-05T23:31:50.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.033+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a8067b50 con 0x7f90a8101820 2024-04-05T23:31:50.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.034+0000 7f909ea00700 1 -- 172.21.15.16:0/1813514314 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6556 (secure 0 0 0) 0x7f90980998d0 con 0x7f90a8101820 2024-04-05T23:31:50.027 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:50.027 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.036+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f908c0a11a0 msgr2=0x7f908c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f908c0a11a0 0x7f908c0a3660 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f909400b2e0 tx=0x7f909400b750 comp rx=0 tx=0).stop 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8101820 msgr2=0x7f90a8196d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8101820 0x7f90a8196d60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f909800e9f0 tx=0x7f909800ed00 comp rx=0 tx=0).stop 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 shutdown_connections 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f908c0a11a0 0x7f908c0a3660 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a8101820 0x7f90a8196d60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90a8106a80 0x7f90a819e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 --2- 172.21.15.16:0/1813514314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90a8104150 0x7f90a81972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 >> 172.21.15.16:0/1813514314 conn(0x7f90a80fb230 msgr2=0x7f90a81036f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:50.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 shutdown_connections 2024-04-05T23:31:50.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:50.037+0000 7f90ad000700 1 -- 172.21.15.16:0/1813514314 wait complete. 2024-04-05T23:31:50.031 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2024-04-05T23:31:50.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:50.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:50.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:50.260 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:50.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:50.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:50.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: Deploying daemon mds.i on smithi130 2024-04-05T23:31:50.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:50.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.27:6836/541968377,v1:172.21.15.27:6837/541968377] up:boot 2024-04-05T23:31:50.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: fsmap 8 up:standby 2024-04-05T23:31:50.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-05T23:31:50.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1813514314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:50.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: Deploying daemon mds.i on smithi130 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.27:6836/541968377,v1:172.21.15.27:6837/541968377] up:boot 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: fsmap 8 up:standby 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-05T23:31:50.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:50 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1813514314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:50.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: Deploying daemon mds.i on smithi130 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.27:6836/541968377,v1:172.21.15.27:6837/541968377] up:boot 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: fsmap 8 up:standby 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-05T23:31:50.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:50 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1813514314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:50.512 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:31:50 smithi130 systemd[1]: Starting Ceph mds.i for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:51.334 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-mds[117971]: -- 172.21.15.130:0/2283181995 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561b5324c1a0 con 0x561b53222800 2024-04-05T23:31:51.334 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:31:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-i[117948]: starting mds.i at 2024-04-05T23:31:51.334 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:31:50 smithi130 bash[117934]: 81b4772bd56808166c7707783efb29b72ec0876b82b3b52434fe7fe15c02f757 2024-04-05T23:31:51.334 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:31:50 smithi130 systemd[1]: Started Ceph mds.i for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:52.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:52.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: Deploying daemon mds.g on smithi016 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] up:boot 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: fsmap 9 up:standby 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-05T23:31:52.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:51 smithi016 ceph-mon[38248]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:52.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:52.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:52.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:52.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:52.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:52.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:52.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: Deploying daemon mds.g on smithi016 2024-04-05T23:31:52.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] up:boot 2024-04-05T23:31:52.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: fsmap 9 up:standby 2024-04-05T23:31:52.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-05T23:31:52.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:51 smithi027 ceph-mon[78597]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.262+0000 7f1013200700 1 -- 172.21.15.16:0/910774305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c102940 msgr2=0x7f100c1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.262+0000 7f1013200700 1 --2- 172.21.15.16:0/910774305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c102940 0x7f100c1091f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f100400b0d0 tx=0x7f100400b3e0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.262+0000 7f1013200700 1 -- 172.21.15.16:0/910774305 shutdown_connections 2024-04-05T23:31:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.262+0000 7f1013200700 1 --2- 172.21.15.16:0/910774305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c102940 0x7f100c1091f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.262+0000 7f1013200700 1 --2- 172.21.15.16:0/910774305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100c100db0 0x7f100c1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.262+0000 7f1013200700 1 --2- 172.21.15.16:0/910774305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f100c101fa0 0x7f100c102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.262+0000 7f1013200700 1 -- 172.21.15.16:0/910774305 >> 172.21.15.16:0/910774305 conn(0x7f100c0fc320 msgr2=0x7f100c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.262+0000 7f1013200700 1 -- 172.21.15.16:0/910774305 shutdown_connections 2024-04-05T23:31:52.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.262+0000 7f1013200700 1 -- 172.21.15.16:0/910774305 wait complete. 2024-04-05T23:31:52.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.263+0000 7f1013200700 1 Processor -- start 2024-04-05T23:31:52.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.263+0000 7f1013200700 1 -- start start 2024-04-05T23:31:52.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f1013200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f100c100db0 0x7f100c199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:52.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f1013200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c101fa0 0x7f100c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:52.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f1013200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100c102940 0x7f100c1a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:52.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f1013200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c1a0b40 con 0x7f100c101fa0 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f1013200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c1a0c80 con 0x7f100c102940 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f1013200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c1a0e10 con 0x7f100c100db0 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f100b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c101fa0 0x7f100c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f100b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c101fa0 0x7f100c1995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48914/0 (socket says 172.21.15.16:48914) 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f100b400700 1 -- 172.21.15.16:0/3265031587 learned_addr learned my addr 172.21.15.16:0/3265031587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f100be00700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f100c100db0 0x7f100c199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f1011200700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100c102940 0x7f100c1a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f100b400700 1 -- 172.21.15.16:0/3265031587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f100c100db0 msgr2=0x7f100c199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f100b400700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f100c100db0 0x7f100c199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f100b400700 1 -- 172.21.15.16:0/3265031587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100c102940 msgr2=0x7f100c1a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f100b400700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100c102940 0x7f100c1a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.264+0000 7f100b400700 1 -- 172.21.15.16:0/3265031587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1004009d00 con 0x7f100c101fa0 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.265+0000 7f100b400700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c101fa0 0x7f100c1995a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0ff800d950 tx=0x7f0ff800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.265+0000 7f1008c00700 1 -- 172.21.15.16:0/3265031587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff800f890 con 0x7f100c101fa0 2024-04-05T23:31:52.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.265+0000 7f1008c00700 1 -- 172.21.15.16:0/3265031587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ff8010460 con 0x7f100c101fa0 2024-04-05T23:31:52.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.265+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100c1a10f0 con 0x7f100c101fa0 2024-04-05T23:31:52.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.265+0000 7f1008c00700 1 -- 172.21.15.16:0/3265031587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff800e610 con 0x7f100c101fa0 2024-04-05T23:31:52.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.265+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c1a1600 con 0x7f100c101fa0 2024-04-05T23:31:52.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.266+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100c067b50 con 0x7f100c101fa0 2024-04-05T23:31:52.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.268+0000 7f1008c00700 1 -- 172.21.15.16:0/3265031587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f0ff80108d0 con 0x7f100c101fa0 2024-04-05T23:31:52.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.269+0000 7f1008c00700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0ff00a1120 0x7f0ff00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:52.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.269+0000 7f1008c00700 1 -- 172.21.15.16:0/3265031587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0ff80cc2a0 con 0x7f100c101fa0 2024-04-05T23:31:52.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.269+0000 7f100be00700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0ff00a1120 0x7f0ff00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:52.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.270+0000 7f100be00700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0ff00a1120 0x7f0ff00a35e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f100c101e00 tx=0x7f0ffc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:52.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.273+0000 7f1008c00700 1 -- 172.21.15.16:0/3265031587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0ff8095e30 con 0x7f100c101fa0 2024-04-05T23:31:52.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:52.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:52.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:52.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:52.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:52.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:52.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: Deploying daemon mds.g on smithi016 2024-04-05T23:31:52.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] up:boot 2024-04-05T23:31:52.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: fsmap 9 up:standby 2024-04-05T23:31:52.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-05T23:31:52.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:51 smithi130 ceph-mon[79417]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.560+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f100c10c620 con 0x7f100c101fa0 2024-04-05T23:31:52.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.561+0000 7f1008c00700 1 -- 172.21.15.16:0/3265031587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7318 (secure 0 0 0) 0x7f0ff8099450 con 0x7f100c101fa0 2024-04-05T23:31:52.554 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:52.554 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.563+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0ff00a1120 msgr2=0x7f0ff00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.563+0000 7f1013200700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0ff00a1120 0x7f0ff00a35e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f100c101e00 tx=0x7f0ffc009260 comp rx=0 tx=0).stop 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c101fa0 msgr2=0x7f100c1995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c101fa0 0x7f100c1995a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0ff800d950 tx=0x7f0ff800dc60 comp rx=0 tx=0).stop 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 shutdown_connections 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0ff00a1120 0x7f0ff00a35e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100c102940 0x7f100c1a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f100c100db0 0x7f100c199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 --2- 172.21.15.16:0/3265031587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c101fa0 0x7f100c1995a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 >> 172.21.15.16:0/3265031587 conn(0x7f100c0fc320 msgr2=0x7f100c073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 shutdown_connections 2024-04-05T23:31:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:52.564+0000 7f1013200700 1 -- 172.21.15.16:0/3265031587 wait complete. 2024-04-05T23:31:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 10 2024-04-05T23:31:53.189 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 05 23:31:52 smithi016 systemd[1]: Starting Ceph mds.g for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:53.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:53 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3265031587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:53.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:53 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3265031587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:53.582 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 05 23:31:53 smithi016 ceph-mds[90037]: -- 172.21.15.16:0/648151178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562d165881a0 con 0x562d16560c00 2024-04-05T23:31:53.583 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 05 23:31:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-g[90014]: starting mds.g at 2024-04-05T23:31:53.583 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 05 23:31:53 smithi016 bash[89999]: b7e9cca0077b543dbd8d0aaf048a689628dcd9a2a13886770b172b567a660ca8 2024-04-05T23:31:53.583 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 05 23:31:53 smithi016 systemd[1]: Started Ceph mds.g for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:53.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:53 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3265031587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:54.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:54 smithi027 ceph-mon[78597]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:54.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:54 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:54.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:54 smithi016 ceph-mon[38248]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:54.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:54.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:54.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:54.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:54.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:54.526 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:54 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:54 smithi130 ceph-mon[79417]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:54.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:54 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:54.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:54.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.805+0000 7ff3f0000700 1 -- 172.21.15.16:0/298729046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e8101fa0 msgr2=0x7ff3e8102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.805+0000 7ff3f0000700 1 --2- 172.21.15.16:0/298729046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e8101fa0 0x7ff3e8102400 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4009920 tx=0x7ff3d4009c30 comp rx=0 tx=0).stop 2024-04-05T23:31:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.805+0000 7ff3f0000700 1 -- 172.21.15.16:0/298729046 shutdown_connections 2024-04-05T23:31:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.805+0000 7ff3f0000700 1 --2- 172.21.15.16:0/298729046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e8102940 0x7ff3e81091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.805+0000 7ff3f0000700 1 --2- 172.21.15.16:0/298729046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e8100db0 0x7ff3e81011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.805+0000 7ff3f0000700 1 --2- 172.21.15.16:0/298729046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e8101fa0 0x7ff3e8102400 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.805+0000 7ff3f0000700 1 -- 172.21.15.16:0/298729046 >> 172.21.15.16:0/298729046 conn(0x7ff3e80fc320 msgr2=0x7ff3e80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.805+0000 7ff3f0000700 1 -- 172.21.15.16:0/298729046 shutdown_connections 2024-04-05T23:31:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.805+0000 7ff3f0000700 1 -- 172.21.15.16:0/298729046 wait complete. 2024-04-05T23:31:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.807+0000 7ff3f0000700 1 Processor -- start 2024-04-05T23:31:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.807+0000 7ff3f0000700 1 -- start start 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.807+0000 7ff3f0000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e8100db0 0x7ff3e8199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.807+0000 7ff3f0000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e8101fa0 0x7ff3e81995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.807+0000 7ff3f0000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e8102940 0x7ff3e81a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.807+0000 7ff3f0000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e81a0b50 con 0x7ff3e8102940 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.807+0000 7ff3f0000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e81a0c90 con 0x7ff3e8101fa0 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.807+0000 7ff3f0000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e81a0e20 con 0x7ff3e8100db0 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ecc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e8101fa0 0x7ff3e81995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ecc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e8101fa0 0x7ff3e81995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47636/0 (socket says 172.21.15.16:47636) 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ecc00700 1 -- 172.21.15.16:0/1112853886 learned_addr learned my addr 172.21.15.16:0/1112853886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ed600700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e8100db0 0x7ff3e8199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ee000700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e8102940 0x7ff3e81a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ed600700 1 -- 172.21.15.16:0/1112853886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e8101fa0 msgr2=0x7ff3e81995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ed600700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e8101fa0 0x7ff3e81995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ed600700 1 -- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e8102940 msgr2=0x7ff3e81a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ed600700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e8102940 0x7ff3e81a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ed600700 1 -- 172.21.15.16:0/1112853886 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc009520 con 0x7ff3e8100db0 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.808+0000 7ff3ecc00700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e8101fa0 0x7ff3e81995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.809+0000 7ff3ed600700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e8100db0 0x7ff3e8199070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00e9c0 tx=0x7ff3dc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.809+0000 7ff3e6000700 1 -- 172.21.15.16:0/1112853886 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc00cb70 con 0x7ff3e8100db0 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.809+0000 7ff3e6000700 1 -- 172.21.15.16:0/1112853886 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3dc00ee60 con 0x7ff3e8100db0 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.809+0000 7ff3e6000700 1 -- 172.21.15.16:0/1112853886 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc018680 con 0x7ff3e8100db0 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.809+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e0009d00 con 0x7ff3e8100db0 2024-04-05T23:31:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.809+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d4009520 con 0x7ff3e8100db0 2024-04-05T23:31:54.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.810+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e8067b50 con 0x7ff3e8100db0 2024-04-05T23:31:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.813+0000 7ff3e6000700 1 -- 172.21.15.16:0/1112853886 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7ff3dc010430 con 0x7ff3e8100db0 2024-04-05T23:31:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.814+0000 7ff3e6000700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff3d00a1120 0x7ff3d00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.815+0000 7ff3e6000700 1 -- 172.21.15.16:0/1112853886 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff3dc014070 con 0x7ff3e8100db0 2024-04-05T23:31:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.815+0000 7ff3ecc00700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff3d00a1120 0x7ff3d00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:54.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.815+0000 7ff3ecc00700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff3d00a1120 0x7ff3d00a35e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c010 tx=0x7ff3d4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:54.819+0000 7ff3e6000700 1 -- 172.21.15.16:0/1112853886 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff3dc096310 con 0x7ff3e8100db0 2024-04-05T23:31:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.104+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3e810c610 con 0x7ff3e8100db0 2024-04-05T23:31:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.105+0000 7ff3e6000700 1 -- 172.21.15.16:0/1112853886 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8073 (secure 0 0 0) 0x7ff3dc00ce60 con 0x7ff3e8100db0 2024-04-05T23:31:55.098 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:55.098 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-04-05T23:31:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff3d00a1120 msgr2=0x7ff3d00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff3d00a1120 0x7ff3d00a35e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c010 tx=0x7ff3d4019040 comp rx=0 tx=0).stop 2024-04-05T23:31:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e8100db0 msgr2=0x7ff3e8199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e8100db0 0x7ff3e8199070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00e9c0 tx=0x7ff3dc00ecd0 comp rx=0 tx=0).stop 2024-04-05T23:31:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 shutdown_connections 2024-04-05T23:31:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff3d00a1120 0x7ff3d00a35e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e8102940 0x7ff3e81a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e8100db0 0x7ff3e8199070 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 --2- 172.21.15.16:0/1112853886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e8101fa0 0x7ff3e81995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 >> 172.21.15.16:0/1112853886 conn(0x7ff3e80fc320 msgr2=0x7ff3e8073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 shutdown_connections 2024-04-05T23:31:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:55.108+0000 7ff3f0000700 1 -- 172.21.15.16:0/1112853886 wait complete. 2024-04-05T23:31:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 11 2024-04-05T23:31:55.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:55 smithi016 ceph-mon[38248]: Deploying daemon mds.a on smithi016 2024-04-05T23:31:55.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:55 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.16:6838/226142538,v1:172.21.15.16:6839/226142538] up:boot 2024-04-05T23:31:55.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:55 smithi016 ceph-mon[38248]: fsmap 10 up:standby 2024-04-05T23:31:55.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:55 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-05T23:31:55.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:55 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1112853886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:55.362 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:31:55 smithi016 systemd[1]: Starting Ceph mds.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:55.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:55 smithi027 ceph-mon[78597]: Deploying daemon mds.a on smithi016 2024-04-05T23:31:55.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:55 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.16:6838/226142538,v1:172.21.15.16:6839/226142538] up:boot 2024-04-05T23:31:55.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:55 smithi027 ceph-mon[78597]: fsmap 10 up:standby 2024-04-05T23:31:55.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:55 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-05T23:31:55.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:55 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1112853886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:55.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:55 smithi130 ceph-mon[79417]: Deploying daemon mds.a on smithi016 2024-04-05T23:31:55.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:55 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.16:6838/226142538,v1:172.21.15.16:6839/226142538] up:boot 2024-04-05T23:31:55.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:55 smithi130 ceph-mon[79417]: fsmap 10 up:standby 2024-04-05T23:31:55.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:55 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-05T23:31:55.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:55 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1112853886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:56.202 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:31:55 smithi016 bash[90385]: 97565fff1df1a1e4724b9feb636fb1157305a773df3542b761640830797d6e99 2024-04-05T23:31:56.203 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:31:55 smithi016 systemd[1]: Started Ceph mds.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:56.203 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:31:55 smithi016 ceph-mds[90424]: -- 172.21.15.16:0/3798683509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55adb92a41a0 con 0x55adb927d000 2024-04-05T23:31:56.203 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:31:55 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-a[90400]: starting mds.a at 2024-04-05T23:31:56.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:56 smithi027 ceph-mon[78597]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:56.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:56.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:56.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:56.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:56.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:56.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:56 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:56.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:56 smithi016 ceph-mon[38248]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:56.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:56.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:56.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:56.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:56.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:56.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:56 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:56.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:56 smithi130 ceph-mon[79417]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:56.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:56.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:56.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:56.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T23:31:56.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T23:31:56.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:56 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:31:57.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.345+0000 7fe95c800700 1 -- 172.21.15.16:0/3295775271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954101fa0 msgr2=0x7fe954102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.345+0000 7fe95c800700 1 --2- 172.21.15.16:0/3295775271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954101fa0 0x7fe954102400 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe9440099c0 tx=0x7fe944009cd0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.345+0000 7fe95c800700 1 -- 172.21.15.16:0/3295775271 shutdown_connections 2024-04-05T23:31:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.345+0000 7fe95c800700 1 --2- 172.21.15.16:0/3295775271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe954102940 0x7fe9541091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.345+0000 7fe95c800700 1 --2- 172.21.15.16:0/3295775271 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe954100db0 0x7fe9541011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.345+0000 7fe95c800700 1 --2- 172.21.15.16:0/3295775271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954101fa0 0x7fe954102400 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.345+0000 7fe95c800700 1 -- 172.21.15.16:0/3295775271 >> 172.21.15.16:0/3295775271 conn(0x7fe9540fc320 msgr2=0x7fe9540fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.345+0000 7fe95c800700 1 -- 172.21.15.16:0/3295775271 shutdown_connections 2024-04-05T23:31:57.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.345+0000 7fe95c800700 1 -- 172.21.15.16:0/3295775271 wait complete. 2024-04-05T23:31:57.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.346+0000 7fe95c800700 1 Processor -- start 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.346+0000 7fe95c800700 1 -- start start 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.346+0000 7fe95c800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe954100db0 0x7fe954198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.346+0000 7fe95c800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954101fa0 0x7fe9541994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.346+0000 7fe95c800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe954102940 0x7fe9541a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.346+0000 7fe95c800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9541a0a70 con 0x7fe954101fa0 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.346+0000 7fe95c800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9541a0bb0 con 0x7fe954102940 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.346+0000 7fe95c800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9541a0d40 con 0x7fe954100db0 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe959e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe954100db0 0x7fe954198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe95a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe954102940 0x7fe9541a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe959e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe954100db0 0x7fe954198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43086/0 (socket says 172.21.15.16:43086) 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe959e00700 1 -- 172.21.15.16:0/509255764 learned_addr learned my addr 172.21.15.16:0/509255764 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe959400700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954101fa0 0x7fe9541994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe95a800700 1 -- 172.21.15.16:0/509255764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe954100db0 msgr2=0x7fe954198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe95a800700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe954100db0 0x7fe954198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe95a800700 1 -- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954101fa0 msgr2=0x7fe9541994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe95a800700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954101fa0 0x7fe9541994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe95a800700 1 -- 172.21.15.16:0/509255764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe950009c40 con 0x7fe954102940 2024-04-05T23:31:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.347+0000 7fe95a800700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe954102940 0x7fe9541a04a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe95000efb0 tx=0x7fe95000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.348+0000 7fe94ea00700 1 -- 172.21.15.16:0/509255764 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe950003a00 con 0x7fe954102940 2024-04-05T23:31:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.348+0000 7fe94ea00700 1 -- 172.21.15.16:0/509255764 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe95000f040 con 0x7fe954102940 2024-04-05T23:31:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.348+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9440095f0 con 0x7fe954102940 2024-04-05T23:31:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.348+0000 7fe94ea00700 1 -- 172.21.15.16:0/509255764 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe950004540 con 0x7fe954102940 2024-04-05T23:31:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.348+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9541a1360 con 0x7fe954102940 2024-04-05T23:31:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.349+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe954067b50 con 0x7fe954102940 2024-04-05T23:31:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.351+0000 7fe94ea00700 1 -- 172.21.15.16:0/509255764 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fe950003bf0 con 0x7fe954102940 2024-04-05T23:31:57.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.352+0000 7fe94ea00700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe93c0a11a0 0x7fe93c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.352+0000 7fe94ea00700 1 -- 172.21.15.16:0/509255764 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe9500cc200 con 0x7fe954102940 2024-04-05T23:31:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.352+0000 7fe959e00700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe93c0a11a0 0x7fe93c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.353+0000 7fe959e00700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe93c0a11a0 0x7fe93c0a3660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe948007cf0 tx=0x7fe948007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:57.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.355+0000 7fe94ea00700 1 -- 172.21.15.16:0/509255764 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe950095e20 con 0x7fe954102940 2024-04-05T23:31:57.466 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:31:57 smithi027 systemd[1]: Starting Ceph mds.b for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:31:57.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:57 smithi027 ceph-mon[78597]: Deploying daemon mds.b on smithi027 2024-04-05T23:31:57.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:57 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.16:6840/2579712996,v1:172.21.15.16:6841/2579712996] up:boot 2024-04-05T23:31:57.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:57 smithi027 ceph-mon[78597]: fsmap 11 up:standby 2024-04-05T23:31:57.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:57 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-05T23:31:57.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:57 smithi016 ceph-mon[38248]: Deploying daemon mds.b on smithi027 2024-04-05T23:31:57.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:57 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.16:6840/2579712996,v1:172.21.15.16:6841/2579712996] up:boot 2024-04-05T23:31:57.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:57 smithi016 ceph-mon[38248]: fsmap 11 up:standby 2024-04-05T23:31:57.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:57 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-05T23:31:57.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:57 smithi130 ceph-mon[79417]: Deploying daemon mds.b on smithi027 2024-04-05T23:31:57.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:57 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.16:6840/2579712996,v1:172.21.15.16:6841/2579712996] up:boot 2024-04-05T23:31:57.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:57 smithi130 ceph-mon[79417]: fsmap 11 up:standby 2024-04-05T23:31:57.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:57 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-05T23:31:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.617+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe95410c860 con 0x7fe954102940 2024-04-05T23:31:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.618+0000 7fe94ea00700 1 -- 172.21.15.16:0/509255764 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8831 (secure 0 0 0) 0x7fe950099440 con 0x7fe954102940 2024-04-05T23:31:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:31:57.611 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14529,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-04-05T23:31:57.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe93c0a11a0 msgr2=0x7fe93c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:57.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe93c0a11a0 0x7fe93c0a3660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe948007cf0 tx=0x7fe948007480 comp rx=0 tx=0).stop 2024-04-05T23:31:57.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe954102940 msgr2=0x7fe9541a04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe954102940 0x7fe9541a04a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe95000efb0 tx=0x7fe95000c3a0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 shutdown_connections 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe93c0a11a0 0x7fe93c0a3660 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe954102940 0x7fe9541a04a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe954100db0 0x7fe954198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 --2- 172.21.15.16:0/509255764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe954101fa0 0x7fe9541994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 >> 172.21.15.16:0/509255764 conn(0x7fe9540fc320 msgr2=0x7fe9541077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.620+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 shutdown_connections 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:57.621+0000 7fe95c800700 1 -- 172.21.15.16:0/509255764 wait complete. 2024-04-05T23:31:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 12 2024-04-05T23:31:57.966 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:31:57 smithi027 ceph-mds[116641]: -- 172.21.15.27:0/2009614546 <== mon.2 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560427df41a0 con 0x560427dcac00 2024-04-05T23:31:57.967 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:31:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-b[116618]: starting mds.b at 2024-04-05T23:31:57.967 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:31:57 smithi027 bash[116603]: 00bd87182a78f39f6c07ee156feaf82311450408eee10870a9d9865848f849aa 2024-04-05T23:31:57.967 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:31:57 smithi027 systemd[1]: Started Ceph mds.b for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:31:58.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:58 smithi027 ceph-mon[78597]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:58.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:58 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/509255764' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:58.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:58 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:58 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:58 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:58 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:58 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:58 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:58.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:58 smithi016 ceph-mon[38248]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:58.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:58 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/509255764' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:58.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:58 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:58.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:58 smithi130 ceph-mon[79417]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:31:58.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:58 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/509255764' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:31:58.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:58 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:58 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:58 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:58 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:58 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:58.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:58 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:31:59.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:59 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] up:boot 2024-04-05T23:31:59.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:59 smithi027 ceph-mon[78597]: fsmap 12 up:standby 2024-04-05T23:31:59.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:59 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-05T23:31:59.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:59 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:59 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:59 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:31:59 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:59 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] up:boot 2024-04-05T23:31:59.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:59 smithi016 ceph-mon[38248]: fsmap 12 up:standby 2024-04-05T23:31:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:59 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-05T23:31:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:59 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:59 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:59 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:31:59 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:59 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] up:boot 2024-04-05T23:31:59.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:59 smithi130 ceph-mon[79417]: fsmap 12 up:standby 2024-04-05T23:31:59.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:59 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-05T23:31:59.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:59 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:59 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:59 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:31:59 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:31:59.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:31:59.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.855+0000 7f6e66200700 1 -- 172.21.15.16:0/1872232996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60108c70 msgr2=0x7f6e6010b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:59.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.855+0000 7f6e66200700 1 --2- 172.21.15.16:0/1872232996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60108c70 0x7f6e6010b100 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6e5800b0a0 tx=0x7f6e5800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:31:59.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.855+0000 7f6e66200700 1 -- 172.21.15.16:0/1872232996 shutdown_connections 2024-04-05T23:31:59.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.855+0000 7f6e66200700 1 --2- 172.21.15.16:0/1872232996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60108c70 0x7f6e6010b100 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:59.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.855+0000 7f6e66200700 1 --2- 172.21.15.16:0/1872232996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e60074f20 0x7f6e60073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:59.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.855+0000 7f6e66200700 1 --2- 172.21.15.16:0/1872232996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e60073940 0x7f6e60073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:59.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.855+0000 7f6e66200700 1 -- 172.21.15.16:0/1872232996 >> 172.21.15.16:0/1872232996 conn(0x7f6e600fc320 msgr2=0x7f6e600fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:31:59.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.855+0000 7f6e66200700 1 -- 172.21.15.16:0/1872232996 shutdown_connections 2024-04-05T23:31:59.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.855+0000 7f6e66200700 1 -- 172.21.15.16:0/1872232996 wait complete. 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.856+0000 7f6e66200700 1 Processor -- start 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.856+0000 7f6e66200700 1 -- start start 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.856+0000 7f6e66200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60073940 0x7f6e60199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.856+0000 7f6e66200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e60074f20 0x7f6e601995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.856+0000 7f6e66200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e60108c70 0x7f6e601a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.856+0000 7f6e66200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e601a0b90 con 0x7f6e60073940 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.856+0000 7f6e66200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e601a0cd0 con 0x7f6e60074f20 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.856+0000 7f6e66200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e601a0e60 con 0x7f6e60108c70 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60073940 0x7f6e60199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:59.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60073940 0x7f6e60199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49028/0 (socket says 172.21.15.16:49028) 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5f400700 1 -- 172.21.15.16:0/1263317065 learned_addr learned my addr 172.21.15.16:0/1263317065 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5ea00700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e60074f20 0x7f6e601995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5fe00700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e60108c70 0x7f6e601a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5f400700 1 -- 172.21.15.16:0/1263317065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e60108c70 msgr2=0x7f6e601a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5f400700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e60108c70 0x7f6e601a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5f400700 1 -- 172.21.15.16:0/1263317065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e60074f20 msgr2=0x7f6e601995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5f400700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e60074f20 0x7f6e601995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5f400700 1 -- 172.21.15.16:0/1263317065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e58009d00 con 0x7f6e60073940 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5fe00700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e60108c70 0x7f6e601a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.857+0000 7f6e5f400700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60073940 0x7f6e60199060 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000e9a0 tx=0x7f6e5000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.858+0000 7f6e57e00700 1 -- 172.21.15.16:0/1263317065 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e50009790 con 0x7f6e60073940 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.858+0000 7f6e57e00700 1 -- 172.21.15.16:0/1263317065 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e5000cbd0 con 0x7f6e60073940 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.858+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e601a1140 con 0x7f6e60073940 2024-04-05T23:31:59.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.858+0000 7f6e57e00700 1 -- 172.21.15.16:0/1263317065 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e50018680 con 0x7f6e60073940 2024-04-05T23:31:59.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.858+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e601a16a0 con 0x7f6e60073940 2024-04-05T23:31:59.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.859+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e60107a30 con 0x7f6e60073940 2024-04-05T23:31:59.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.861+0000 7f6e57e00700 1 -- 172.21.15.16:0/1263317065 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f6e5000cd40 con 0x7f6e60073940 2024-04-05T23:31:59.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.863+0000 7f6e57e00700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6e440a11a0 0x7f6e440a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:31:59.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.863+0000 7f6e57e00700 1 -- 172.21.15.16:0/1263317065 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6e50014070 con 0x7f6e60073940 2024-04-05T23:31:59.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.863+0000 7f6e5ea00700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6e440a11a0 0x7f6e440a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:31:59.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.864+0000 7f6e5ea00700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6e440a11a0 0x7f6e440a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6e60074c40 tx=0x7f6e4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:31:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:31:59.866+0000 7f6e57e00700 1 -- 172.21.15.16:0/1263317065 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6e50096580 con 0x7f6e60073940 2024-04-05T23:32:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.146+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e60067b50 con 0x7f6e60073940 2024-04-05T23:32:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.147+0000 7f6e57e00700 1 -- 172.21.15.16:0/1263317065 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9586 (secure 0 0 0) 0x7f6e50099ba0 con 0x7f6e60073940 2024-04-05T23:32:00.140 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:00.140 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14529,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2024-04-05T23:32:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.149+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6e440a11a0 msgr2=0x7f6e440a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6e440a11a0 0x7f6e440a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6e60074c40 tx=0x7f6e4c009260 comp rx=0 tx=0).stop 2024-04-05T23:32:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60073940 msgr2=0x7f6e60199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60073940 0x7f6e60199060 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000e9a0 tx=0x7f6e5000ecb0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 shutdown_connections 2024-04-05T23:32:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6e440a11a0 0x7f6e440a3660 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e60073940 0x7f6e60199060 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e60108c70 0x7f6e601a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 --2- 172.21.15.16:0/1263317065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e60074f20 0x7f6e601995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 >> 172.21.15.16:0/1263317065 conn(0x7f6e600fc320 msgr2=0x7f6e60104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 shutdown_connections 2024-04-05T23:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.150+0000 7f6e66200700 1 -- 172.21.15.16:0/1263317065 wait complete. 2024-04-05T23:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2024-04-05T23:32:00.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:32:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.382+0000 7f62be400700 1 -- 172.21.15.16:0/2862717015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62b8104120 msgr2=0x7f62b8106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.382+0000 7f62be400700 1 --2- 172.21.15.16:0/2862717015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62b8104120 0x7f62b8106510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f62a40098f0 tx=0x7f62a4009c00 comp rx=0 tx=0).stop 2024-04-05T23:32:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.382+0000 7f62be400700 1 -- 172.21.15.16:0/2862717015 shutdown_connections 2024-04-05T23:32:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.382+0000 7f62be400700 1 --2- 172.21.15.16:0/2862717015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b81017f0 0x7f62b8103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.382+0000 7f62be400700 1 --2- 172.21.15.16:0/2862717015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8106a50 0x7f62b8108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.382+0000 7f62be400700 1 --2- 172.21.15.16:0/2862717015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62b8104120 0x7f62b8106510 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.382+0000 7f62be400700 1 -- 172.21.15.16:0/2862717015 >> 172.21.15.16:0/2862717015 conn(0x7f62b80fb200 msgr2=0x7f62b80fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:00.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.382+0000 7f62be400700 1 -- 172.21.15.16:0/2862717015 shutdown_connections 2024-04-05T23:32:00.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.382+0000 7f62be400700 1 -- 172.21.15.16:0/2862717015 wait complete. 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.383+0000 7f62be400700 1 Processor -- start 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.383+0000 7f62be400700 1 -- start start 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62be400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b81017f0 0x7f62b8196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62be400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8104120 0x7f62b81972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62be400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62b8106a50 0x7f62b819e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62be400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819e880 con 0x7f62b81017f0 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62be400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819e9f0 con 0x7f62b8106a50 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62be400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819ebb0 con 0x7f62b8104120 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62bd000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b81017f0 0x7f62b8196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62bd000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b81017f0 0x7f62b8196d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49036/0 (socket says 172.21.15.16:49036) 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62bd000700 1 -- 172.21.15.16:0/296779962 learned_addr learned my addr 172.21.15.16:0/296779962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62b7e00700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8104120 0x7f62b81972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62bda00700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62b8106a50 0x7f62b819e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62bd000700 1 -- 172.21.15.16:0/296779962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8104120 msgr2=0x7f62b81972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62bd000700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8104120 0x7f62b81972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62bd000700 1 -- 172.21.15.16:0/296779962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62b8106a50 msgr2=0x7f62b819e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62bd000700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62b8106a50 0x7f62b819e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.384+0000 7f62bd000700 1 -- 172.21.15.16:0/296779962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a4009520 con 0x7f62b81017f0 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.385+0000 7f62bd000700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b81017f0 0x7f62b8196d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00e9c0 tx=0x7f62ac00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.385+0000 7f62b5600700 1 -- 172.21.15.16:0/296779962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac004a30 con 0x7f62b81017f0 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.385+0000 7f62b5600700 1 -- 172.21.15.16:0/296779962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62ac00cc80 con 0x7f62b81017f0 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.385+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b819ee90 con 0x7f62b81017f0 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.385+0000 7f62b5600700 1 -- 172.21.15.16:0/296779962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac0054e0 con 0x7f62b81017f0 2024-04-05T23:32:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.385+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b819f2b0 con 0x7f62b81017f0 2024-04-05T23:32:00.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.386+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b80fcd80 con 0x7f62b81017f0 2024-04-05T23:32:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.388+0000 7f62b5600700 1 -- 172.21.15.16:0/296779962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f62ac004b90 con 0x7f62b81017f0 2024-04-05T23:32:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.389+0000 7f62b5600700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f62a00a1120 0x7f62a00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.390+0000 7f62b5600700 1 -- 172.21.15.16:0/296779962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f62ac014070 con 0x7f62b81017f0 2024-04-05T23:32:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.390+0000 7f62b7e00700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f62a00a1120 0x7f62a00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.390+0000 7f62b7e00700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f62a00a1120 0x7f62a00a35e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f62a400fa40 tx=0x7f62a4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.395+0000 7f62b5600700 1 -- 172.21.15.16:0/296779962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f62ac0962f0 con 0x7f62b81017f0 2024-04-05T23:32:00.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:00 smithi027 ceph-mon[78597]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:00 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:00 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:00 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:32:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:00 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:32:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:00 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:00 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:32:00.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:00 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1263317065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:32:00.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:00 smithi016 ceph-mon[38248]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:00.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:00 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:00 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:00 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:32:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:00 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:32:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:00 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:00 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:32:00.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:00 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1263317065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:32:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:00 smithi130 ceph-mon[79417]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:00 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:00 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:00 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:32:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:00 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:32:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:00 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:00 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:32:00.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:00 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1263317065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:32:00.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.623+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62b8067ae0 con 0x7f62b81017f0 2024-04-05T23:32:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.625+0000 7f62b5600700 1 -- 172.21.15.16:0/296779962 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+18387 (secure 0 0 0) 0x7f62ac099910 con 0x7f62b81017f0 2024-04-05T23:32:00.618 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:00.619 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":75,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:31:26.220652+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:32:00.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f62a00a1120 msgr2=0x7f62a00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f62a00a1120 0x7f62a00a35e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f62a400fa40 tx=0x7f62a4019040 comp rx=0 tx=0).stop 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b81017f0 msgr2=0x7f62b8196d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b81017f0 0x7f62b8196d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00e9c0 tx=0x7f62ac00ecd0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 shutdown_connections 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f62a00a1120 0x7f62a00a35e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b81017f0 0x7f62b8196d90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62b8106a50 0x7f62b819e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 --2- 172.21.15.16:0/296779962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8104120 0x7f62b81972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 >> 172.21.15.16:0/296779962 conn(0x7f62b80fb200 msgr2=0x7f62b8103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 shutdown_connections 2024-04-05T23:32:00.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.627+0000 7f62be400700 1 -- 172.21.15.16:0/296779962 wait complete. 2024-04-05T23:32:00.640 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:32:00.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:32:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.851+0000 7f532c000700 1 -- 172.21.15.16:0/316075072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5324074f20 msgr2=0x7f5324073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.851+0000 7f532c000700 1 --2- 172.21.15.16:0/316075072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5324074f20 0x7f5324073370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5318009990 tx=0x7f5318009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.851+0000 7f532c000700 1 -- 172.21.15.16:0/316075072 shutdown_connections 2024-04-05T23:32:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.851+0000 7f532c000700 1 --2- 172.21.15.16:0/316075072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5324108d70 0x7f532410b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.851+0000 7f532c000700 1 --2- 172.21.15.16:0/316075072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5324074f20 0x7f5324073370 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.851+0000 7f532c000700 1 --2- 172.21.15.16:0/316075072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324073940 0x7f5324073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.851+0000 7f532c000700 1 -- 172.21.15.16:0/316075072 >> 172.21.15.16:0/316075072 conn(0x7f53240fc440 msgr2=0x7f53240fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.852+0000 7f532c000700 1 -- 172.21.15.16:0/316075072 shutdown_connections 2024-04-05T23:32:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.852+0000 7f532c000700 1 -- 172.21.15.16:0/316075072 wait complete. 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f532c000700 1 Processor -- start 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f532c000700 1 -- start start 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f532c000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5324073940 0x7f53241990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f532c000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324074f20 0x7f53241995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f532c000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5324108d70 0x7f53241a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f532c000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53241a0b80 con 0x7f5324073940 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f532c000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53241a0cc0 con 0x7f5324074f20 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f532c000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53241a0e50 con 0x7f5324108d70 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f5328c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324074f20 0x7f53241995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f5329600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5324073940 0x7f53241990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f532a000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5324108d70 0x7f53241a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f5328c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324074f20 0x7f53241995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47738/0 (socket says 172.21.15.16:47738) 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.853+0000 7f5328c00700 1 -- 172.21.15.16:0/3851531545 learned_addr learned my addr 172.21.15.16:0/3851531545 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f5328c00700 1 -- 172.21.15.16:0/3851531545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5324108d70 msgr2=0x7f53241a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f5328c00700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5324108d70 0x7f53241a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f5328c00700 1 -- 172.21.15.16:0/3851531545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5324073940 msgr2=0x7f53241990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f5328c00700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5324073940 0x7f53241990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f5328c00700 1 -- 172.21.15.16:0/3851531545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5310009520 con 0x7f5324074f20 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f5328c00700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324074f20 0x7f53241995e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f531000ea50 tx=0x7f531000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f5322000700 1 -- 172.21.15.16:0/3851531545 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531000cbd0 con 0x7f5324074f20 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f5322000700 1 -- 172.21.15.16:0/3851531545 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f531000cd30 con 0x7f5324074f20 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53180095f0 con 0x7f5324074f20 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f5322000700 1 -- 172.21.15.16:0/3851531545 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310018680 con 0x7f5324074f20 2024-04-05T23:32:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.854+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53241a1500 con 0x7f5324074f20 2024-04-05T23:32:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.855+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5324107b30 con 0x7f5324074f20 2024-04-05T23:32:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.857+0000 7f5322000700 1 -- 172.21.15.16:0/3851531545 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f5310010430 con 0x7f5324074f20 2024-04-05T23:32:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.858+0000 7f5322000700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f530c0a11a0 0x7f530c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.859+0000 7f5322000700 1 -- 172.21.15.16:0/3851531545 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5310014070 con 0x7f5324074f20 2024-04-05T23:32:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.859+0000 7f5329600700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f530c0a11a0 0x7f530c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.859+0000 7f5329600700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f530c0a11a0 0x7f530c0a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f531800b2e0 tx=0x7f5318005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:00.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:00.862+0000 7f5322000700 1 -- 172.21.15.16:0/3851531545 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f53100964d0 con 0x7f5324074f20 2024-04-05T23:32:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.084+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f532410eb20 con 0x7f5324074f20 2024-04-05T23:32:01.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.084+0000 7f5322000700 1 -- 172.21.15.16:0/3851531545 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5310099af0 con 0x7f5324074f20 2024-04-05T23:32:01.077 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f530c0a11a0 msgr2=0x7f530c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f530c0a11a0 0x7f530c0a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f531800b2e0 tx=0x7f5318005fd0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324074f20 msgr2=0x7f53241995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324074f20 0x7f53241995e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f531000ea50 tx=0x7f531000ed60 comp rx=0 tx=0).stop 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 shutdown_connections 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f530c0a11a0 0x7f530c0a3660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5324073940 0x7f53241990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5324108d70 0x7f53241a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 --2- 172.21.15.16:0/3851531545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324074f20 0x7f53241995e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 >> 172.21.15.16:0/3851531545 conn(0x7f53240fc440 msgr2=0x7f5324104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 shutdown_connections 2024-04-05T23:32:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.087+0000 7f532c000700 1 -- 172.21.15.16:0/3851531545 wait complete. 2024-04-05T23:32:01.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-05T23:32:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.320+0000 7fc264400700 1 -- 172.21.15.16:0/4152730407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc25c074f20 msgr2=0x7fc25c073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.320+0000 7fc264400700 1 --2- 172.21.15.16:0/4152730407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc25c074f20 0x7fc25c073370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc250009990 tx=0x7fc250009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.320+0000 7fc264400700 1 -- 172.21.15.16:0/4152730407 shutdown_connections 2024-04-05T23:32:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.320+0000 7fc264400700 1 --2- 172.21.15.16:0/4152730407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc25c108c70 0x7fc25c10b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.320+0000 7fc264400700 1 --2- 172.21.15.16:0/4152730407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc25c074f20 0x7fc25c073370 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.320+0000 7fc264400700 1 --2- 172.21.15.16:0/4152730407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25c073940 0x7fc25c073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.320+0000 7fc264400700 1 -- 172.21.15.16:0/4152730407 >> 172.21.15.16:0/4152730407 conn(0x7fc25c0fc320 msgr2=0x7fc25c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.321+0000 7fc264400700 1 -- 172.21.15.16:0/4152730407 shutdown_connections 2024-04-05T23:32:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.321+0000 7fc264400700 1 -- 172.21.15.16:0/4152730407 wait complete. 2024-04-05T23:32:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.322+0000 7fc264400700 1 Processor -- start 2024-04-05T23:32:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.322+0000 7fc264400700 1 -- start start 2024-04-05T23:32:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc264400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25c073940 0x7fc25c198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc264400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc25c074f20 0x7fc25c1994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc264400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc25c108c70 0x7fc25c1a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc264400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c1a0aa0 con 0x7fc25c073940 2024-04-05T23:32:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc264400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c1a0be0 con 0x7fc25c074f20 2024-04-05T23:32:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc264400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c1a0d70 con 0x7fc25c108c70 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc261a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25c073940 0x7fc25c198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc261a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25c073940 0x7fc25c198f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49084/0 (socket says 172.21.15.16:49084) 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc261a00700 1 -- 172.21.15.16:0/1892167629 learned_addr learned my addr 172.21.15.16:0/1892167629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc261000700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc25c074f20 0x7fc25c1994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc262400700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc25c108c70 0x7fc25c1a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc261a00700 1 -- 172.21.15.16:0/1892167629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc25c108c70 msgr2=0x7fc25c1a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc261a00700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc25c108c70 0x7fc25c1a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc261a00700 1 -- 172.21.15.16:0/1892167629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc25c074f20 msgr2=0x7fc25c1994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc261a00700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc25c074f20 0x7fc25c1994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.323+0000 7fc261a00700 1 -- 172.21.15.16:0/1892167629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2500095f0 con 0x7fc25c073940 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.324+0000 7fc261a00700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25c073940 0x7fc25c198f70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc250000c00 tx=0x7fc250005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.324+0000 7fc25a000700 1 -- 172.21.15.16:0/1892167629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25001d070 con 0x7fc25c073940 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.324+0000 7fc25a000700 1 -- 172.21.15.16:0/1892167629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc250021410 con 0x7fc25c073940 2024-04-05T23:32:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.324+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25c1a0ff0 con 0x7fc25c073940 2024-04-05T23:32:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.324+0000 7fc25a000700 1 -- 172.21.15.16:0/1892167629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250017650 con 0x7fc25c073940 2024-04-05T23:32:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.324+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c1a13b0 con 0x7fc25c073940 2024-04-05T23:32:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.325+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25c107a30 con 0x7fc25c073940 2024-04-05T23:32:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.327+0000 7fc25a000700 1 -- 172.21.15.16:0/1892167629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fc2500217f0 con 0x7fc25c073940 2024-04-05T23:32:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.328+0000 7fc25a000700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2440a11a0 0x7fc2440a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.328+0000 7fc25a000700 1 -- 172.21.15.16:0/1892167629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc2500cd8e0 con 0x7fc25c073940 2024-04-05T23:32:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.329+0000 7fc261000700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2440a11a0 0x7fc2440a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:01.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.329+0000 7fc261000700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2440a11a0 0x7fc2440a3660 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc25c074c40 tx=0x7fc24800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.334+0000 7fc25a000700 1 -- 172.21.15.16:0/1892167629 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc250097500 con 0x7fc25c073940 2024-04-05T23:32:01.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:01 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/296779962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:01.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:01 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:01 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:01 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3851531545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:32:01.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:01 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:01 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:01 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/296779962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:01.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:01 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:01 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:01 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3851531545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:32:01.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:01 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:01 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.620+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc25c02cde0 con 0x7fc25c073940 2024-04-05T23:32:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.620+0000 7fc25a000700 1 -- 172.21.15.16:0/1892167629 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc25009ab20 con 0x7fc25c073940 2024-04-05T23:32:01.613 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:01.613 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-04-05T23:32:01.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:32:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-04-05T23:32:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-04-05T23:32:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:32:01.614 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-04-05T23:32:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2440a11a0 msgr2=0x7fc2440a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2440a11a0 0x7fc2440a3660 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc25c074c40 tx=0x7fc24800b410 comp rx=0 tx=0).stop 2024-04-05T23:32:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25c073940 msgr2=0x7fc25c198f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25c073940 0x7fc25c198f70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc250000c00 tx=0x7fc250005740 comp rx=0 tx=0).stop 2024-04-05T23:32:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 shutdown_connections 2024-04-05T23:32:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2440a11a0 0x7fc2440a3660 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25c073940 0x7fc25c198f70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc25c108c70 0x7fc25c1a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 --2- 172.21.15.16:0/1892167629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc25c074f20 0x7fc25c1994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 >> 172.21.15.16:0/1892167629 conn(0x7fc25c0fc320 msgr2=0x7fc25c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 shutdown_connections 2024-04-05T23:32:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.623+0000 7fc264400700 1 -- 172.21.15.16:0/1892167629 wait complete. 2024-04-05T23:32:01.643 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-05T23:32:01.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-05T23:32:01.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:01 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/296779962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:01.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:01 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:01 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:01 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3851531545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:32:01.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:01 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:01 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:01.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.866+0000 7fbdb5400700 1 -- 172.21.15.16:0/1188213438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0101920 msgr2=0x7fbdb0103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.866+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1188213438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0101920 0x7fbdb0103d10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbda0009990 tx=0x7fbda0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.866+0000 7fbdb5400700 1 -- 172.21.15.16:0/1188213438 shutdown_connections 2024-04-05T23:32:01.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.866+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1188213438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb0106b80 0x7fbdb0108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.866+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1188213438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0101920 0x7fbdb0103d10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.866+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1188213438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbdb0104250 0x7fbdb0106640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.866+0000 7fbdb5400700 1 -- 172.21.15.16:0/1188213438 >> 172.21.15.16:0/1188213438 conn(0x7fbdb00fb330 msgr2=0x7fbdb00fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:01.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.866+0000 7fbdb5400700 1 -- 172.21.15.16:0/1188213438 shutdown_connections 2024-04-05T23:32:01.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.866+0000 7fbdb5400700 1 -- 172.21.15.16:0/1188213438 wait complete. 2024-04-05T23:32:01.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.867+0000 7fbdb5400700 1 Processor -- start 2024-04-05T23:32:01.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdb5400700 1 -- start start 2024-04-05T23:32:01.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdb5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbdb0101920 0x7fbdb0199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:01.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdb5400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb0104250 0x7fbdb0199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdb5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0106b80 0x7fbdb01a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdb5400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb01a0b80 con 0x7fbdb0104250 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdb5400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb01a0cc0 con 0x7fbdb0106b80 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdb5400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb01a0e50 con 0x7fbdb0101920 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdade00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb0104250 0x7fbdb0199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdaf200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0106b80 0x7fbdb01a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdae800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbdb0101920 0x7fbdb0199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdade00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb0104250 0x7fbdb0199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49098/0 (socket says 172.21.15.16:49098) 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdade00700 1 -- 172.21.15.16:0/1934566704 learned_addr learned my addr 172.21.15.16:0/1934566704 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdaf200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0106b80 0x7fbdb01a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47784/0 (socket says 172.21.15.16:47784) 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.868+0000 7fbdae800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbdb0101920 0x7fbdb0199050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43186/0 (socket says 172.21.15.16:43186) 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbdaf200700 1 -- 172.21.15.16:0/1934566704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbdb0101920 msgr2=0x7fbdb0199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbdaf200700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbdb0101920 0x7fbdb0199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbdaf200700 1 -- 172.21.15.16:0/1934566704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb0104250 msgr2=0x7fbdb0199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbdaf200700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb0104250 0x7fbdb0199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbdaf200700 1 -- 172.21.15.16:0/1934566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda00095f0 con 0x7fbdb0106b80 2024-04-05T23:32:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbdaf200700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0106b80 0x7fbdb01a05b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbda800efc0 tx=0x7fbda800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbda7400700 1 -- 172.21.15.16:0/1934566704 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda8018070 con 0x7fbdb0106b80 2024-04-05T23:32:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb01a10d0 con 0x7fbdb0106b80 2024-04-05T23:32:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbda7400700 1 -- 172.21.15.16:0/1934566704 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbda800f040 con 0x7fbdb0106b80 2024-04-05T23:32:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.869+0000 7fbda7400700 1 -- 172.21.15.16:0/1934566704 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda800cd40 con 0x7fbdb0106b80 2024-04-05T23:32:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.870+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb01a1590 con 0x7fbdb0106b80 2024-04-05T23:32:01.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.871+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb0108750 con 0x7fbdb0106b80 2024-04-05T23:32:01.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.873+0000 7fbda7400700 1 -- 172.21.15.16:0/1934566704 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fbda8003a00 con 0x7fbdb0106b80 2024-04-05T23:32:01.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.874+0000 7fbda7400700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbd940a1120 0x7fbd940a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:01.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.874+0000 7fbda7400700 1 -- 172.21.15.16:0/1934566704 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbda80cc210 con 0x7fbdb0106b80 2024-04-05T23:32:01.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.874+0000 7fbdae800700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbd940a1120 0x7fbd940a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:01.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.875+0000 7fbdae800700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbd940a1120 0x7fbd940a35e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbda000b2e0 tx=0x7fbda0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:01.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:01.876+0000 7fbda7400700 1 -- 172.21.15.16:0/1934566704 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbda8095e30 con 0x7fbdb0106b80 2024-04-05T23:32:02.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.119+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbdb0067c00 con 0x7fbdb0106b80 2024-04-05T23:32:02.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:02 smithi027 ceph-mon[78597]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:02.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:02 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1892167629' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-05T23:32:02.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:02.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:02.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:32:02.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:32:02.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:02.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:02 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1934566704' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-05T23:32:02.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.981+0000 7fbda7400700 1 -- 172.21.15.16:0/1934566704 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v76) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbda8099450 con 0x7fbdb0106b80 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.986+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbd940a1120 msgr2=0x7fbd940a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.986+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbd940a1120 0x7fbd940a35e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbda000b2e0 tx=0x7fbda0005fd0 comp rx=0 tx=0).stop 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.986+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0106b80 msgr2=0x7fbdb01a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.986+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0106b80 0x7fbdb01a05b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbda800efc0 tx=0x7fbda800c3a0 comp rx=0 tx=0).stop 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.987+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 shutdown_connections 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.987+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbd940a1120 0x7fbd940a35e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.987+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdb0106b80 0x7fbdb01a05b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.987+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbdb0101920 0x7fbdb0199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.987+0000 7fbdb5400700 1 --2- 172.21.15.16:0/1934566704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb0104250 0x7fbdb0199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.987+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 >> 172.21.15.16:0/1934566704 conn(0x7fbdb00fb330 msgr2=0x7fbdb0101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.987+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 shutdown_connections 2024-04-05T23:32:02.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:02.987+0000 7fbdb5400700 1 -- 172.21.15.16:0/1934566704 wait complete. 2024-04-05T23:32:02.981 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-04-05T23:32:02.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:02 smithi016 ceph-mon[38248]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:02.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:02 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1892167629' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-05T23:32:02.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:02.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:02.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:32:02.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:32:02.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:02.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:02 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1934566704' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-05T23:32:03.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-05T23:32:03.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:02 smithi130 ceph-mon[79417]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:03.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:02 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1892167629' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-05T23:32:03.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:03.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:03.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:32:03.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:32:03.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:03.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:02 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1934566704' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-05T23:32:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.215+0000 7f11eec00700 1 -- 172.21.15.16:0/3658658694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8102050 msgr2=0x7f11e81024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.215+0000 7f11eec00700 1 --2- 172.21.15.16:0/3658658694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8102050 0x7f11e81024d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009990 tx=0x7f11d4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.215+0000 7f11eec00700 1 -- 172.21.15.16:0/3658658694 shutdown_connections 2024-04-05T23:32:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.215+0000 7f11eec00700 1 --2- 172.21.15.16:0/3658658694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8102a10 0x7f11e8109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.215+0000 7f11eec00700 1 --2- 172.21.15.16:0/3658658694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8100e70 0x7f11e8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.215+0000 7f11eec00700 1 --2- 172.21.15.16:0/3658658694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8102050 0x7f11e81024d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.215+0000 7f11eec00700 1 -- 172.21.15.16:0/3658658694 >> 172.21.15.16:0/3658658694 conn(0x7f11e80fc430 msgr2=0x7f11e80fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.215+0000 7f11eec00700 1 -- 172.21.15.16:0/3658658694 shutdown_connections 2024-04-05T23:32:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.215+0000 7f11eec00700 1 -- 172.21.15.16:0/3658658694 wait complete. 2024-04-05T23:32:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.216+0000 7f11eec00700 1 Processor -- start 2024-04-05T23:32:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.216+0000 7f11eec00700 1 -- start start 2024-04-05T23:32:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.216+0000 7f11eec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8100e70 0x7f11e8199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.216+0000 7f11eec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8102050 0x7f11e81995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.216+0000 7f11eec00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8102a10 0x7f11e81a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.216+0000 7f11eec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e81a0ba0 con 0x7f11e8100e70 2024-04-05T23:32:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.216+0000 7f11eec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e81a0ce0 con 0x7f11e8102050 2024-04-05T23:32:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.216+0000 7f11eec00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e81a0e70 con 0x7f11e8102a10 2024-04-05T23:32:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8100e70 0x7f11e8199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8100e70 0x7f11e8199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57062/0 (socket says 172.21.15.16:57062) 2024-04-05T23:32:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7e00700 1 -- 172.21.15.16:0/1654247106 learned_addr learned my addr 172.21.15.16:0/1654247106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7400700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8102050 0x7f11e81995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11ecc00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8102a10 0x7f11e81a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7e00700 1 -- 172.21.15.16:0/1654247106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8102a10 msgr2=0x7f11e81a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7e00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8102a10 0x7f11e81a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7e00700 1 -- 172.21.15.16:0/1654247106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8102050 msgr2=0x7f11e81995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7e00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8102050 0x7f11e81995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7e00700 1 -- 172.21.15.16:0/1654247106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40095f0 con 0x7f11e8100e70 2024-04-05T23:32:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e7e00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8100e70 0x7f11e8199070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f11d800e970 tx=0x7f11d800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.217+0000 7f11e4c00700 1 -- 172.21.15.16:0/1654247106 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8009790 con 0x7f11e8100e70 2024-04-05T23:32:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.218+0000 7f11e4c00700 1 -- 172.21.15.16:0/1654247106 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11d800c7e0 con 0x7f11e8100e70 2024-04-05T23:32:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.218+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e81a1150 con 0x7f11e8100e70 2024-04-05T23:32:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.218+0000 7f11e4c00700 1 -- 172.21.15.16:0/1654247106 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8003c70 con 0x7f11e8100e70 2024-04-05T23:32:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.218+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e81a1660 con 0x7f11e8100e70 2024-04-05T23:32:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.219+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e8067c30 con 0x7f11e8100e70 2024-04-05T23:32:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.221+0000 7f11e4c00700 1 -- 172.21.15.16:0/1654247106 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f11d800cc50 con 0x7f11e8100e70 2024-04-05T23:32:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.222+0000 7f11e4c00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f11cc0a1120 0x7f11cc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.222+0000 7f11e4c00700 1 -- 172.21.15.16:0/1654247106 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f11d8014070 con 0x7f11e8100e70 2024-04-05T23:32:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.223+0000 7f11e7400700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f11cc0a1120 0x7f11cc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.223+0000 7f11e7400700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f11cc0a1120 0x7f11cc0a35e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f11d4000c00 tx=0x7f11d4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.225+0000 7f11e4c00700 1 -- 172.21.15.16:0/1654247106 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f11d8096360 con 0x7f11e8100e70 2024-04-05T23:32:03.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.442+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f11e810c730 con 0x7f11e8100e70 2024-04-05T23:32:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.986+0000 7f11e4c00700 1 -- 172.21.15.16:0/1654247106 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v77) v1 ==== 170+0+0 (secure 0 0 0) 0x7f11d8099980 con 0x7f11e8100e70 2024-04-05T23:32:03.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.991+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f11cc0a1120 msgr2=0x7f11cc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.991+0000 7f11eec00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f11cc0a1120 0x7f11cc0a35e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f11d4000c00 tx=0x7f11d4005fb0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.991+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8100e70 msgr2=0x7f11e8199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.991+0000 7f11eec00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8100e70 0x7f11e8199070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f11d800e970 tx=0x7f11d800ec80 comp rx=0 tx=0).stop 2024-04-05T23:32:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.992+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 shutdown_connections 2024-04-05T23:32:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.992+0000 7f11eec00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f11cc0a1120 0x7f11cc0a35e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.992+0000 7f11eec00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11e8100e70 0x7f11e8199070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.992+0000 7f11eec00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8102a10 0x7f11e81a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.992+0000 7f11eec00700 1 --2- 172.21.15.16:0/1654247106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e8102050 0x7f11e81995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.992+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 >> 172.21.15.16:0/1654247106 conn(0x7f11e80fc430 msgr2=0x7f11e81078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.992+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 shutdown_connections 2024-04-05T23:32:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:03.993+0000 7f11eec00700 1 -- 172.21.15.16:0/1654247106 wait complete. 2024-04-05T23:32:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-04-05T23:32:04.014 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-05T23:32:04.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.224+0000 7ff895600700 1 -- 172.21.15.16:0/2083505604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff890102040 msgr2=0x7ff8901024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.224+0000 7ff895600700 1 --2- 172.21.15.16:0/2083505604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff890102040 0x7ff8901024c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009990 tx=0x7ff87c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.224+0000 7ff895600700 1 -- 172.21.15.16:0/2083505604 shutdown_connections 2024-04-05T23:32:04.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.224+0000 7ff895600700 1 --2- 172.21.15.16:0/2083505604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff890102a00 0x7ff890109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.224+0000 7ff895600700 1 --2- 172.21.15.16:0/2083505604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff890100e60 0x7ff890101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.224+0000 7ff895600700 1 --2- 172.21.15.16:0/2083505604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff890102040 0x7ff8901024c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.224+0000 7ff895600700 1 -- 172.21.15.16:0/2083505604 >> 172.21.15.16:0/2083505604 conn(0x7ff8900fc440 msgr2=0x7ff8900fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:04.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.224+0000 7ff895600700 1 -- 172.21.15.16:0/2083505604 shutdown_connections 2024-04-05T23:32:04.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.224+0000 7ff895600700 1 -- 172.21.15.16:0/2083505604 wait complete. 2024-04-05T23:32:04.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:03 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-05T23:32:04.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:03 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:04.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:03 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-05T23:32:04.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:03 smithi027 ceph-mon[78597]: osdmap e76: 12 total, 12 up, 12 in 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.225+0000 7ff895600700 1 Processor -- start 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.225+0000 7ff895600700 1 -- start start 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff895600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff890100e60 0x7ff8901990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff895600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff890102040 0x7ff890199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff895600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff890102a00 0x7ff8901a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff895600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8901a0bf0 con 0x7ff890100e60 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff895600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8901a0d30 con 0x7ff890102a00 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff895600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8901a0ec0 con 0x7ff890102040 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff88f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff890102a00 0x7ff8901a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:04.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff88f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff890102a00 0x7ff8901a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:58918/0 (socket says 172.21.15.16:58918) 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff88f400700 1 -- 172.21.15.16:0/3638624713 learned_addr learned my addr 172.21.15.16:0/3638624713 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff88e000700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff890102040 0x7ff890199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff88ea00700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff890100e60 0x7ff8901990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.226+0000 7ff88f400700 1 -- 172.21.15.16:0/3638624713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff890102040 msgr2=0x7ff890199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff88f400700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff890102040 0x7ff890199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff88f400700 1 -- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff890100e60 msgr2=0x7ff8901990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff88f400700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff890100e60 0x7ff8901990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff88f400700 1 -- 172.21.15.16:0/3638624713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0095f0 con 0x7ff890102a00 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff88e000700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff890102040 0x7ff890199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff88f400700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff890102a00 0x7ff8901a0620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff888007ee0 tx=0x7ff88800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff887400700 1 -- 172.21.15.16:0/3638624713 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888017070 con 0x7ff890102a00 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8901a11a0 con 0x7ff890102a00 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff887400700 1 -- 172.21.15.16:0/3638624713 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff88800f040 con 0x7ff890102a00 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff887400700 1 -- 172.21.15.16:0/3638624713 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888005550 con 0x7ff890102a00 2024-04-05T23:32:04.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.227+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8901a1700 con 0x7ff890102a00 2024-04-05T23:32:04.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.228+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff890067c30 con 0x7ff890102a00 2024-04-05T23:32:04.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.230+0000 7ff887400700 1 -- 172.21.15.16:0/3638624713 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7ff88800db20 con 0x7ff890102a00 2024-04-05T23:32:04.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.231+0000 7ff887400700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff8740a11a0 0x7ff8740a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:04.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.232+0000 7ff887400700 1 -- 172.21.15.16:0/3638624713 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7879+0+0 (secure 0 0 0) 0x7ff8880cc4d0 con 0x7ff890102a00 2024-04-05T23:32:04.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.232+0000 7ff88ea00700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff8740a11a0 0x7ff8740a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:04.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.232+0000 7ff88ea00700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff8740a11a0 0x7ff8740a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff890101bb0 tx=0x7ff880009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:04.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.234+0000 7ff887400700 1 -- 172.21.15.16:0/3638624713 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff888095d70 con 0x7ff890102a00 2024-04-05T23:32:04.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:03 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-05T23:32:04.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:03 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:04.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:03 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-05T23:32:04.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:03 smithi016 ceph-mon[38248]: osdmap e76: 12 total, 12 up, 12 in 2024-04-05T23:32:04.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:03 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-05T23:32:04.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:03 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:32:04.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:03 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-05T23:32:04.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:03 smithi130 ceph-mon[79417]: osdmap e76: 12 total, 12 up, 12 in 2024-04-05T23:32:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.509+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff89010c720 con 0x7ff890102a00 2024-04-05T23:32:04.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.519+0000 7ff887400700 1 -- 172.21.15.16:0/3638624713 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff888099390 con 0x7ff890102a00 2024-04-05T23:32:04.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.524+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff8740a11a0 msgr2=0x7ff8740a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.524+0000 7ff895600700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff8740a11a0 0x7ff8740a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff890101bb0 tx=0x7ff880009260 comp rx=0 tx=0).stop 2024-04-05T23:32:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.524+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff890102a00 msgr2=0x7ff8901a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.524+0000 7ff895600700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff890102a00 0x7ff8901a0620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff888007ee0 tx=0x7ff88800d3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.525+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 shutdown_connections 2024-04-05T23:32:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.525+0000 7ff895600700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff8740a11a0 0x7ff8740a3660 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.525+0000 7ff895600700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff890100e60 0x7ff8901990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.525+0000 7ff895600700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff890102a00 0x7ff8901a0620 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.525+0000 7ff895600700 1 --2- 172.21.15.16:0/3638624713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff890102040 0x7ff890199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.525+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 >> 172.21.15.16:0/3638624713 conn(0x7ff8900fc440 msgr2=0x7ff890073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.526+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 shutdown_connections 2024-04-05T23:32:04.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.526+0000 7ff895600700 1 -- 172.21.15.16:0/3638624713 wait complete. 2024-04-05T23:32:04.523 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2024-04-05T23:32:04.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:32:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.755+0000 7fb291c00700 1 -- 172.21.15.16:0/975836495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 msgr2=0x7fb28c109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.755+0000 7fb291c00700 1 --2- 172.21.15.16:0/975836495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 0x7fb28c109200 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb28400b0d0 tx=0x7fb28400b3e0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.755+0000 7fb291c00700 1 -- 172.21.15.16:0/975836495 shutdown_connections 2024-04-05T23:32:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.755+0000 7fb291c00700 1 --2- 172.21.15.16:0/975836495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 0x7fb28c109200 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.755+0000 7fb291c00700 1 --2- 172.21.15.16:0/975836495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb28c100dc0 0x7fb28c1011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.755+0000 7fb291c00700 1 --2- 172.21.15.16:0/975836495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb28c101fb0 0x7fb28c102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.755+0000 7fb291c00700 1 -- 172.21.15.16:0/975836495 >> 172.21.15.16:0/975836495 conn(0x7fb28c0fc350 msgr2=0x7fb28c0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.755+0000 7fb291c00700 1 -- 172.21.15.16:0/975836495 shutdown_connections 2024-04-05T23:32:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.755+0000 7fb291c00700 1 -- 172.21.15.16:0/975836495 wait complete. 2024-04-05T23:32:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.756+0000 7fb291c00700 1 Processor -- start 2024-04-05T23:32:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.756+0000 7fb291c00700 1 -- start start 2024-04-05T23:32:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb291c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb28c100dc0 0x7fb28c074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb291c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb28c101fb0 0x7fb28c0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb291c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 0x7fb28c073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb291c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c073fb0 con 0x7fb28c102950 2024-04-05T23:32:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb291c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c074120 con 0x7fb28c101fb0 2024-04-05T23:32:04.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb291c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c1a0eb0 con 0x7fb28c100dc0 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 0x7fb28c073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb28c101fb0 0x7fb28c0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb28c101fb0 0x7fb28c0732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:58938/0 (socket says 172.21.15.16:58938) 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28a000700 1 -- 172.21.15.16:0/3512579784 learned_addr learned my addr 172.21.15.16:0/3512579784 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 0x7fb28c073810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57114/0 (socket says 172.21.15.16:57114) 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28aa00700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb28c100dc0 0x7fb28c074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28a000700 1 -- 172.21.15.16:0/3512579784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb28c100dc0 msgr2=0x7fb28c074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28a000700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb28c100dc0 0x7fb28c074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28a000700 1 -- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 msgr2=0x7fb28c073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28a000700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 0x7fb28c073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28a000700 1 -- 172.21.15.16:0/3512579784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb284009d00 con 0x7fb28c101fb0 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.757+0000 7fb28b400700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 0x7fb28c073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.758+0000 7fb28a000700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb28c101fb0 0x7fb28c0732d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb27800e9a0 tx=0x7fb27800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:04.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.758+0000 7fb283400700 1 -- 172.21.15.16:0/3512579784 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278009790 con 0x7fb28c101fb0 2024-04-05T23:32:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.758+0000 7fb283400700 1 -- 172.21.15.16:0/3512579784 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb27800ce70 con 0x7fb28c101fb0 2024-04-05T23:32:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.758+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28c1a1140 con 0x7fb28c101fb0 2024-04-05T23:32:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.758+0000 7fb283400700 1 -- 172.21.15.16:0/3512579784 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278018420 con 0x7fb28c101fb0 2024-04-05T23:32:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.758+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28c1a1650 con 0x7fb28c101fb0 2024-04-05T23:32:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.759+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28c108210 con 0x7fb28c101fb0 2024-04-05T23:32:04.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.761+0000 7fb283400700 1 -- 172.21.15.16:0/3512579784 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fb2780106f0 con 0x7fb28c101fb0 2024-04-05T23:32:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.762+0000 7fb283400700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2700a1120 0x7fb2700a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.762+0000 7fb283400700 1 -- 172.21.15.16:0/3512579784 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb278014070 con 0x7fb28c101fb0 2024-04-05T23:32:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.762+0000 7fb28aa00700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2700a1120 0x7fb2700a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.763+0000 7fb28aa00700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2700a1120 0x7fb2700a35e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb28c101e10 tx=0x7fb27c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:04.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.766+0000 7fb283400700 1 -- 172.21.15.16:0/3512579784 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb278052ab0 con 0x7fb28c101fb0 2024-04-05T23:32:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.985+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb28c10c630 con 0x7fb28c101fb0 2024-04-05T23:32:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.987+0000 7fb283400700 1 -- 172.21.15.16:0/3512579784 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+22097 (secure 0 0 0) 0x7fb278096230 con 0x7fb28c101fb0 2024-04-05T23:32:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:04.979 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:04.511251+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:32:04.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2700a1120 msgr2=0x7fb2700a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2700a1120 0x7fb2700a35e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb28c101e10 tx=0x7fb27c009260 comp rx=0 tx=0).stop 2024-04-05T23:32:04.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb28c101fb0 msgr2=0x7fb28c0732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb28c101fb0 0x7fb28c0732d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb27800e9a0 tx=0x7fb27800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 shutdown_connections 2024-04-05T23:32:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2700a1120 0x7fb2700a35e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb28c102950 0x7fb28c073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb28c100dc0 0x7fb28c074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 --2- 172.21.15.16:0/3512579784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb28c101fb0 0x7fb28c0732d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 >> 172.21.15.16:0/3512579784 conn(0x7fb28c0fc350 msgr2=0x7fb28c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 shutdown_connections 2024-04-05T23:32:04.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:04.989+0000 7fb291c00700 1 -- 172.21.15.16:0/3512579784 wait complete. 2024-04-05T23:32:05.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:32:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.206+0000 7f1f77000700 1 -- 172.21.15.16:0/2516581968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70102940 msgr2=0x7f1f701091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.206+0000 7f1f77000700 1 --2- 172.21.15.16:0/2516581968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70102940 0x7f1f701091f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009fc0 tx=0x7f1f6800b330 comp rx=0 tx=0).stop 2024-04-05T23:32:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.206+0000 7f1f77000700 1 -- 172.21.15.16:0/2516581968 shutdown_connections 2024-04-05T23:32:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.206+0000 7f1f77000700 1 --2- 172.21.15.16:0/2516581968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70102940 0x7f1f701091f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.206+0000 7f1f77000700 1 --2- 172.21.15.16:0/2516581968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f70100db0 0x7f1f701011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.206+0000 7f1f77000700 1 --2- 172.21.15.16:0/2516581968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f70101fa0 0x7f1f70102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.206+0000 7f1f77000700 1 -- 172.21.15.16:0/2516581968 >> 172.21.15.16:0/2516581968 conn(0x7f1f700fc320 msgr2=0x7f1f700fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.206+0000 7f1f77000700 1 -- 172.21.15.16:0/2516581968 shutdown_connections 2024-04-05T23:32:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.206+0000 7f1f77000700 1 -- 172.21.15.16:0/2516581968 wait complete. 2024-04-05T23:32:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.207+0000 7f1f77000700 1 Processor -- start 2024-04-05T23:32:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.207+0000 7f1f77000700 1 -- start start 2024-04-05T23:32:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.207+0000 7f1f77000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f70100db0 0x7f1f70074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.207+0000 7f1f77000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70101fa0 0x7f1f700732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f77000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f70102940 0x7f1f70073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f77000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70073fb0 con 0x7f1f70101fa0 2024-04-05T23:32:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f77000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70074120 con 0x7f1f70102940 2024-04-05T23:32:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f77000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f701a0e80 con 0x7f1f70100db0 2024-04-05T23:32:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70101fa0 0x7f1f700732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70101fa0 0x7f1f700732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57128/0 (socket says 172.21.15.16:57128) 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6f400700 1 -- 172.21.15.16:0/916483684 learned_addr learned my addr 172.21.15.16:0/916483684 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f70100db0 0x7f1f70074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f75000700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f70102940 0x7f1f70073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6f400700 1 -- 172.21.15.16:0/916483684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f70100db0 msgr2=0x7f1f70074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6f400700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f70100db0 0x7f1f70074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6f400700 1 -- 172.21.15.16:0/916483684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f70102940 msgr2=0x7f1f70073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6f400700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f70102940 0x7f1f70073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6f400700 1 -- 172.21.15.16:0/916483684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f68009c40 con 0x7f1f70101fa0 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f75000700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f70102940 0x7f1f70073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.208+0000 7f1f6f400700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70101fa0 0x7f1f700732d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00e9c0 tx=0x7f1f5c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.209+0000 7f1f6cc00700 1 -- 172.21.15.16:0/916483684 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c00cb70 con 0x7f1f70101fa0 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.209+0000 7f1f6cc00700 1 -- 172.21.15.16:0/916483684 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f5c00eeb0 con 0x7f1f70101fa0 2024-04-05T23:32:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.209+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f701a1110 con 0x7f1f70101fa0 2024-04-05T23:32:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.209+0000 7f1f6cc00700 1 -- 172.21.15.16:0/916483684 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c0054c0 con 0x7f1f70101fa0 2024-04-05T23:32:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.209+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f701a1620 con 0x7f1f70101fa0 2024-04-05T23:32:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.210+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f701081e0 con 0x7f1f70101fa0 2024-04-05T23:32:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.212+0000 7f1f6cc00700 1 -- 172.21.15.16:0/916483684 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f1f5c005620 con 0x7f1f70101fa0 2024-04-05T23:32:05.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.213+0000 7f1f6cc00700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1f540a1120 0x7f1f540a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:05.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.213+0000 7f1f6cc00700 1 -- 172.21.15.16:0/916483684 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1f5c014070 con 0x7f1f70101fa0 2024-04-05T23:32:05.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.213+0000 7f1f6fe00700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1f540a1120 0x7f1f540a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:05.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.214+0000 7f1f6fe00700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1f540a1120 0x7f1f540a35e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1f70101e00 tx=0x7f1f60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:05.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.217+0000 7f1f6cc00700 1 -- 172.21.15.16:0/916483684 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1f5c096200 con 0x7f1f70101fa0 2024-04-05T23:32:05.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: pgmap v229: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1654247106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1654247106' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: osdmap e77: 12 total, 12 up, 12 in 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3638624713' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: osdmap e78: 12 total, 12 up, 12 in 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-05T23:32:05.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:04 smithi016 ceph-mon[38248]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: pgmap v229: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1654247106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1654247106' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: osdmap e77: 12 total, 12 up, 12 in 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3638624713' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: osdmap e78: 12 total, 12 up, 12 in 2024-04-05T23:32:05.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-05T23:32:05.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-05T23:32:05.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:04 smithi130 ceph-mon[79417]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-05T23:32:05.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.437+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f70067b50 con 0x7f1f70101fa0 2024-04-05T23:32:05.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.439+0000 7f1f6cc00700 1 -- 172.21.15.16:0/916483684 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+22097 (secure 0 0 0) 0x7f1f5c099820 con 0x7f1f70101fa0 2024-04-05T23:32:05.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:05.432 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:04.511251+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.441+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1f540a1120 msgr2=0x7f1f540a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.441+0000 7f1f77000700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1f540a1120 0x7f1f540a35e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1f70101e00 tx=0x7f1f60009260 comp rx=0 tx=0).stop 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.441+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70101fa0 msgr2=0x7f1f700732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.442+0000 7f1f77000700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70101fa0 0x7f1f700732d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00e9c0 tx=0x7f1f5c00ecd0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.442+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 shutdown_connections 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.442+0000 7f1f77000700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1f540a1120 0x7f1f540a35e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.442+0000 7f1f77000700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f70102940 0x7f1f70073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.442+0000 7f1f77000700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f70100db0 0x7f1f70074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.442+0000 7f1f77000700 1 --2- 172.21.15.16:0/916483684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f70101fa0 0x7f1f700732d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.442+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 >> 172.21.15.16:0/916483684 conn(0x7f1f700fc320 msgr2=0x7f1f70105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.442+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 shutdown_connections 2024-04-05T23:32:05.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.442+0000 7f1f77000700 1 -- 172.21.15.16:0/916483684 wait complete. 2024-04-05T23:32:05.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-05T23:32:05.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: pgmap v229: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:05.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1654247106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-05T23:32:05.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-05T23:32:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1654247106' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-05T23:32:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: osdmap e77: 12 total, 12 up, 12 in 2024-04-05T23:32:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3638624713' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-05T23:32:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-05T23:32:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-05T23:32:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: osdmap e78: 12 total, 12 up, 12 in 2024-04-05T23:32:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-05T23:32:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-05T23:32:05.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:04 smithi027 ceph-mon[78597]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-05T23:32:05.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.669+0000 7feb46800700 1 -- 172.21.15.16:0/630884379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40102050 msgr2=0x7feb401024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:05.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.669+0000 7feb46800700 1 --2- 172.21.15.16:0/630884379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40102050 0x7feb401024d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009990 tx=0x7feb2c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.669+0000 7feb46800700 1 -- 172.21.15.16:0/630884379 shutdown_connections 2024-04-05T23:32:05.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.669+0000 7feb46800700 1 --2- 172.21.15.16:0/630884379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40102a10 0x7feb40109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.669+0000 7feb46800700 1 --2- 172.21.15.16:0/630884379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb40100e70 0x7feb40101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.669+0000 7feb46800700 1 --2- 172.21.15.16:0/630884379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40102050 0x7feb401024d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.669+0000 7feb46800700 1 -- 172.21.15.16:0/630884379 >> 172.21.15.16:0/630884379 conn(0x7feb400fc430 msgr2=0x7feb400fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:05.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.669+0000 7feb46800700 1 -- 172.21.15.16:0/630884379 shutdown_connections 2024-04-05T23:32:05.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.669+0000 7feb46800700 1 -- 172.21.15.16:0/630884379 wait complete. 2024-04-05T23:32:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.670+0000 7feb46800700 1 Processor -- start 2024-04-05T23:32:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.670+0000 7feb46800700 1 -- start start 2024-04-05T23:32:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb46800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40100e70 0x7feb40199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb46800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40102050 0x7feb40199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb46800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb40102a10 0x7feb401a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb46800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb401a0bf0 con 0x7feb40100e70 2024-04-05T23:32:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb46800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb401a0d30 con 0x7feb40102a10 2024-04-05T23:32:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb46800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb401a0ec0 con 0x7feb40102050 2024-04-05T23:32:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40100e70 0x7feb40199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40100e70 0x7feb40199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57156/0 (socket says 172.21.15.16:57156) 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3f400700 1 -- 172.21.15.16:0/1993888161 learned_addr learned my addr 172.21.15.16:0/1993888161 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3fe00700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb40102a10 0x7feb401a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3ea00700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40102050 0x7feb40199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3f400700 1 -- 172.21.15.16:0/1993888161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40102050 msgr2=0x7feb40199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3f400700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40102050 0x7feb40199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3f400700 1 -- 172.21.15.16:0/1993888161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb40102a10 msgr2=0x7feb401a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3f400700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb40102a10 0x7feb401a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.671+0000 7feb3f400700 1 -- 172.21.15.16:0/1993888161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0095f0 con 0x7feb40100e70 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.672+0000 7feb3f400700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40100e70 0x7feb40199110 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feb3000e970 tx=0x7feb3000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.672+0000 7feb37e00700 1 -- 172.21.15.16:0/1993888161 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30009790 con 0x7feb40100e70 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.672+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb401a11a0 con 0x7feb40100e70 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.672+0000 7feb37e00700 1 -- 172.21.15.16:0/1993888161 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb3000c7e0 con 0x7feb40100e70 2024-04-05T23:32:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.672+0000 7feb37e00700 1 -- 172.21.15.16:0/1993888161 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30003c10 con 0x7feb40100e70 2024-04-05T23:32:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.672+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb401a1700 con 0x7feb40100e70 2024-04-05T23:32:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.673+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb40067c30 con 0x7feb40100e70 2024-04-05T23:32:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.675+0000 7feb37e00700 1 -- 172.21.15.16:0/1993888161 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7feb3000cc50 con 0x7feb40100e70 2024-04-05T23:32:05.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.676+0000 7feb37e00700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb240a1120 0x7feb240a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:05.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.676+0000 7feb37e00700 1 -- 172.21.15.16:0/1993888161 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7feb30014070 con 0x7feb40100e70 2024-04-05T23:32:05.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.676+0000 7feb3ea00700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb240a1120 0x7feb240a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:05.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.677+0000 7feb3ea00700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb240a1120 0x7feb240a35e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feb2c000c00 tx=0x7feb2c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:05.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.680+0000 7feb37e00700 1 -- 172.21.15.16:0/1993888161 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7feb300961c0 con 0x7feb40100e70 2024-04-05T23:32:05.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:05.947+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7feb4010c730 con 0x7feb40100e70 2024-04-05T23:32:06.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:05 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3512579784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:06.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:05 smithi016 ceph-mon[38248]: pgmap v232: 97 pgs: 8 creating+peering, 10 active+clean, 79 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:06.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:05 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/916483684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:06.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:05 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] up:active 2024-04-05T23:32:06.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:05 smithi016 ceph-mon[38248]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:06.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:05 smithi016 ceph-mon[38248]: osdmap e79: 12 total, 12 up, 12 in 2024-04-05T23:32:06.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:05 smithi016 ceph-mon[38248]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:06.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:05 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1993888161' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-05T23:32:06.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:05 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3512579784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:06.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:05 smithi130 ceph-mon[79417]: pgmap v232: 97 pgs: 8 creating+peering, 10 active+clean, 79 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:06.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:05 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/916483684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:06.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:05 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] up:active 2024-04-05T23:32:06.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:05 smithi130 ceph-mon[79417]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:06.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:05 smithi130 ceph-mon[79417]: osdmap e79: 12 total, 12 up, 12 in 2024-04-05T23:32:06.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:05 smithi130 ceph-mon[79417]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:06.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:05 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1993888161' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-05T23:32:06.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:05 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3512579784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:05 smithi027 ceph-mon[78597]: pgmap v232: 97 pgs: 8 creating+peering, 10 active+clean, 79 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:32:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:05 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/916483684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:05 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] up:active 2024-04-05T23:32:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:05 smithi027 ceph-mon[78597]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:05 smithi027 ceph-mon[78597]: osdmap e79: 12 total, 12 up, 12 in 2024-04-05T23:32:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:05 smithi027 ceph-mon[78597]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:06.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:05 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1993888161' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-05T23:32:06.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.524+0000 7feb37e00700 1 -- 172.21.15.16:0/1993888161 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7feb300997e0 con 0x7feb40100e70 2024-04-05T23:32:06.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb240a1120 msgr2=0x7feb240a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:06.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb240a1120 0x7feb240a35e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feb2c000c00 tx=0x7feb2c005fb0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40100e70 msgr2=0x7feb40199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:06.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40100e70 0x7feb40199110 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feb3000e970 tx=0x7feb3000ec80 comp rx=0 tx=0).stop 2024-04-05T23:32:06.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 shutdown_connections 2024-04-05T23:32:06.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb240a1120 0x7feb240a35e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb40100e70 0x7feb40199110 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb40102a10 0x7feb401a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 --2- 172.21.15.16:0/1993888161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40102050 0x7feb40199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 >> 172.21.15.16:0/1993888161 conn(0x7feb400fc430 msgr2=0x7feb40073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 shutdown_connections 2024-04-05T23:32:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.527+0000 7feb46800700 1 -- 172.21.15.16:0/1993888161 wait complete. 2024-04-05T23:32:06.541 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-05T23:32:06.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-05T23:32:06.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.754+0000 7fa1cde00700 1 -- 172.21.15.16:0/4172580434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 msgr2=0x7fa1c8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:06.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.754+0000 7fa1cde00700 1 --2- 172.21.15.16:0/4172580434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 0x7fa1c8101280 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009990 tx=0x7fa1b8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.754+0000 7fa1cde00700 1 -- 172.21.15.16:0/4172580434 shutdown_connections 2024-04-05T23:32:06.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.754+0000 7fa1cde00700 1 --2- 172.21.15.16:0/4172580434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 0x7fa1c8101280 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.754+0000 7fa1cde00700 1 --2- 172.21.15.16:0/4172580434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1c8102a00 0x7fa1c8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.754+0000 7fa1cde00700 1 --2- 172.21.15.16:0/4172580434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1c8102040 0x7fa1c81024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.754+0000 7fa1cde00700 1 -- 172.21.15.16:0/4172580434 >> 172.21.15.16:0/4172580434 conn(0x7fa1c80fc440 msgr2=0x7fa1c80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:06.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.754+0000 7fa1cde00700 1 -- 172.21.15.16:0/4172580434 shutdown_connections 2024-04-05T23:32:06.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.754+0000 7fa1cde00700 1 -- 172.21.15.16:0/4172580434 wait complete. 2024-04-05T23:32:06.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.755+0000 7fa1cde00700 1 Processor -- start 2024-04-05T23:32:06.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1cde00700 1 -- start start 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1cde00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 0x7fa1c8199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1cde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1c8102040 0x7fa1c81995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1cde00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1c8102a00 0x7fa1c81a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1cde00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c81a0bc0 con 0x7fa1c8100e60 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1cde00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c81a0d00 con 0x7fa1c8102040 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1cde00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c81a0e90 con 0x7fa1c8102a00 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1c6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 0x7fa1c8199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1c6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1c8102040 0x7fa1c81995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1c6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 0x7fa1c8199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57172/0 (socket says 172.21.15.16:57172) 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1c6a00700 1 -- 172.21.15.16:0/1807774232 learned_addr learned my addr 172.21.15.16:0/1807774232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1c6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1c8102040 0x7fa1c81995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59002/0 (socket says 172.21.15.16:59002) 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1c7400700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1c8102a00 0x7fa1c81a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:06.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1c6a00700 1 -- 172.21.15.16:0/1807774232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1c8102a00 msgr2=0x7fa1c81a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1c6a00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1c8102a00 0x7fa1c81a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.756+0000 7fa1c6a00700 1 -- 172.21.15.16:0/1807774232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1c8102040 msgr2=0x7fa1c81995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.757+0000 7fa1c6a00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1c8102040 0x7fa1c81995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.757+0000 7fa1c6a00700 1 -- 172.21.15.16:0/1807774232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b80095f0 con 0x7fa1c8100e60 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.757+0000 7fa1c6000700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1c8102040 0x7fa1c81995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.757+0000 7fa1c6a00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 0x7fa1c8199090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800b680 tx=0x7fa1b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.757+0000 7fa1cca00700 1 -- 172.21.15.16:0/1807774232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b801d070 con 0x7fa1c8100e60 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.757+0000 7fa1cca00700 1 -- 172.21.15.16:0/1807774232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1b8021410 con 0x7fa1c8100e60 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.757+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c81a1110 con 0x7fa1c8100e60 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.757+0000 7fa1cca00700 1 -- 172.21.15.16:0/1807774232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8017650 con 0x7fa1c8100e60 2024-04-05T23:32:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.757+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c81a1530 con 0x7fa1c8100e60 2024-04-05T23:32:06.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.758+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c8067c30 con 0x7fa1c8100e60 2024-04-05T23:32:06.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.760+0000 7fa1cca00700 1 -- 172.21.15.16:0/1807774232 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fa1b8021580 con 0x7fa1c8100e60 2024-04-05T23:32:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.762+0000 7fa1cca00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1ac0a1120 0x7fa1ac0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.762+0000 7fa1cca00700 1 -- 172.21.15.16:0/1807774232 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa1b80cdbc0 con 0x7fa1c8100e60 2024-04-05T23:32:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.762+0000 7fa1c6000700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1ac0a1120 0x7fa1ac0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:06.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.763+0000 7fa1c6000700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1ac0a1120 0x7fa1ac0a35e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8100b80 tx=0x7fa1b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:06.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:06.766+0000 7fa1cca00700 1 -- 172.21.15.16:0/1807774232 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa1b8097490 con 0x7fa1c8100e60 2024-04-05T23:32:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.040+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fa1c810c990 con 0x7fa1c8100e60 2024-04-05T23:32:07.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:06 smithi016 ceph-mon[38248]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-05T23:32:07.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:06 smithi016 ceph-mon[38248]: Cluster is now healthy 2024-04-05T23:32:07.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:06 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1993888161' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-05T23:32:07.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:06 smithi016 ceph-mon[38248]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:07.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:06 smithi130 ceph-mon[79417]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-05T23:32:07.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:06 smithi130 ceph-mon[79417]: Cluster is now healthy 2024-04-05T23:32:07.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:06 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1993888161' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-05T23:32:07.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:06 smithi130 ceph-mon[79417]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:07.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:06 smithi027 ceph-mon[78597]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-05T23:32:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:06 smithi027 ceph-mon[78597]: Cluster is now healthy 2024-04-05T23:32:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:06 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1993888161' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-05T23:32:07.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:06 smithi027 ceph-mon[78597]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.992+0000 7fa1cca00700 1 -- 172.21.15.16:0/1807774232 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa1b809aab0 con 0x7fa1c8100e60 2024-04-05T23:32:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.995+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1ac0a1120 msgr2=0x7fa1ac0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.995+0000 7fa1cde00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1ac0a1120 0x7fa1ac0a35e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8100b80 tx=0x7fa1b4009260 comp rx=0 tx=0).stop 2024-04-05T23:32:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.995+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 msgr2=0x7fa1c8199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.996+0000 7fa1cde00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 0x7fa1c8199090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800b680 tx=0x7fa1b8004730 comp rx=0 tx=0).stop 2024-04-05T23:32:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.996+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 shutdown_connections 2024-04-05T23:32:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.996+0000 7fa1cde00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1ac0a1120 0x7fa1ac0a35e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.996+0000 7fa1cde00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1c8100e60 0x7fa1c8199090 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.996+0000 7fa1cde00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1c8102a00 0x7fa1c81a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.996+0000 7fa1cde00700 1 --2- 172.21.15.16:0/1807774232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1c8102040 0x7fa1c81995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.996+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 >> 172.21.15.16:0/1807774232 conn(0x7fa1c80fc440 msgr2=0x7fa1c81078f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.996+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 shutdown_connections 2024-04-05T23:32:07.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:07.996+0000 7fa1cde00700 1 -- 172.21.15.16:0/1807774232 wait complete. 2024-04-05T23:32:08.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-05T23:32:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.230+0000 7f3125200700 1 -- 172.21.15.16:0/1951726518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31200ffda0 msgr2=0x7f31201001c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.230+0000 7f3125200700 1 --2- 172.21.15.16:0/1951726518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31200ffda0 0x7f31201001c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31100099c0 tx=0x7f3110009cd0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.230+0000 7f3125200700 1 -- 172.21.15.16:0/1951726518 shutdown_connections 2024-04-05T23:32:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.230+0000 7f3125200700 1 --2- 172.21.15.16:0/1951726518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31200ffda0 0x7f31201001c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.230+0000 7f3125200700 1 --2- 172.21.15.16:0/1951726518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3120101930 0x7f31201081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.230+0000 7f3125200700 1 --2- 172.21.15.16:0/1951726518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3120100f90 0x7f31201013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.230+0000 7f3125200700 1 -- 172.21.15.16:0/1951726518 >> 172.21.15.16:0/1951726518 conn(0x7f31200fb330 msgr2=0x7f31200fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.230+0000 7f3125200700 1 -- 172.21.15.16:0/1951726518 shutdown_connections 2024-04-05T23:32:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.230+0000 7f3125200700 1 -- 172.21.15.16:0/1951726518 wait complete. 2024-04-05T23:32:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.231+0000 7f3125200700 1 Processor -- start 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.231+0000 7f3125200700 1 -- start start 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.231+0000 7f3125200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31200ffda0 0x7f3120199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.231+0000 7f3125200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3120100f90 0x7f3120199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.231+0000 7f3125200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3120101930 0x7f31201a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.231+0000 7f3125200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31201a0b70 con 0x7f3120101930 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.231+0000 7f3125200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31201a0cb0 con 0x7f3120100f90 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.231+0000 7f3125200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31201a0e40 con 0x7f31200ffda0 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3120101930 0x7f31201a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3120101930 0x7f31201a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57182/0 (socket says 172.21.15.16:57182) 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311e800700 1 -- 172.21.15.16:0/4110127154 learned_addr learned my addr 172.21.15.16:0/4110127154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311de00700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31200ffda0 0x7f3120199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311d400700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3120100f90 0x7f3120199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311e800700 1 -- 172.21.15.16:0/4110127154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31200ffda0 msgr2=0x7f3120199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311e800700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31200ffda0 0x7f3120199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311e800700 1 -- 172.21.15.16:0/4110127154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3120100f90 msgr2=0x7f3120199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311e800700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3120100f90 0x7f3120199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311e800700 1 -- 172.21.15.16:0/4110127154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100095f0 con 0x7f3120101930 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311de00700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31200ffda0 0x7f3120199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f311e800700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3120101930 0x7f31201a05a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f311800ef30 tx=0x7f311800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f3116a00700 1 -- 172.21.15.16:0/4110127154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118018070 con 0x7f3120101930 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f3116a00700 1 -- 172.21.15.16:0/4110127154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f311800f040 con 0x7f3120101930 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f3116a00700 1 -- 172.21.15.16:0/4110127154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311800cdb0 con 0x7f3120101930 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.232+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31201a10c0 con 0x7f3120101930 2024-04-05T23:32:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.233+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31201a1580 con 0x7f3120101930 2024-04-05T23:32:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.233+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3120067c30 con 0x7f3120101930 2024-04-05T23:32:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.235+0000 7f3116a00700 1 -- 172.21.15.16:0/4110127154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f3118003a00 con 0x7f3120101930 2024-04-05T23:32:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.237+0000 7f3116a00700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f31040a11a0 0x7f31040a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.237+0000 7f3116a00700 1 -- 172.21.15.16:0/4110127154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f31180cc370 con 0x7f3120101930 2024-04-05T23:32:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.237+0000 7f311de00700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f31040a11a0 0x7f31040a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.238+0000 7f311de00700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f31040a11a0 0x7f31040a3660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3110009fd0 tx=0x7f3110019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:08.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.241+0000 7f3116a00700 1 -- 172.21.15.16:0/4110127154 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3118095c40 con 0x7f3120101930 2024-04-05T23:32:08.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:07 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1807774232' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-05T23:32:08.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:07 smithi016 ceph-mon[38248]: pgmap v234: 97 pgs: 8 creating+peering, 61 active+clean, 28 unknown; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-05T23:32:08.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:07 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1807774232' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-05T23:32:08.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:07 smithi130 ceph-mon[79417]: pgmap v234: 97 pgs: 8 creating+peering, 61 active+clean, 28 unknown; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-05T23:32:08.466 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:32:08 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-b[116618]: 2024-04-05T23:32:08.224+0000 7ff979be4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-05T23:32:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:07 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1807774232' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-05T23:32:08.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:07 smithi027 ceph-mon[78597]: pgmap v234: 97 pgs: 8 creating+peering, 61 active+clean, 28 unknown; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-05T23:32:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:08.507+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f312010b600 con 0x7f3120101930 2024-04-05T23:32:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.000+0000 7f3116a00700 1 -- 172.21.15.16:0/4110127154 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f31180b7e60 con 0x7f3120101930 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f31040a11a0 msgr2=0x7f31040a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f31040a11a0 0x7f31040a3660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3110009fd0 tx=0x7f3110019040 comp rx=0 tx=0).stop 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3120101930 msgr2=0x7f31201a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3120101930 0x7f31201a05a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f311800ef30 tx=0x7f311800c3a0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 shutdown_connections 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f31040a11a0 0x7f31040a3660 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3120101930 0x7f31201a05a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31200ffda0 0x7f3120199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 --2- 172.21.15.16:0/4110127154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3120100f90 0x7f3120199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 >> 172.21.15.16:0/4110127154 conn(0x7f31200fb330 msgr2=0x7f3120067f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 shutdown_connections 2024-04-05T23:32:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.004+0000 7f3125200700 1 -- 172.21.15.16:0/4110127154 wait complete. 2024-04-05T23:32:09.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-05T23:32:09.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.236+0000 7fe46ec00700 1 -- 172.21.15.16:0/130411132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 msgr2=0x7fe468109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.236+0000 7fe46ec00700 1 --2- 172.21.15.16:0/130411132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 0x7fe468109320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe46000b0a0 tx=0x7fe46000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.236+0000 7fe46ec00700 1 -- 172.21.15.16:0/130411132 shutdown_connections 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.236+0000 7fe46ec00700 1 --2- 172.21.15.16:0/130411132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 0x7fe468109320 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.236+0000 7fe46ec00700 1 --2- 172.21.15.16:0/130411132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe468100e60 0x7fe468101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.236+0000 7fe46ec00700 1 --2- 172.21.15.16:0/130411132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe468102040 0x7fe4681024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.236+0000 7fe46ec00700 1 -- 172.21.15.16:0/130411132 >> 172.21.15.16:0/130411132 conn(0x7fe4680fc440 msgr2=0x7fe4680fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.236+0000 7fe46ec00700 1 -- 172.21.15.16:0/130411132 shutdown_connections 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.236+0000 7fe46ec00700 1 -- 172.21.15.16:0/130411132 wait complete. 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.237+0000 7fe46ec00700 1 Processor -- start 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.237+0000 7fe46ec00700 1 -- start start 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46ec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe468100e60 0x7fe468199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46ec00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe468102040 0x7fe468199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46ec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 0x7fe4681a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46ec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4681a0c60 con 0x7fe468102a00 2024-04-05T23:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46ec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4681a0da0 con 0x7fe468100e60 2024-04-05T23:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46ec00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4681a0f30 con 0x7fe468102040 2024-04-05T23:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 0x7fe4681a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 0x7fe4681a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57200/0 (socket says 172.21.15.16:57200) 2024-04-05T23:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe467e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe468100e60 0x7fe468199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46cc00700 1 -- 172.21.15.16:0/1887739232 learned_addr learned my addr 172.21.15.16:0/1887739232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe467400700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe468102040 0x7fe468199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46cc00700 1 -- 172.21.15.16:0/1887739232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe468102040 msgr2=0x7fe468199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46cc00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe468102040 0x7fe468199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46cc00700 1 -- 172.21.15.16:0/1887739232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe468100e60 msgr2=0x7fe468199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46cc00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe468100e60 0x7fe468199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.238+0000 7fe46cc00700 1 -- 172.21.15.16:0/1887739232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460009d00 con 0x7fe468102a00 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.239+0000 7fe46cc00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 0x7fe4681a0690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe460007a60 tx=0x7fe460007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.239+0000 7fe464c00700 1 -- 172.21.15.16:0/1887739232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46000e040 con 0x7fe468102a00 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.239+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4681a11b0 con 0x7fe468102a00 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.239+0000 7fe464c00700 1 -- 172.21.15.16:0/1887739232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe460004c50 con 0x7fe468102a00 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.239+0000 7fe464c00700 1 -- 172.21.15.16:0/1887739232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46001b6c0 con 0x7fe468102a00 2024-04-05T23:32:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.239+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4681a1580 con 0x7fe468102a00 2024-04-05T23:32:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.240+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe468067c30 con 0x7fe468102a00 2024-04-05T23:32:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.242+0000 7fe464c00700 1 -- 172.21.15.16:0/1887739232 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fe46001d470 con 0x7fe468102a00 2024-04-05T23:32:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.243+0000 7fe464c00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe44c0a1120 0x7fe44c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.243+0000 7fe464c00700 1 -- 172.21.15.16:0/1887739232 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe4600cdae0 con 0x7fe468102a00 2024-04-05T23:32:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.244+0000 7fe467e00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe44c0a1120 0x7fe44c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.244+0000 7fe467e00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe44c0a1120 0x7fe44c0a35e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe468101bb0 tx=0x7fe458009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.247+0000 7fe464c00700 1 -- 172.21.15.16:0/1887739232 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe4600973b0 con 0x7fe468102a00 2024-04-05T23:32:09.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:08 smithi016 ceph-mon[38248]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T23:32:09.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:08 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1807774232' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-05T23:32:09.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:08 smithi016 ceph-mon[38248]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-05T23:32:09.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:08 smithi016 ceph-mon[38248]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:09.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:08 smithi016 ceph-mon[38248]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-05T23:32:09.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:08 smithi016 ceph-mon[38248]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-05T23:32:09.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:08 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4110127154' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-05T23:32:09.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:08 smithi130 ceph-mon[79417]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T23:32:09.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:08 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1807774232' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-05T23:32:09.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:08 smithi130 ceph-mon[79417]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-05T23:32:09.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:08 smithi130 ceph-mon[79417]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:09.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:08 smithi130 ceph-mon[79417]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-05T23:32:09.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:08 smithi130 ceph-mon[79417]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-05T23:32:09.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:08 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4110127154' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-05T23:32:09.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:08 smithi027 ceph-mon[78597]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T23:32:09.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:08 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1807774232' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-05T23:32:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:08 smithi027 ceph-mon[78597]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-05T23:32:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:08 smithi027 ceph-mon[78597]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-05T23:32:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:08 smithi027 ceph-mon[78597]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-05T23:32:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:08 smithi027 ceph-mon[78597]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-05T23:32:09.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:08 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4110127154' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-05T23:32:09.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:09.530+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fe46810c720 con 0x7fe468102a00 2024-04-05T23:32:10.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] up:active 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4110127154' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: pgmap v235: 97 pgs: 6 creating+peering, 91 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] up:active 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: daemon mds.d assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=d=up:creating} 3 up:standby-replay 5 up:standby 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: daemon mds.d is now active in filesystem cephfs as rank 3 2024-04-05T23:32:10.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1887739232' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] up:active 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4110127154' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: pgmap v235: 97 pgs: 6 creating+peering, 91 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] up:active 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: daemon mds.d assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-05T23:32:10.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=d=up:creating} 3 up:standby-replay 5 up:standby 2024-04-05T23:32:10.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: daemon mds.d is now active in filesystem cephfs as rank 3 2024-04-05T23:32:10.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:10 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1887739232' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-05T23:32:10.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.413+0000 7fe464c00700 1 -- 172.21.15.16:0/1887739232 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v24) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe46009a9d0 con 0x7fe468102a00 2024-04-05T23:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.416+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe44c0a1120 msgr2=0x7fe44c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.416+0000 7fe46ec00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe44c0a1120 0x7fe44c0a35e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe468101bb0 tx=0x7fe458009260 comp rx=0 tx=0).stop 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.416+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 msgr2=0x7fe4681a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.416+0000 7fe46ec00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 0x7fe4681a0690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe460007a60 tx=0x7fe460007930 comp rx=0 tx=0).stop 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.417+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 shutdown_connections 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.417+0000 7fe46ec00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe44c0a1120 0x7fe44c0a35e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.417+0000 7fe46ec00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe468102a00 0x7fe4681a0690 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.417+0000 7fe46ec00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe468100e60 0x7fe468199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.417+0000 7fe46ec00700 1 --2- 172.21.15.16:0/1887739232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe468102040 0x7fe468199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.417+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 >> 172.21.15.16:0/1887739232 conn(0x7fe4680fc440 msgr2=0x7fe468073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.417+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 shutdown_connections 2024-04-05T23:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.417+0000 7fe46ec00700 1 -- 172.21.15.16:0/1887739232 wait complete. 2024-04-05T23:32:10.434 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-05T23:32:10.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-05T23:32:10.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] up:active 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4110127154' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: pgmap v235: 97 pgs: 6 creating+peering, 91 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] up:active 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: daemon mds.d assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=d=up:creating} 3 up:standby-replay 5 up:standby 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: daemon mds.d is now active in filesystem cephfs as rank 3 2024-04-05T23:32:10.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:10 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1887739232' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-05T23:32:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.648+0000 7f5b8f200700 1 -- 172.21.15.16:0/1908023591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88108d80 msgr2=0x7f5b8810b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.648+0000 7f5b8f200700 1 --2- 172.21.15.16:0/1908023591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88108d80 0x7f5b8810b210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009f90 tx=0x7f5b8000b330 comp rx=0 tx=0).stop 2024-04-05T23:32:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.648+0000 7f5b8f200700 1 -- 172.21.15.16:0/1908023591 shutdown_connections 2024-04-05T23:32:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.648+0000 7f5b8f200700 1 --2- 172.21.15.16:0/1908023591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88108d80 0x7f5b8810b210 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.648+0000 7f5b8f200700 1 --2- 172.21.15.16:0/1908023591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b88074f20 0x7f5b88073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.648+0000 7f5b8f200700 1 --2- 172.21.15.16:0/1908023591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88073940 0x7f5b88073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.648+0000 7f5b8f200700 1 -- 172.21.15.16:0/1908023591 >> 172.21.15.16:0/1908023591 conn(0x7f5b880fc430 msgr2=0x7f5b880fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.648+0000 7f5b8f200700 1 -- 172.21.15.16:0/1908023591 shutdown_connections 2024-04-05T23:32:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.648+0000 7f5b8f200700 1 -- 172.21.15.16:0/1908023591 wait complete. 2024-04-05T23:32:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.649+0000 7f5b8f200700 1 Processor -- start 2024-04-05T23:32:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.649+0000 7f5b8f200700 1 -- start start 2024-04-05T23:32:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b8f200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88073940 0x7f5b88199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b8f200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88074f20 0x7f5b88199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b8f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b88108d80 0x7f5b881a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b8f200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881a0c50 con 0x7f5b88074f20 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b8f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881a0d90 con 0x7f5b88108d80 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b8f200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881a0f20 con 0x7f5b88073940 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b87400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88074f20 0x7f5b88199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b87400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88074f20 0x7f5b88199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57228/0 (socket says 172.21.15.16:57228) 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b87400700 1 -- 172.21.15.16:0/3068286399 learned_addr learned my addr 172.21.15.16:0/3068286399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b8d200700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b88108d80 0x7f5b881a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b87e00700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88073940 0x7f5b88199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b87400700 1 -- 172.21.15.16:0/3068286399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88073940 msgr2=0x7f5b88199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b87400700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88073940 0x7f5b88199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b87400700 1 -- 172.21.15.16:0/3068286399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b88108d80 msgr2=0x7f5b881a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b87400700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b88108d80 0x7f5b881a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b87400700 1 -- 172.21.15.16:0/3068286399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0095f0 con 0x7f5b88074f20 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.650+0000 7f5b8d200700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b88108d80 0x7f5b881a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.651+0000 7f5b87400700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88074f20 0x7f5b88199660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400e970 tx=0x7f5b7400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.651+0000 7f5b84c00700 1 -- 172.21.15.16:0/3068286399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74009790 con 0x7f5b88074f20 2024-04-05T23:32:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.651+0000 7f5b84c00700 1 -- 172.21.15.16:0/3068286399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b7400c7e0 con 0x7f5b88074f20 2024-04-05T23:32:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.651+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b80009c40 con 0x7f5b88074f20 2024-04-05T23:32:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.651+0000 7f5b84c00700 1 -- 172.21.15.16:0/3068286399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74003c70 con 0x7f5b88074f20 2024-04-05T23:32:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.651+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b881a15d0 con 0x7f5b88074f20 2024-04-05T23:32:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.653+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b88107b40 con 0x7f5b88074f20 2024-04-05T23:32:10.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.655+0000 7f5b84c00700 1 -- 172.21.15.16:0/3068286399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7f5b7400cc50 con 0x7f5b88074f20 2024-04-05T23:32:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.656+0000 7f5b84c00700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b700a11a0 0x7f5b700a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.656+0000 7f5b84c00700 1 -- 172.21.15.16:0/3068286399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5b74014070 con 0x7f5b88074f20 2024-04-05T23:32:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.656+0000 7f5b87e00700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b700a11a0 0x7f5b700a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.657+0000 7f5b87e00700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b700a11a0 0x7f5b700a3660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c005fd0 tx=0x7f5b7c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.660+0000 7f5b84c00700 1 -- 172.21.15.16:0/3068286399 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5b740961a0 con 0x7f5b88074f20 2024-04-05T23:32:10.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.909+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b88061320 con 0x7f5b700a11a0 2024-04-05T23:32:10.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.932+0000 7f5b84c00700 1 -- 172.21.15.16:0/3068286399 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5b88061320 con 0x7f5b700a11a0 2024-04-05T23:32:10.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b700a11a0 msgr2=0x7f5b700a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b700a11a0 0x7f5b700a3660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c005fd0 tx=0x7f5b7c009310 comp rx=0 tx=0).stop 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88074f20 msgr2=0x7f5b88199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88074f20 0x7f5b88199660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400e970 tx=0x7f5b7400ec80 comp rx=0 tx=0).stop 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 shutdown_connections 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b700a11a0 0x7f5b700a3660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b88108d80 0x7f5b881a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88073940 0x7f5b88199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 --2- 172.21.15.16:0/3068286399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b88074f20 0x7f5b88199660 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 >> 172.21.15.16:0/3068286399 conn(0x7f5b880fc430 msgr2=0x7f5b88104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.934+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 shutdown_connections 2024-04-05T23:32:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:10.935+0000 7f5b8f200700 1 -- 172.21.15.16:0/3068286399 wait complete. 2024-04-05T23:32:10.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-05T23:32:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.156+0000 7fe76ea00700 1 -- 172.21.15.16:0/2902987459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102040 msgr2=0x7fe7681024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.156+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2902987459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102040 0x7fe7681024c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe750009990 tx=0x7fe750009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.156+0000 7fe76ea00700 1 -- 172.21.15.16:0/2902987459 shutdown_connections 2024-04-05T23:32:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.156+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2902987459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe768102a00 0x7fe768109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.156+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2902987459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe768100e60 0x7fe768101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.156+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2902987459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102040 0x7fe7681024c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.156+0000 7fe76ea00700 1 -- 172.21.15.16:0/2902987459 >> 172.21.15.16:0/2902987459 conn(0x7fe7680fc440 msgr2=0x7fe7680fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.157+0000 7fe76ea00700 1 -- 172.21.15.16:0/2902987459 shutdown_connections 2024-04-05T23:32:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.157+0000 7fe76ea00700 1 -- 172.21.15.16:0/2902987459 wait complete. 2024-04-05T23:32:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ea00700 1 Processor -- start 2024-04-05T23:32:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ea00700 1 -- start start 2024-04-05T23:32:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe768100e60 0x7fe768199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe768102040 0x7fe7681995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102a00 0x7fe7681a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ea00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681a0b90 con 0x7fe768102a00 2024-04-05T23:32:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681a0cd0 con 0x7fe768100e60 2024-04-05T23:32:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ea00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681a0e60 con 0x7fe768102040 2024-04-05T23:32:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ca00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102a00 0x7fe7681a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe767e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe768102040 0x7fe7681995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ca00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102a00 0x7fe7681a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57270/0 (socket says 172.21.15.16:57270) 2024-04-05T23:32:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe76ca00700 1 -- 172.21.15.16:0/2523940000 learned_addr learned my addr 172.21.15.16:0/2523940000 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.158+0000 7fe75fe00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe768100e60 0x7fe768199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe767e00700 1 -- 172.21.15.16:0/2523940000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe768100e60 msgr2=0x7fe768199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe767e00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe768100e60 0x7fe768199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe767e00700 1 -- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102a00 msgr2=0x7fe7681a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe767e00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102a00 0x7fe7681a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe767e00700 1 -- 172.21.15.16:0/2523940000 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500095f0 con 0x7fe768102040 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe76ca00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102a00 0x7fe7681a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe767e00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe768102040 0x7fe7681995a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe750000c00 tx=0x7fe750009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe765600700 1 -- 172.21.15.16:0/2523940000 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75001d070 con 0x7fe768102040 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7681a10e0 con 0x7fe768102040 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe765600700 1 -- 172.21.15.16:0/2523940000 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe750021410 con 0x7fe768102040 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe765600700 1 -- 172.21.15.16:0/2523940000 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750017650 con 0x7fe768102040 2024-04-05T23:32:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.159+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7681a1510 con 0x7fe768102040 2024-04-05T23:32:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.160+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe768067c30 con 0x7fe768102040 2024-04-05T23:32:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.163+0000 7fe765600700 1 -- 172.21.15.16:0/2523940000 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135873+0+0 (secure 0 0 0) 0x7fe750021580 con 0x7fe768102040 2024-04-05T23:32:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.164+0000 7fe765600700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe74c0a1120 0x7fe74c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.164+0000 7fe765600700 1 -- 172.21.15.16:0/2523940000 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe7500ce190 con 0x7fe768102040 2024-04-05T23:32:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.164+0000 7fe75fe00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe74c0a1120 0x7fe74c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.165+0000 7fe75fe00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe74c0a1120 0x7fe74c0a35e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe768101bb0 tx=0x7fe758009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:11.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.166+0000 7fe765600700 1 -- 172.21.15.16:0/2523940000 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe750097a60 con 0x7fe768102040 2024-04-05T23:32:11.332 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:32:10.915+0000 7fdcfa800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:32:11.332 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:32:10.915+0000 7fdcfa800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:32:11.332 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:32:10.915+0000 7fdcfa800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:32:11.332 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:32:10.915+0000 7fdcfa800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:32:11.332 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:32:10 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:32:10.915+0000 7fdcfa800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:32:11.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.429+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fe768108fc0 con 0x7fe74c0a1120 2024-04-05T23:32:11.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.436+0000 7fe765600700 1 -- 172.21.15.16:0/2523940000 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fe750097e30 con 0x7fe768102040 2024-04-05T23:32:11.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.456+0000 7fe765600700 1 -- 172.21.15.16:0/2523940000 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe768108fc0 con 0x7fe74c0a1120 2024-04-05T23:32:11.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe74c0a1120 msgr2=0x7fe74c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe74c0a1120 0x7fe74c0a35e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe768101bb0 tx=0x7fe758009260 comp rx=0 tx=0).stop 2024-04-05T23:32:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe768102040 msgr2=0x7fe7681995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe768102040 0x7fe7681995a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe750000c00 tx=0x7fe750009580 comp rx=0 tx=0).stop 2024-04-05T23:32:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 shutdown_connections 2024-04-05T23:32:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe74c0a1120 0x7fe74c0a35e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe768102a00 0x7fe7681a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe768100e60 0x7fe768199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 --2- 172.21.15.16:0/2523940000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe768102040 0x7fe7681995a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 >> 172.21.15.16:0/2523940000 conn(0x7fe7680fc440 msgr2=0x7fe7681078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:11.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 shutdown_connections 2024-04-05T23:32:11.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.459+0000 7fe76ea00700 1 -- 172.21.15.16:0/2523940000 wait complete. 2024-04-05T23:32:11.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-05T23:32:11.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.683+0000 7f858a800700 1 -- 172.21.15.16:0/772832130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 msgr2=0x7f8584109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.683+0000 7f858a800700 1 --2- 172.21.15.16:0/772832130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 0x7f8584109320 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f857c00b0a0 tx=0x7f857c00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.683+0000 7f858a800700 1 -- 172.21.15.16:0/772832130 shutdown_connections 2024-04-05T23:32:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.683+0000 7f858a800700 1 --2- 172.21.15.16:0/772832130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 0x7f8584109320 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.683+0000 7f858a800700 1 --2- 172.21.15.16:0/772832130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584100e60 0x7f8584101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.683+0000 7f858a800700 1 --2- 172.21.15.16:0/772832130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8584102040 0x7f85841024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.683+0000 7f858a800700 1 -- 172.21.15.16:0/772832130 >> 172.21.15.16:0/772832130 conn(0x7f85840fc440 msgr2=0x7f85840fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.683+0000 7f858a800700 1 -- 172.21.15.16:0/772832130 shutdown_connections 2024-04-05T23:32:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.683+0000 7f858a800700 1 -- 172.21.15.16:0/772832130 wait complete. 2024-04-05T23:32:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.684+0000 7f858a800700 1 Processor -- start 2024-04-05T23:32:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f858a800700 1 -- start start 2024-04-05T23:32:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f858a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584100e60 0x7f8584196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f858a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8584102040 0x7f8584197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f858a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 0x7f858419e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f858a800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858419ea50 con 0x7f8584102a00 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f858a800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858419eb90 con 0x7f8584100e60 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f858a800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858419ed20 con 0x7f8584102040 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f8583e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 0x7f858419e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f8583e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 0x7f858419e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57288/0 (socket says 172.21.15.16:57288) 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f8583e00700 1 -- 172.21.15.16:0/600576588 learned_addr learned my addr 172.21.15.16:0/600576588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f857b400700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584100e60 0x7f8584196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f8583400700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8584102040 0x7f8584197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f8583e00700 1 -- 172.21.15.16:0/600576588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8584102040 msgr2=0x7f8584197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.685+0000 7f8583e00700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8584102040 0x7f8584197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.686+0000 7f8583e00700 1 -- 172.21.15.16:0/600576588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584100e60 msgr2=0x7f8584196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.686+0000 7f8583e00700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584100e60 0x7f8584196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.686+0000 7f8583e00700 1 -- 172.21.15.16:0/600576588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8574009520 con 0x7f8584102a00 2024-04-05T23:32:11.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.686+0000 7f8583e00700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 0x7f858419e480 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f857c007a60 tx=0x7f857c007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.686+0000 7f8580c00700 1 -- 172.21.15.16:0/600576588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c00e040 con 0x7f8584102a00 2024-04-05T23:32:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.686+0000 7f8580c00700 1 -- 172.21.15.16:0/600576588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f857c004c50 con 0x7f8584102a00 2024-04-05T23:32:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.686+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857c009d00 con 0x7f8584102a00 2024-04-05T23:32:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.686+0000 7f8580c00700 1 -- 172.21.15.16:0/600576588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c01b6c0 con 0x7f8584102a00 2024-04-05T23:32:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.686+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858419f1d0 con 0x7f8584102a00 2024-04-05T23:32:11.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.687+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8584108290 con 0x7f8584102a00 2024-04-05T23:32:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.689+0000 7f8580c00700 1 -- 172.21.15.16:0/600576588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f857c01d470 con 0x7f8584102a00 2024-04-05T23:32:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.690+0000 7f8580c00700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f85680a1230 0x7f85680a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.690+0000 7f8580c00700 1 -- 172.21.15.16:0/600576588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f857c0cdbe0 con 0x7f8584102a00 2024-04-05T23:32:11.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.691+0000 7f857b400700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f85680a1230 0x7f85680a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:11.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.691+0000 7f857b400700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f85680a1230 0x7f85680a36f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8584101bb0 tx=0x7f8574009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.695+0000 7f8580c00700 1 -- 172.21.15.16:0/600576588 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f857c097350 con 0x7f8584102a00 2024-04-05T23:32:11.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:11 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] up:active 2024-04-05T23:32:11.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:11 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1887739232' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-05T23:32:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:11 smithi027 ceph-mon[78597]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-05T23:32:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:11 smithi027 ceph-mon[78597]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T23:32:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:11 smithi027 ceph-mon[78597]: Cluster is now healthy 2024-04-05T23:32:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:11 smithi027 ceph-mon[78597]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=d=up:active} 3 up:standby-replay 5 up:standby 2024-04-05T23:32:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:11 smithi027 ceph-mon[78597]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-04-05T23:32:11.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:11 smithi027 ceph-mon[78597]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-04-05T23:32:11.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:11 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] up:active 2024-04-05T23:32:11.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:11 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1887739232' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-05T23:32:11.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:11 smithi016 ceph-mon[38248]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-05T23:32:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:11 smithi016 ceph-mon[38248]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T23:32:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:11 smithi016 ceph-mon[38248]: Cluster is now healthy 2024-04-05T23:32:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:11 smithi016 ceph-mon[38248]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=d=up:active} 3 up:standby-replay 5 up:standby 2024-04-05T23:32:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:11 smithi016 ceph-mon[38248]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-04-05T23:32:11.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:11 smithi016 ceph-mon[38248]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-04-05T23:32:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:11 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] up:active 2024-04-05T23:32:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:11 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1887739232' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-05T23:32:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:11 smithi130 ceph-mon[79417]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-05T23:32:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:11 smithi130 ceph-mon[79417]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T23:32:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:11 smithi130 ceph-mon[79417]: Cluster is now healthy 2024-04-05T23:32:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:11 smithi130 ceph-mon[79417]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=d=up:active} 3 up:standby-replay 5 up:standby 2024-04-05T23:32:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:11 smithi130 ceph-mon[79417]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-04-05T23:32:11.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:11 smithi130 ceph-mon[79417]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-04-05T23:32:11.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.948+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f858402ce00 con 0x7f85680a1230 2024-04-05T23:32:11.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.953+0000 7f8580c00700 1 -- 172.21.15.16:0/600576588 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f858402ce00 con 0x7f85680a1230 2024-04-05T23:32:11.945 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_0/7d48e481-b421-4587-8960-f94cf8941bc0 2024-04-05T23:32:11.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f85680a1230 msgr2=0x7f85680a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f85680a1230 0x7f85680a36f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8584101bb0 tx=0x7f8574009260 comp rx=0 tx=0).stop 2024-04-05T23:32:11.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 msgr2=0x7f858419e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:11.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 0x7f858419e480 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f857c007a60 tx=0x7f857c007930 comp rx=0 tx=0).stop 2024-04-05T23:32:11.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 shutdown_connections 2024-04-05T23:32:11.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f85680a1230 0x7f85680a36f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8584102a00 0x7f858419e480 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584100e60 0x7f8584196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 --2- 172.21.15.16:0/600576588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8584102040 0x7f8584197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:11.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 >> 172.21.15.16:0/600576588 conn(0x7f85840fc440 msgr2=0x7f85841053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:11.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 shutdown_connections 2024-04-05T23:32:11.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:11.955+0000 7f858a800700 1 -- 172.21.15.16:0/600576588 wait complete. 2024-04-05T23:32:11.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-05T23:32:12.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.180+0000 7fa03e000700 1 -- 172.21.15.16:0/4168637418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038102040 msgr2=0x7fa0381024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.180+0000 7fa03e000700 1 --2- 172.21.15.16:0/4168637418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038102040 0x7fa0381024c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa024009990 tx=0x7fa024009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.180+0000 7fa03e000700 1 -- 172.21.15.16:0/4168637418 shutdown_connections 2024-04-05T23:32:12.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.180+0000 7fa03e000700 1 --2- 172.21.15.16:0/4168637418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa038102a00 0x7fa038109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.180+0000 7fa03e000700 1 --2- 172.21.15.16:0/4168637418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa038100e60 0x7fa038101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.180+0000 7fa03e000700 1 --2- 172.21.15.16:0/4168637418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038102040 0x7fa0381024c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.180+0000 7fa03e000700 1 -- 172.21.15.16:0/4168637418 >> 172.21.15.16:0/4168637418 conn(0x7fa0380fc440 msgr2=0x7fa0380fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:12.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.180+0000 7fa03e000700 1 -- 172.21.15.16:0/4168637418 shutdown_connections 2024-04-05T23:32:12.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.180+0000 7fa03e000700 1 -- 172.21.15.16:0/4168637418 wait complete. 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.181+0000 7fa03e000700 1 Processor -- start 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.181+0000 7fa03e000700 1 -- start start 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.181+0000 7fa03e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038100e60 0x7fa038074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.181+0000 7fa03e000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa038102040 0x7fa0380732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.181+0000 7fa03e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa038102a00 0x7fa038073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.181+0000 7fa03e000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa038073fb0 con 0x7fa038100e60 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.181+0000 7fa03e000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa038074120 con 0x7fa038102a00 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.181+0000 7fa03e000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381a0ed0 con 0x7fa038102040 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038100e60 0x7fa038074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:12.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa036a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa038102040 0x7fa0380732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038100e60 0x7fa038074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48382/0 (socket says 172.21.15.16:48382) 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037400700 1 -- 172.21.15.16:0/590337981 learned_addr learned my addr 172.21.15.16:0/590337981 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037e00700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa038102a00 0x7fa038073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037400700 1 -- 172.21.15.16:0/590337981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa038102040 msgr2=0x7fa0380732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037400700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa038102040 0x7fa0380732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037400700 1 -- 172.21.15.16:0/590337981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa038102a00 msgr2=0x7fa038073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037400700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa038102a00 0x7fa038073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037400700 1 -- 172.21.15.16:0/590337981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0240095f0 con 0x7fa038100e60 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037e00700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa038102a00 0x7fa038073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa037400700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038100e60 0x7fa038074c40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa02800e970 tx=0x7fa02800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa02fe00700 1 -- 172.21.15.16:0/590337981 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa028009790 con 0x7fa038100e60 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa02fe00700 1 -- 172.21.15.16:0/590337981 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa02800eec0 con 0x7fa038100e60 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0381a1160 con 0x7fa038100e60 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.182+0000 7fa02fe00700 1 -- 172.21.15.16:0/590337981 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa028003c90 con 0x7fa038100e60 2024-04-05T23:32:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.183+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0381a1670 con 0x7fa038100e60 2024-04-05T23:32:12.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.184+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0381083b0 con 0x7fa038100e60 2024-04-05T23:32:12.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.186+0000 7fa02fe00700 1 -- 172.21.15.16:0/590337981 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fa02800cc70 con 0x7fa038100e60 2024-04-05T23:32:12.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.187+0000 7fa02fe00700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa01c0a11b0 0x7fa01c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:12.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.187+0000 7fa02fe00700 1 -- 172.21.15.16:0/590337981 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa028014070 con 0x7fa038100e60 2024-04-05T23:32:12.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.187+0000 7fa036a00700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa01c0a11b0 0x7fa01c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:12.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.188+0000 7fa036a00700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa01c0a11b0 0x7fa01c0a3670 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa02400b680 tx=0x7fa024005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:12.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.191+0000 7fa02fe00700 1 -- 172.21.15.16:0/590337981 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa0280962f0 con 0x7fa038100e60 2024-04-05T23:32:12.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.448+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fa038061320 con 0x7fa01c0a11b0 2024-04-05T23:32:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.471+0000 7fa02fe00700 1 -- 172.21.15.16:0/590337981 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa038061320 con 0x7fa01c0a11b0 2024-04-05T23:32:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa01c0a11b0 msgr2=0x7fa01c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa01c0a11b0 0x7fa01c0a3670 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa02400b680 tx=0x7fa024005fd0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038100e60 msgr2=0x7fa038074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038100e60 0x7fa038074c40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa02800e970 tx=0x7fa02800ec80 comp rx=0 tx=0).stop 2024-04-05T23:32:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 shutdown_connections 2024-04-05T23:32:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa01c0a11b0 0x7fa01c0a3670 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa038100e60 0x7fa038074c40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa038102a00 0x7fa038073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 --2- 172.21.15.16:0/590337981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa038102040 0x7fa0380732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 >> 172.21.15.16:0/590337981 conn(0x7fa0380fc440 msgr2=0x7fa0381054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 shutdown_connections 2024-04-05T23:32:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.474+0000 7fa03e000700 1 -- 172.21.15.16:0/590337981 wait complete. 2024-04-05T23:32:12.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-05T23:32:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.700+0000 7fb463800700 1 -- 172.21.15.16:0/3630095795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104250 msgr2=0x7fb45c106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.700+0000 7fb463800700 1 --2- 172.21.15.16:0/3630095795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104250 0x7fb45c106640 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb448009990 tx=0x7fb448009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.700+0000 7fb463800700 1 -- 172.21.15.16:0/3630095795 shutdown_connections 2024-04-05T23:32:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.700+0000 7fb463800700 1 --2- 172.21.15.16:0/3630095795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c106b80 0x7fb45c108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.700+0000 7fb463800700 1 --2- 172.21.15.16:0/3630095795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb45c101920 0x7fb45c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.700+0000 7fb463800700 1 --2- 172.21.15.16:0/3630095795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c104250 0x7fb45c106640 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.700+0000 7fb463800700 1 -- 172.21.15.16:0/3630095795 >> 172.21.15.16:0/3630095795 conn(0x7fb45c0fb330 msgr2=0x7fb45c0fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:12.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.700+0000 7fb463800700 1 -- 172.21.15.16:0/3630095795 shutdown_connections 2024-04-05T23:32:12.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.700+0000 7fb463800700 1 -- 172.21.15.16:0/3630095795 wait complete. 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.701+0000 7fb463800700 1 Processor -- start 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.701+0000 7fb463800700 1 -- start start 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.701+0000 7fb463800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c101920 0x7fb45c199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.701+0000 7fb463800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb45c104250 0x7fb45c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.701+0000 7fb463800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c106b80 0x7fb45c1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.701+0000 7fb463800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c1a0b90 con 0x7fb45c101920 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.701+0000 7fb463800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c1a0cd0 con 0x7fb45c106b80 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.701+0000 7fb463800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c1a0e60 con 0x7fb45c104250 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb460e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c101920 0x7fb45c199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:12.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb460e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c101920 0x7fb45c199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48404/0 (socket says 172.21.15.16:48404) 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb460e00700 1 -- 172.21.15.16:0/3200981216 learned_addr learned my addr 172.21.15.16:0/3200981216 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb461800700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c106b80 0x7fb45c1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb45be00700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb45c104250 0x7fb45c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb460e00700 1 -- 172.21.15.16:0/3200981216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb45c104250 msgr2=0x7fb45c1995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb460e00700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb45c104250 0x7fb45c1995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb460e00700 1 -- 172.21.15.16:0/3200981216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c106b80 msgr2=0x7fb45c1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb460e00700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c106b80 0x7fb45c1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb460e00700 1 -- 172.21.15.16:0/3200981216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480095f0 con 0x7fb45c101920 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.702+0000 7fb460e00700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c101920 0x7fb45c199060 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb45000d950 tx=0x7fb45000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.703+0000 7fb459600700 1 -- 172.21.15.16:0/3200981216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45000f890 con 0x7fb45c101920 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.703+0000 7fb459600700 1 -- 172.21.15.16:0/3200981216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb450010460 con 0x7fb45c101920 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.703+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c1a1140 con 0x7fb45c101920 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.703+0000 7fb459600700 1 -- 172.21.15.16:0/3200981216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45000e610 con 0x7fb45c101920 2024-04-05T23:32:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.703+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c1a1650 con 0x7fb45c101920 2024-04-05T23:32:12.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.704+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c108750 con 0x7fb45c101920 2024-04-05T23:32:12.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.706+0000 7fb459600700 1 -- 172.21.15.16:0/3200981216 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fb4500108d0 con 0x7fb45c101920 2024-04-05T23:32:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.707+0000 7fb459600700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4440a11b0 0x7fb4440a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.707+0000 7fb459600700 1 -- 172.21.15.16:0/3200981216 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb4500cc780 con 0x7fb45c101920 2024-04-05T23:32:12.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.707+0000 7fb45be00700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4440a11b0 0x7fb4440a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:12.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.708+0000 7fb45be00700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4440a11b0 0x7fb4440a3670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb44800b2e0 tx=0x7fb448005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:12.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.711+0000 7fb459600700 1 -- 172.21.15.16:0/3200981216 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb450095e60 con 0x7fb45c101920 2024-04-05T23:32:12.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:12 smithi027 ceph-mon[78597]: from='client.14625 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:12.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:12 smithi027 ceph-mon[78597]: pgmap v236: 97 pgs: 97 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 840 B/s rd, 5.9 KiB/s wr, 14 op/s 2024-04-05T23:32:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:12 smithi027 ceph-mon[78597]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:12 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] up:active 2024-04-05T23:32:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:12 smithi027 ceph-mon[78597]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-05T23:32:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:12 smithi027 ceph-mon[78597]: mgrmap e17: x(active, since 5m), standbys: y, z 2024-04-05T23:32:12.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:12 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:32:12.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:12 smithi016 ceph-mon[38248]: from='client.14625 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:12.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:12 smithi016 ceph-mon[38248]: pgmap v236: 97 pgs: 97 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 840 B/s rd, 5.9 KiB/s wr, 14 op/s 2024-04-05T23:32:12.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:12 smithi016 ceph-mon[38248]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:12.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:12 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] up:active 2024-04-05T23:32:12.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:12 smithi016 ceph-mon[38248]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-05T23:32:12.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:12 smithi016 ceph-mon[38248]: mgrmap e17: x(active, since 5m), standbys: y, z 2024-04-05T23:32:12.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:12 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:32:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:12 smithi130 ceph-mon[79417]: from='client.14625 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:12 smithi130 ceph-mon[79417]: pgmap v236: 97 pgs: 97 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 840 B/s rd, 5.9 KiB/s wr, 14 op/s 2024-04-05T23:32:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:12 smithi130 ceph-mon[79417]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:12 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] up:active 2024-04-05T23:32:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:12 smithi130 ceph-mon[79417]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-05T23:32:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:12 smithi130 ceph-mon[79417]: mgrmap e17: x(active, since 5m), standbys: y, z 2024-04-05T23:32:12.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:12 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:32:12.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.968+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb45c061320 con 0x7fb4440a11b0 2024-04-05T23:32:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.972+0000 7fb459600700 1 -- 172.21.15.16:0/3200981216 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fb45c061320 con 0x7fb4440a11b0 2024-04-05T23:32:12.965 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa 2024-04-05T23:32:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4440a11b0 msgr2=0x7fb4440a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4440a11b0 0x7fb4440a3670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb44800b2e0 tx=0x7fb448005fb0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c101920 msgr2=0x7fb45c199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c101920 0x7fb45c199060 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb45000d950 tx=0x7fb45000dc60 comp rx=0 tx=0).stop 2024-04-05T23:32:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 shutdown_connections 2024-04-05T23:32:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4440a11b0 0x7fb4440a3670 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c101920 0x7fb45c199060 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c106b80 0x7fb45c1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 --2- 172.21.15.16:0/3200981216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb45c104250 0x7fb45c1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 >> 172.21.15.16:0/3200981216 conn(0x7fb45c0fb330 msgr2=0x7fb45c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 shutdown_connections 2024-04-05T23:32:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:12.975+0000 7fb463800700 1 -- 172.21.15.16:0/3200981216 wait complete. 2024-04-05T23:32:12.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:32:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.191+0000 7f1711200700 1 -- 172.21.15.16:0/3803361060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 msgr2=0x7f170c102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.191+0000 7f1711200700 1 --2- 172.21.15.16:0/3803361060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 0x7f170c102400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f16f80099c0 tx=0x7f16f8009cd0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.191+0000 7f1711200700 1 -- 172.21.15.16:0/3803361060 shutdown_connections 2024-04-05T23:32:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.191+0000 7f1711200700 1 --2- 172.21.15.16:0/3803361060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f170c100db0 0x7f170c1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.191+0000 7f1711200700 1 --2- 172.21.15.16:0/3803361060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f170c102940 0x7f170c1091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.191+0000 7f1711200700 1 --2- 172.21.15.16:0/3803361060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 0x7f170c102400 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.191+0000 7f1711200700 1 -- 172.21.15.16:0/3803361060 >> 172.21.15.16:0/3803361060 conn(0x7f170c0fc320 msgr2=0x7f170c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.191+0000 7f1711200700 1 -- 172.21.15.16:0/3803361060 shutdown_connections 2024-04-05T23:32:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.192+0000 7f1711200700 1 -- 172.21.15.16:0/3803361060 wait complete. 2024-04-05T23:32:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.192+0000 7f1711200700 1 Processor -- start 2024-04-05T23:32:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1711200700 1 -- start start 2024-04-05T23:32:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1711200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f170c100db0 0x7f170c198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1711200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 0x7f170c1994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1711200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f170c102940 0x7f170c1a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1711200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c1a0ac0 con 0x7f170c102940 2024-04-05T23:32:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1711200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c1a0c00 con 0x7f170c101fa0 2024-04-05T23:32:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1711200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c1a0d90 con 0x7f170c100db0 2024-04-05T23:32:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f170a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f170c102940 0x7f170c1a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1709400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 0x7f170c1994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1709400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 0x7f170c1994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:46914/0 (socket says 172.21.15.16:46914) 2024-04-05T23:32:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f170a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f170c102940 0x7f170c1a04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48424/0 (socket says 172.21.15.16:48424) 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1709400700 1 -- 172.21.15.16:0/3552222405 learned_addr learned my addr 172.21.15.16:0/3552222405 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.193+0000 7f1709e00700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f170c100db0 0x7f170c198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1709400700 1 -- 172.21.15.16:0/3552222405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f170c100db0 msgr2=0x7f170c198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1709400700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f170c100db0 0x7f170c198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1709400700 1 -- 172.21.15.16:0/3552222405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f170c102940 msgr2=0x7f170c1a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1709400700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f170c102940 0x7f170c1a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1709400700 1 -- 172.21.15.16:0/3552222405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80095f0 con 0x7f170c101fa0 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1709400700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 0x7f170c1994d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f16f8003730 tx=0x7f16f800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1702a00700 1 -- 172.21.15.16:0/3552222405 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f801d070 con 0x7f170c101fa0 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1702a00700 1 -- 172.21.15.16:0/3552222405 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16f800bda0 con 0x7f170c101fa0 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c1a1010 con 0x7f170c101fa0 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1702a00700 1 -- 172.21.15.16:0/3552222405 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8004bd0 con 0x7f170c101fa0 2024-04-05T23:32:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.194+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c1a1430 con 0x7f170c101fa0 2024-04-05T23:32:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.195+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c067b50 con 0x7f170c101fa0 2024-04-05T23:32:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.197+0000 7f1702a00700 1 -- 172.21.15.16:0/3552222405 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f16f8004d30 con 0x7f170c101fa0 2024-04-05T23:32:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.198+0000 7f1702a00700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16f00a11b0 0x7f16f00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.199+0000 7f1702a00700 1 -- 172.21.15.16:0/3552222405 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f16f80cdad0 con 0x7f170c101fa0 2024-04-05T23:32:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.199+0000 7f1709e00700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16f00a11b0 0x7f16f00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.199+0000 7f1709e00700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16f00a11b0 0x7f16f00a3670 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f170c101e00 tx=0x7f16fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.201+0000 7f1702a00700 1 -- 172.21.15.16:0/3552222405 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f16f8097340 con 0x7f170c101fa0 2024-04-05T23:32:13.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.473+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f170c10c610 con 0x7f170c101fa0 2024-04-05T23:32:13.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.474+0000 7f1702a00700 1 -- 172.21.15.16:0/3552222405 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10902 (secure 0 0 0) 0x7f16f809a960 con 0x7f170c101fa0 2024-04-05T23:32:13.466 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:13.466 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:32:11.436182+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:32:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16f00a11b0 msgr2=0x7f16f00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16f00a11b0 0x7f16f00a3670 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f170c101e00 tx=0x7f16fc009260 comp rx=0 tx=0).stop 2024-04-05T23:32:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 msgr2=0x7f170c1994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 0x7f170c1994d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f16f8003730 tx=0x7f16f800fa00 comp rx=0 tx=0).stop 2024-04-05T23:32:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 shutdown_connections 2024-04-05T23:32:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f16f00a11b0 0x7f16f00a3670 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f170c102940 0x7f170c1a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f170c100db0 0x7f170c198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 --2- 172.21.15.16:0/3552222405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c101fa0 0x7f170c1994d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 >> 172.21.15.16:0/3552222405 conn(0x7f170c0fc320 msgr2=0x7f170c073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 shutdown_connections 2024-04-05T23:32:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.477+0000 7f1711200700 1 -- 172.21.15.16:0/3552222405 wait complete. 2024-04-05T23:32:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-04-05T23:32:13.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:32:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.694+0000 7f6d06000700 1 -- 172.21.15.16:0/3616576280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d00101fa0 msgr2=0x7f6d00102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.694+0000 7f6d06000700 1 --2- 172.21.15.16:0/3616576280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d00101fa0 0x7f6d00102400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6cec009990 tx=0x7f6cec009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.694+0000 7f6d06000700 1 -- 172.21.15.16:0/3616576280 shutdown_connections 2024-04-05T23:32:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.694+0000 7f6d06000700 1 --2- 172.21.15.16:0/3616576280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d00102940 0x7f6d001091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.694+0000 7f6d06000700 1 --2- 172.21.15.16:0/3616576280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d00100db0 0x7f6d001011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.694+0000 7f6d06000700 1 --2- 172.21.15.16:0/3616576280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d00101fa0 0x7f6d00102400 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.694+0000 7f6d06000700 1 -- 172.21.15.16:0/3616576280 >> 172.21.15.16:0/3616576280 conn(0x7f6d000fc320 msgr2=0x7f6d000fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.694+0000 7f6d06000700 1 -- 172.21.15.16:0/3616576280 shutdown_connections 2024-04-05T23:32:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.694+0000 7f6d06000700 1 -- 172.21.15.16:0/3616576280 wait complete. 2024-04-05T23:32:13.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.695+0000 7f6d06000700 1 Processor -- start 2024-04-05T23:32:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.695+0000 7f6d06000700 1 -- start start 2024-04-05T23:32:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6d06000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d00100db0 0x7f6d00198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6d06000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d00101fa0 0x7f6d00199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6d06000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d00102940 0x7f6d001a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6d06000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001a0ac0 con 0x7f6d00100db0 2024-04-05T23:32:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6d06000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001a0c00 con 0x7f6d00101fa0 2024-04-05T23:32:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6d06000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001a0d90 con 0x7f6d00102940 2024-04-05T23:32:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cff400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d00100db0 0x7f6d00198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:13.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cff400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d00100db0 0x7f6d00198fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48444/0 (socket says 172.21.15.16:48444) 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cff400700 1 -- 172.21.15.16:0/4205277373 learned_addr learned my addr 172.21.15.16:0/4205277373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cfea00700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d00101fa0 0x7f6d00199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cffe00700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d00102940 0x7f6d001a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cff400700 1 -- 172.21.15.16:0/4205277373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d00102940 msgr2=0x7f6d001a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cff400700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d00102940 0x7f6d001a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cff400700 1 -- 172.21.15.16:0/4205277373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d00101fa0 msgr2=0x7f6d00199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cff400700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d00101fa0 0x7f6d00199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.696+0000 7f6cff400700 1 -- 172.21.15.16:0/4205277373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0095f0 con 0x7f6d00100db0 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.697+0000 7f6cff400700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d00100db0 0x7f6d00198fe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e970 tx=0x7f6cf000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:13.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.697+0000 7f6cf7e00700 1 -- 172.21.15.16:0/4205277373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0009790 con 0x7f6d00100db0 2024-04-05T23:32:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.697+0000 7f6cf7e00700 1 -- 172.21.15.16:0/4205277373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6cf000c7e0 con 0x7f6d00100db0 2024-04-05T23:32:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.697+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d001a1070 con 0x7f6d00100db0 2024-04-05T23:32:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.697+0000 7f6cf7e00700 1 -- 172.21.15.16:0/4205277373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0003c70 con 0x7f6d00100db0 2024-04-05T23:32:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.697+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d001a1580 con 0x7f6d00100db0 2024-04-05T23:32:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.698+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d00067b50 con 0x7f6d00100db0 2024-04-05T23:32:13.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.700+0000 7f6cf7e00700 1 -- 172.21.15.16:0/4205277373 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f6cf000cc50 con 0x7f6d00100db0 2024-04-05T23:32:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.701+0000 7f6cf7e00700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6ce40a11b0 0x7f6ce40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.701+0000 7f6cf7e00700 1 -- 172.21.15.16:0/4205277373 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6cf0014070 con 0x7f6d00100db0 2024-04-05T23:32:13.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.702+0000 7f6cfea00700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6ce40a11b0 0x7f6ce40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:13.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.702+0000 7f6cfea00700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6ce40a11b0 0x7f6ce40a3670 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00b2e0 tx=0x7f6cec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:13.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.705+0000 7f6cf7e00700 1 -- 172.21.15.16:0/4205277373 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6cf0096270 con 0x7f6d00100db0 2024-04-05T23:32:13.716 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:32:13 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-b[116618]: 2024-04-05T23:32:13.223+0000 7ff979be4700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-05T23:32:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:13 smithi027 ceph-mon[78597]: from='client.14643 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:13.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:13 smithi027 ceph-mon[78597]: from='client.14649 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:13.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:13 smithi016 ceph-mon[38248]: from='client.14643 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:13.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:13 smithi016 ceph-mon[38248]: from='client.14649 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:13.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:13 smithi130 ceph-mon[79417]: from='client.14643 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:13.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:13 smithi130 ceph-mon[79417]: from='client.14649 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:13.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.924+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d0010c610 con 0x7f6d00100db0 2024-04-05T23:32:13.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.925+0000 7f6cf7e00700 1 -- 172.21.15.16:0/4205277373 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+22084 (secure 0 0 0) 0x7f6cf0003dd0 con 0x7f6d00100db0 2024-04-05T23:32:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:13.918 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":79,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:05.514031+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:32:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6ce40a11b0 msgr2=0x7f6ce40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6ce40a11b0 0x7f6ce40a3670 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00b2e0 tx=0x7f6cec005fd0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d00100db0 msgr2=0x7f6d00198fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d00100db0 0x7f6d00198fe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e970 tx=0x7f6cf000ec80 comp rx=0 tx=0).stop 2024-04-05T23:32:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 shutdown_connections 2024-04-05T23:32:13.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f6ce40a11b0 0x7f6ce40a3670 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d00100db0 0x7f6d00198fe0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d00102940 0x7f6d001a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 --2- 172.21.15.16:0/4205277373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d00101fa0 0x7f6d00199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 >> 172.21.15.16:0/4205277373 conn(0x7f6d000fc320 msgr2=0x7f6d00073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 shutdown_connections 2024-04-05T23:32:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:13.928+0000 7f6d06000700 1 -- 172.21.15.16:0/4205277373 wait complete. 2024-04-05T23:32:13.939 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-05T23:32:13.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-05T23:32:14.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.145+0000 7fce43400700 1 -- 172.21.15.16:0/3650553458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce3c102940 msgr2=0x7fce3c1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.145+0000 7fce43400700 1 --2- 172.21.15.16:0/3650553458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce3c102940 0x7fce3c1091f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fce3400b0a0 tx=0x7fce3400b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.145+0000 7fce43400700 1 -- 172.21.15.16:0/3650553458 shutdown_connections 2024-04-05T23:32:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.145+0000 7fce43400700 1 --2- 172.21.15.16:0/3650553458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce3c100db0 0x7fce3c1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.145+0000 7fce43400700 1 --2- 172.21.15.16:0/3650553458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce3c102940 0x7fce3c1091f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.145+0000 7fce43400700 1 --2- 172.21.15.16:0/3650553458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c101fa0 0x7fce3c102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.145+0000 7fce43400700 1 -- 172.21.15.16:0/3650553458 >> 172.21.15.16:0/3650553458 conn(0x7fce3c0fc320 msgr2=0x7fce3c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.146+0000 7fce43400700 1 -- 172.21.15.16:0/3650553458 shutdown_connections 2024-04-05T23:32:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.146+0000 7fce43400700 1 -- 172.21.15.16:0/3650553458 wait complete. 2024-04-05T23:32:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.146+0000 7fce43400700 1 Processor -- start 2024-04-05T23:32:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce43400700 1 -- start start 2024-04-05T23:32:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce43400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce3c100db0 0x7fce3c1990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce43400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce3c101fa0 0x7fce3c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce43400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c102940 0x7fce3c1a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce43400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c1a0bd0 con 0x7fce3c100db0 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce43400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c1a0d10 con 0x7fce3c102940 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce43400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c1a0ea0 con 0x7fce3c101fa0 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce40a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce3c100db0 0x7fce3c1990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce40a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce3c100db0 0x7fce3c1990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48470/0 (socket says 172.21.15.16:48470) 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce40a00700 1 -- 172.21.15.16:0/3978412661 learned_addr learned my addr 172.21.15.16:0/3978412661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce41400700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c102940 0x7fce3c1a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.147+0000 7fce3be00700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce3c101fa0 0x7fce3c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce40a00700 1 -- 172.21.15.16:0/3978412661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce3c101fa0 msgr2=0x7fce3c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce40a00700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce3c101fa0 0x7fce3c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce40a00700 1 -- 172.21.15.16:0/3978412661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c102940 msgr2=0x7fce3c1a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce40a00700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c102940 0x7fce3c1a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce40a00700 1 -- 172.21.15.16:0/3978412661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce34009d00 con 0x7fce3c100db0 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce41400700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c102940 0x7fce3c1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce40a00700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce3c100db0 0x7fce3c1990a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fce3000e970 tx=0x7fce3000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce39600700 1 -- 172.21.15.16:0/3978412661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30009790 con 0x7fce3c100db0 2024-04-05T23:32:14.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce39600700 1 -- 172.21.15.16:0/3978412661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce3000c7e0 con 0x7fce3c100db0 2024-04-05T23:32:14.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3c1a1180 con 0x7fce3c100db0 2024-04-05T23:32:14.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce39600700 1 -- 172.21.15.16:0/3978412661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30003c70 con 0x7fce3c100db0 2024-04-05T23:32:14.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.148+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3c1a1690 con 0x7fce3c100db0 2024-04-05T23:32:14.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.149+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3c067b50 con 0x7fce3c100db0 2024-04-05T23:32:14.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.151+0000 7fce39600700 1 -- 172.21.15.16:0/3978412661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fce3000cc50 con 0x7fce3c100db0 2024-04-05T23:32:14.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.152+0000 7fce39600700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce240a11b0 0x7fce240a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.153+0000 7fce39600700 1 -- 172.21.15.16:0/3978412661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fce30014070 con 0x7fce3c100db0 2024-04-05T23:32:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.153+0000 7fce3be00700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce240a11b0 0x7fce240a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.153+0000 7fce3be00700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce240a11b0 0x7fce240a3670 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fce28009c60 tx=0x7fce28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.154+0000 7fce39600700 1 -- 172.21.15.16:0/3978412661 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fce300963e0 con 0x7fce3c100db0 2024-04-05T23:32:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.371+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce3c074830 con 0x7fce240a11b0 2024-04-05T23:32:14.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.383+0000 7fce39600700 1 -- 172.21.15.16:0/3978412661 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+333654 (secure 0 0 0) 0x7fce3c074830 con 0x7fce240a11b0 2024-04-05T23:32:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce240a11b0 msgr2=0x7fce240a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce240a11b0 0x7fce240a3670 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fce28009c60 tx=0x7fce28009260 comp rx=0 tx=0).stop 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce3c100db0 msgr2=0x7fce3c1990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce3c100db0 0x7fce3c1990a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fce3000e970 tx=0x7fce3000ec80 comp rx=0 tx=0).stop 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 shutdown_connections 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce240a11b0 0x7fce240a3670 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce3c100db0 0x7fce3c1990a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c102940 0x7fce3c1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 --2- 172.21.15.16:0/3978412661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce3c101fa0 0x7fce3c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 >> 172.21.15.16:0/3978412661 conn(0x7fce3c0fc320 msgr2=0x7fce3c073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 shutdown_connections 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.388+0000 7fce43400700 1 -- 172.21.15.16:0/3978412661 wait complete. 2024-04-05T23:32:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-05T23:32:14.399 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-04-05T23:32:13.293797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616802,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":174,"num_read_kb":153,"num_write":357,"num_write_kb":3019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364896,"kb_used_data":12028,"kb_used_omap":18,"kb_used_meta":352749,"kb_avail":1124576928,"statfs":{"total":1151940427776,"available":1151566774272,"internally_reserved":0,"allocated":12316672,"data_stored":6759210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":361215301},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26434,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":38,"num_read_kb":34,"num_write":176,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.779254"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561860+0000","last_change":"2024-04-05T23:32:04.536879+0000","last_active":"2024-04-05T23:32:05.561860+0000","last_peered":"2024-04-05T23:32:05.561860+0000","last_clean":"2024-04-05T23:32:05.561860+0000","last_became_active":"2024-04-05T23:32:04.536183+0000","last_became_peered":"2024-04-05T23:32:04.536183+0000","last_unstale":"2024-04-05T23:32:05.561860+0000","last_undegraded":"2024-04-05T23:32:05.561860+0000","last_fullsized":"2024-04-05T23:32:05.561860+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:13:22.589464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561585+0000","last_change":"2024-04-05T23:32:04.540791+0000","last_active":"2024-04-05T23:32:05.561585+0000","last_peered":"2024-04-05T23:32:05.561585+0000","last_clean":"2024-04-05T23:32:05.561585+0000","last_became_active":"2024-04-05T23:32:04.540455+0000","last_became_peered":"2024-04-05T23:32:04.540455+0000","last_unstale":"2024-04-05T23:32:05.561585+0000","last_undegraded":"2024-04-05T23:32:05.561585+0000","last_fullsized":"2024-04-05T23:32:05.561585+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:06:19.090672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561467+0000","last_change":"2024-04-05T23:32:04.543428+0000","last_active":"2024-04-05T23:32:05.561467+0000","last_peered":"2024-04-05T23:32:05.561467+0000","last_clean":"2024-04-05T23:32:05.561467+0000","last_became_active":"2024-04-05T23:32:04.543071+0000","last_became_peered":"2024-04-05T23:32:04.543071+0000","last_unstale":"2024-04-05T23:32:05.561467+0000","last_undegraded":"2024-04-05T23:32:05.561467+0000","last_fullsized":"2024-04-05T23:32:05.561467+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:35:44.911749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561794+0000","last_change":"2024-04-05T23:32:04.531444+0000","last_active":"2024-04-05T23:32:05.561794+0000","last_peered":"2024-04-05T23:32:05.561794+0000","last_clean":"2024-04-05T23:32:05.561794+0000","last_became_active":"2024-04-05T23:32:04.529556+0000","last_became_peered":"2024-04-05T23:32:04.529556+0000","last_unstale":"2024-04-05T23:32:05.561794+0000","last_undegraded":"2024-04-05T23:32:05.561794+0000","last_fullsized":"2024-04-05T23:32:05.561794+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:48.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.537368+0000","last_change":"2024-04-05T23:32:04.529565+0000","last_active":"2024-04-05T23:32:04.537368+0000","last_peered":"2024-04-05T23:32:04.537368+0000","last_clean":"2024-04-05T23:32:04.537368+0000","last_became_active":"2024-04-05T23:32:04.529226+0000","last_became_peered":"2024-04-05T23:32:04.529226+0000","last_unstale":"2024-04-05T23:32:04.537368+0000","last_undegraded":"2024-04-05T23:32:04.537368+0000","last_fullsized":"2024-04-05T23:32:04.537368+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:44:25.208996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562159+0000","last_change":"2024-04-05T23:32:04.546260+0000","last_active":"2024-04-05T23:32:05.562159+0000","last_peered":"2024-04-05T23:32:05.562159+0000","last_clean":"2024-04-05T23:32:05.562159+0000","last_became_active":"2024-04-05T23:32:04.545851+0000","last_became_peered":"2024-04-05T23:32:04.545851+0000","last_unstale":"2024-04-05T23:32:05.562159+0000","last_undegraded":"2024-04-05T23:32:05.562159+0000","last_fullsized":"2024-04-05T23:32:05.562159+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:59:16.735635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.560777+0000","last_change":"2024-04-05T23:32:04.536348+0000","last_active":"2024-04-05T23:32:05.560777+0000","last_peered":"2024-04-05T23:32:05.560777+0000","last_clean":"2024-04-05T23:32:05.560777+0000","last_became_active":"2024-04-05T23:32:04.536015+0000","last_became_peered":"2024-04-05T23:32:04.536015+0000","last_unstale":"2024-04-05T23:32:05.560777+0000","last_undegraded":"2024-04-05T23:32:05.560777+0000","last_fullsized":"2024-04-05T23:32:05.560777+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:02:24.897957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522127+0000","last_change":"2024-04-05T23:32:04.532528+0000","last_active":"2024-04-05T23:32:05.522127+0000","last_peered":"2024-04-05T23:32:05.522127+0000","last_clean":"2024-04-05T23:32:05.522127+0000","last_became_active":"2024-04-05T23:32:04.532104+0000","last_became_peered":"2024-04-05T23:32:04.532104+0000","last_unstale":"2024-04-05T23:32:05.522127+0000","last_undegraded":"2024-04-05T23:32:05.522127+0000","last_fullsized":"2024-04-05T23:32:05.522127+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:06.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390633+0000","last_change":"2024-04-05T23:32:04.531223+0000","last_active":"2024-04-05T23:32:05.390633+0000","last_peered":"2024-04-05T23:32:05.390633+0000","last_clean":"2024-04-05T23:32:05.390633+0000","last_became_active":"2024-04-05T23:32:04.530863+0000","last_became_peered":"2024-04-05T23:32:04.530863+0000","last_unstale":"2024-04-05T23:32:05.390633+0000","last_undegraded":"2024-04-05T23:32:05.390633+0000","last_fullsized":"2024-04-05T23:32:05.390633+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:03:35.912984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562452+0000","last_change":"2024-04-05T23:32:04.541481+0000","last_active":"2024-04-05T23:32:05.562452+0000","last_peered":"2024-04-05T23:32:05.562452+0000","last_clean":"2024-04-05T23:32:05.562452+0000","last_became_active":"2024-04-05T23:32:04.541133+0000","last_became_peered":"2024-04-05T23:32:04.541133+0000","last_unstale":"2024-04-05T23:32:05.562452+0000","last_undegraded":"2024-04-05T23:32:05.562452+0000","last_fullsized":"2024-04-05T23:32:05.562452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:01:22.825587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562374+0000","last_change":"2024-04-05T23:32:04.542390+0000","last_active":"2024-04-05T23:32:05.562374+0000","last_peered":"2024-04-05T23:32:05.562374+0000","last_clean":"2024-04-05T23:32:05.562374+0000","last_became_active":"2024-04-05T23:32:04.542026+0000","last_became_peered":"2024-04-05T23:32:04.542026+0000","last_unstale":"2024-04-05T23:32:05.562374+0000","last_undegraded":"2024-04-05T23:32:05.562374+0000","last_fullsized":"2024-04-05T23:32:05.562374+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:15:42.243807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561754+0000","last_change":"2024-04-05T23:32:04.540096+0000","last_active":"2024-04-05T23:32:05.561754+0000","last_peered":"2024-04-05T23:32:05.561754+0000","last_clean":"2024-04-05T23:32:05.561754+0000","last_became_active":"2024-04-05T23:32:04.539764+0000","last_became_peered":"2024-04-05T23:32:04.539764+0000","last_unstale":"2024-04-05T23:32:05.561754+0000","last_undegraded":"2024-04-05T23:32:05.561754+0000","last_fullsized":"2024-04-05T23:32:05.561754+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:57:36.442393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561070+0000","last_change":"2024-04-05T23:32:04.540839+0000","last_active":"2024-04-05T23:32:05.561070+0000","last_peered":"2024-04-05T23:32:05.561070+0000","last_clean":"2024-04-05T23:32:05.561070+0000","last_became_active":"2024-04-05T23:32:04.540511+0000","last_became_peered":"2024-04-05T23:32:04.540511+0000","last_unstale":"2024-04-05T23:32:05.561070+0000","last_undegraded":"2024-04-05T23:32:05.561070+0000","last_fullsized":"2024-04-05T23:32:05.561070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:58:24.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561331+0000","last_change":"2024-04-05T23:32:04.539222+0000","last_active":"2024-04-05T23:32:05.561331+0000","last_peered":"2024-04-05T23:32:05.561331+0000","last_clean":"2024-04-05T23:32:05.561331+0000","last_became_active":"2024-04-05T23:32:04.538906+0000","last_became_peered":"2024-04-05T23:32:04.538906+0000","last_unstale":"2024-04-05T23:32:05.561331+0000","last_undegraded":"2024-04-05T23:32:05.561331+0000","last_fullsized":"2024-04-05T23:32:05.561331+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:05:24.429153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561895+0000","last_change":"2024-04-05T23:32:04.544155+0000","last_active":"2024-04-05T23:32:05.561895+0000","last_peered":"2024-04-05T23:32:05.561895+0000","last_clean":"2024-04-05T23:32:05.561895+0000","last_became_active":"2024-04-05T23:32:04.543852+0000","last_became_peered":"2024-04-05T23:32:04.543852+0000","last_unstale":"2024-04-05T23:32:05.561895+0000","last_undegraded":"2024-04-05T23:32:05.561895+0000","last_fullsized":"2024-04-05T23:32:05.561895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:18:34.353542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390483+0000","last_change":"2024-04-05T23:32:04.533772+0000","last_active":"2024-04-05T23:32:05.390483+0000","last_peered":"2024-04-05T23:32:05.390483+0000","last_clean":"2024-04-05T23:32:05.390483+0000","last_became_active":"2024-04-05T23:32:04.533416+0000","last_became_peered":"2024-04-05T23:32:04.533416+0000","last_unstale":"2024-04-05T23:32:05.390483+0000","last_undegraded":"2024-04-05T23:32:05.390483+0000","last_fullsized":"2024-04-05T23:32:05.390483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:36:55.072349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561961+0000","last_change":"2024-04-05T23:32:04.551902+0000","last_active":"2024-04-05T23:32:05.561961+0000","last_peered":"2024-04-05T23:32:05.561961+0000","last_clean":"2024-04-05T23:32:05.561961+0000","last_became_active":"2024-04-05T23:32:04.551595+0000","last_became_peered":"2024-04-05T23:32:04.551595+0000","last_unstale":"2024-04-05T23:32:05.561961+0000","last_undegraded":"2024-04-05T23:32:05.561961+0000","last_fullsized":"2024-04-05T23:32:05.561961+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:14:46.648789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.546726+0000","last_change":"2024-04-05T23:32:04.542967+0000","last_active":"2024-04-05T23:32:04.546726+0000","last_peered":"2024-04-05T23:32:04.546726+0000","last_clean":"2024-04-05T23:32:04.546726+0000","last_became_active":"2024-04-05T23:32:04.541767+0000","last_became_peered":"2024-04-05T23:32:04.541767+0000","last_unstale":"2024-04-05T23:32:04.546726+0000","last_undegraded":"2024-04-05T23:32:04.546726+0000","last_fullsized":"2024-04-05T23:32:04.546726+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:27:46.150249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561256+0000","last_change":"2024-04-05T23:32:04.549512+0000","last_active":"2024-04-05T23:32:05.561256+0000","last_peered":"2024-04-05T23:32:05.561256+0000","last_clean":"2024-04-05T23:32:05.561256+0000","last_became_active":"2024-04-05T23:32:04.549184+0000","last_became_peered":"2024-04-05T23:32:04.549184+0000","last_unstale":"2024-04-05T23:32:05.561256+0000","last_undegraded":"2024-04-05T23:32:05.561256+0000","last_fullsized":"2024-04-05T23:32:05.561256+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:48:24.186408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.014599+0000","last_change":"2024-04-05T23:32:04.005188+0000","last_active":"2024-04-05T23:32:09.014599+0000","last_peered":"2024-04-05T23:32:09.014599+0000","last_clean":"2024-04-05T23:32:09.014599+0000","last_became_active":"2024-04-05T23:32:04.004818+0000","last_became_peered":"2024-04-05T23:32:04.004818+0000","last_unstale":"2024-04-05T23:32:09.014599+0000","last_undegraded":"2024-04-05T23:32:09.014599+0000","last_fullsized":"2024-04-05T23:32:09.014599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:49:41.146761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522767+0000","last_change":"2024-04-05T23:32:04.544881+0000","last_active":"2024-04-05T23:32:05.522767+0000","last_peered":"2024-04-05T23:32:05.522767+0000","last_clean":"2024-04-05T23:32:05.522767+0000","last_became_active":"2024-04-05T23:32:04.542149+0000","last_became_peered":"2024-04-05T23:32:04.542149+0000","last_unstale":"2024-04-05T23:32:05.522767+0000","last_undegraded":"2024-04-05T23:32:05.522767+0000","last_fullsized":"2024-04-05T23:32:05.522767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:28:51.713053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.549110+0000","last_change":"2024-04-05T23:32:04.011450+0000","last_active":"2024-04-05T23:32:04.549110+0000","last_peered":"2024-04-05T23:32:04.549110+0000","last_clean":"2024-04-05T23:32:04.549110+0000","last_became_active":"2024-04-05T23:32:04.009607+0000","last_became_peered":"2024-04-05T23:32:04.009607+0000","last_unstale":"2024-04-05T23:32:04.549110+0000","last_undegraded":"2024-04-05T23:32:04.549110+0000","last_fullsized":"2024-04-05T23:32:04.549110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:16:45.483108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561707+0000","last_change":"2024-04-05T23:32:04.550370+0000","last_active":"2024-04-05T23:32:05.561707+0000","last_peered":"2024-04-05T23:32:05.561707+0000","last_clean":"2024-04-05T23:32:05.561707+0000","last_became_active":"2024-04-05T23:32:04.549980+0000","last_became_peered":"2024-04-05T23:32:04.549980+0000","last_unstale":"2024-04-05T23:32:05.561707+0000","last_undegraded":"2024-04-05T23:32:05.561707+0000","last_fullsized":"2024-04-05T23:32:05.561707+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:44:30.704366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.011377+0000","last_change":"2024-04-05T23:32:04.004169+0000","last_active":"2024-04-05T23:32:09.011377+0000","last_peered":"2024-04-05T23:32:09.011377+0000","last_clean":"2024-04-05T23:32:09.011377+0000","last_became_active":"2024-04-05T23:32:04.003646+0000","last_became_peered":"2024-04-05T23:32:04.003646+0000","last_unstale":"2024-04-05T23:32:09.011377+0000","last_undegraded":"2024-04-05T23:32:09.011377+0000","last_fullsized":"2024-04-05T23:32:09.011377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:57:02.781815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390563+0000","last_change":"2024-04-05T23:32:04.542066+0000","last_active":"2024-04-05T23:32:05.390563+0000","last_peered":"2024-04-05T23:32:05.390563+0000","last_clean":"2024-04-05T23:32:05.390563+0000","last_became_active":"2024-04-05T23:32:04.540735+0000","last_became_peered":"2024-04-05T23:32:04.540735+0000","last_unstale":"2024-04-05T23:32:05.390563+0000","last_undegraded":"2024-04-05T23:32:05.390563+0000","last_fullsized":"2024-04-05T23:32:05.390563+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:02:03.595888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.544085+0000","last_change":"2024-04-05T23:32:03.999943+0000","last_active":"2024-04-05T23:32:04.544085+0000","last_peered":"2024-04-05T23:32:04.544085+0000","last_clean":"2024-04-05T23:32:04.544085+0000","last_became_active":"2024-04-05T23:32:03.999518+0000","last_became_peered":"2024-04-05T23:32:03.999518+0000","last_unstale":"2024-04-05T23:32:04.544085+0000","last_undegraded":"2024-04-05T23:32:04.544085+0000","last_fullsized":"2024-04-05T23:32:04.544085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:27:52.486637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346372+0000","last_change":"2024-04-05T23:32:04.545780+0000","last_active":"2024-04-05T23:32:05.346372+0000","last_peered":"2024-04-05T23:32:05.346372+0000","last_clean":"2024-04-05T23:32:05.346372+0000","last_became_active":"2024-04-05T23:32:04.545279+0000","last_became_peered":"2024-04-05T23:32:04.545279+0000","last_unstale":"2024-04-05T23:32:05.346372+0000","last_undegraded":"2024-04-05T23:32:05.346372+0000","last_fullsized":"2024-04-05T23:32:05.346372+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:10.577280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.411051+0000","last_change":"2024-04-05T23:32:04.004189+0000","last_active":"2024-04-05T23:32:09.411051+0000","last_peered":"2024-04-05T23:32:09.411051+0000","last_clean":"2024-04-05T23:32:09.411051+0000","last_became_active":"2024-04-05T23:32:04.003841+0000","last_became_peered":"2024-04-05T23:32:04.003841+0000","last_unstale":"2024-04-05T23:32:09.411051+0000","last_undegraded":"2024-04-05T23:32:09.411051+0000","last_fullsized":"2024-04-05T23:32:09.411051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:33:26.898657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.537327+0000","last_change":"2024-04-05T23:32:04.529538+0000","last_active":"2024-04-05T23:32:04.537327+0000","last_peered":"2024-04-05T23:32:04.537327+0000","last_clean":"2024-04-05T23:32:04.537327+0000","last_became_active":"2024-04-05T23:32:04.529231+0000","last_became_peered":"2024-04-05T23:32:04.529231+0000","last_unstale":"2024-04-05T23:32:04.537327+0000","last_undegraded":"2024-04-05T23:32:04.537327+0000","last_fullsized":"2024-04-05T23:32:04.537327+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:49:15.608966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.421236+0000","last_change":"2024-04-05T23:32:04.002763+0000","last_active":"2024-04-05T23:32:09.421236+0000","last_peered":"2024-04-05T23:32:09.421236+0000","last_clean":"2024-04-05T23:32:09.421236+0000","last_became_active":"2024-04-05T23:32:04.002410+0000","last_became_peered":"2024-04-05T23:32:04.002410+0000","last_unstale":"2024-04-05T23:32:09.421236+0000","last_undegraded":"2024-04-05T23:32:09.421236+0000","last_fullsized":"2024-04-05T23:32:09.421236+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:15.193813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346346+0000","last_change":"2024-04-05T23:32:04.545381+0000","last_active":"2024-04-05T23:32:05.346346+0000","last_peered":"2024-04-05T23:32:05.346346+0000","last_clean":"2024-04-05T23:32:05.346346+0000","last_became_active":"2024-04-05T23:32:04.545128+0000","last_became_peered":"2024-04-05T23:32:04.545128+0000","last_unstale":"2024-04-05T23:32:05.346346+0000","last_undegraded":"2024-04-05T23:32:05.346346+0000","last_fullsized":"2024-04-05T23:32:05.346346+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:45:21.931160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.014686+0000","last_change":"2024-04-05T23:32:04.001937+0000","last_active":"2024-04-05T23:32:09.014686+0000","last_peered":"2024-04-05T23:32:09.014686+0000","last_clean":"2024-04-05T23:32:09.014686+0000","last_became_active":"2024-04-05T23:32:04.001508+0000","last_became_peered":"2024-04-05T23:32:04.001508+0000","last_unstale":"2024-04-05T23:32:09.014686+0000","last_undegraded":"2024-04-05T23:32:09.014686+0000","last_fullsized":"2024-04-05T23:32:09.014686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:09:20.682084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522193+0000","last_change":"2024-04-05T23:32:04.531576+0000","last_active":"2024-04-05T23:32:05.522193+0000","last_peered":"2024-04-05T23:32:05.522193+0000","last_clean":"2024-04-05T23:32:05.522193+0000","last_became_active":"2024-04-05T23:32:04.531234+0000","last_became_peered":"2024-04-05T23:32:04.531234+0000","last_unstale":"2024-04-05T23:32:05.522193+0000","last_undegraded":"2024-04-05T23:32:05.522193+0000","last_fullsized":"2024-04-05T23:32:05.522193+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:21:57.769861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'2","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.008501+0000","last_change":"2024-04-05T23:32:04.001570+0000","last_active":"2024-04-05T23:32:08.008501+0000","last_peered":"2024-04-05T23:32:08.008501+0000","last_clean":"2024-04-05T23:32:08.008501+0000","last_became_active":"2024-04-05T23:32:04.001224+0000","last_became_peered":"2024-04-05T23:32:04.001224+0000","last_unstale":"2024-04-05T23:32:08.008501+0000","last_undegraded":"2024-04-05T23:32:08.008501+0000","last_fullsized":"2024-04-05T23:32:08.008501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:44:41.070205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.546699+0000","last_change":"2024-04-05T23:32:04.542029+0000","last_active":"2024-04-05T23:32:04.546699+0000","last_peered":"2024-04-05T23:32:04.546699+0000","last_clean":"2024-04-05T23:32:04.546699+0000","last_became_active":"2024-04-05T23:32:04.541684+0000","last_became_peered":"2024-04-05T23:32:04.541684+0000","last_unstale":"2024-04-05T23:32:04.546699+0000","last_undegraded":"2024-04-05T23:32:04.546699+0000","last_fullsized":"2024-04-05T23:32:04.546699+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:07:06.456917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:11.414830+0000","last_change":"2024-04-05T23:32:04.004986+0000","last_active":"2024-04-05T23:32:11.414830+0000","last_peered":"2024-04-05T23:32:11.414830+0000","last_clean":"2024-04-05T23:32:11.414830+0000","last_became_active":"2024-04-05T23:32:04.004606+0000","last_became_peered":"2024-04-05T23:32:04.004606+0000","last_unstale":"2024-04-05T23:32:11.414830+0000","last_undegraded":"2024-04-05T23:32:11.414830+0000","last_fullsized":"2024-04-05T23:32:11.414830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:23:41.060500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390610+0000","last_change":"2024-04-05T23:32:04.541334+0000","last_active":"2024-04-05T23:32:05.390610+0000","last_peered":"2024-04-05T23:32:05.390610+0000","last_clean":"2024-04-05T23:32:05.390610+0000","last_became_active":"2024-04-05T23:32:04.540967+0000","last_became_peered":"2024-04-05T23:32:04.540967+0000","last_unstale":"2024-04-05T23:32:05.390610+0000","last_undegraded":"2024-04-05T23:32:05.390610+0000","last_fullsized":"2024-04-05T23:32:05.390610+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:16:33.045172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.425394+0000","last_change":"2024-04-05T23:32:04.003655+0000","last_active":"2024-04-05T23:32:10.425394+0000","last_peered":"2024-04-05T23:32:10.425394+0000","last_clean":"2024-04-05T23:32:10.425394+0000","last_became_active":"2024-04-05T23:32:04.003312+0000","last_became_peered":"2024-04-05T23:32:04.003312+0000","last_unstale":"2024-04-05T23:32:10.425394+0000","last_undegraded":"2024-04-05T23:32:10.425394+0000","last_fullsized":"2024-04-05T23:32:10.425394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:11:49.770921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561707+0000","last_change":"2024-04-05T23:32:04.533477+0000","last_active":"2024-04-05T23:32:05.561707+0000","last_peered":"2024-04-05T23:32:05.561707+0000","last_clean":"2024-04-05T23:32:05.561707+0000","last_became_active":"2024-04-05T23:32:04.533109+0000","last_became_peered":"2024-04-05T23:32:04.533109+0000","last_unstale":"2024-04-05T23:32:05.561707+0000","last_undegraded":"2024-04-05T23:32:05.561707+0000","last_fullsized":"2024-04-05T23:32:05.561707+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:35:08.383911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.411791+0000","last_change":"2024-04-05T23:32:04.000697+0000","last_active":"2024-04-05T23:32:09.411791+0000","last_peered":"2024-04-05T23:32:09.411791+0000","last_clean":"2024-04-05T23:32:09.411791+0000","last_became_active":"2024-04-05T23:32:04.000377+0000","last_became_peered":"2024-04-05T23:32:04.000377+0000","last_unstale":"2024-04-05T23:32:09.411791+0000","last_undegraded":"2024-04-05T23:32:09.411791+0000","last_fullsized":"2024-04-05T23:32:09.411791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:55:11.227637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390587+0000","last_change":"2024-04-05T23:32:04.528939+0000","last_active":"2024-04-05T23:32:05.390587+0000","last_peered":"2024-04-05T23:32:05.390587+0000","last_clean":"2024-04-05T23:32:05.390587+0000","last_became_active":"2024-04-05T23:32:04.528610+0000","last_became_peered":"2024-04-05T23:32:04.528610+0000","last_unstale":"2024-04-05T23:32:05.390587+0000","last_undegraded":"2024-04-05T23:32:05.390587+0000","last_fullsized":"2024-04-05T23:32:05.390587+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:08:35.685691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'17","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:11.450020+0000","last_change":"2024-04-05T23:32:04.007322+0000","last_active":"2024-04-05T23:32:11.450020+0000","last_peered":"2024-04-05T23:32:11.450020+0000","last_clean":"2024-04-05T23:32:11.450020+0000","last_became_active":"2024-04-05T23:32:04.006971+0000","last_became_peered":"2024-04-05T23:32:04.006971+0000","last_unstale":"2024-04-05T23:32:11.450020+0000","last_undegraded":"2024-04-05T23:32:11.450020+0000","last_fullsized":"2024-04-05T23:32:11.450020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:04:38.413482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20849,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":12,"num_write":19,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.540756+0000","last_change":"2024-04-05T23:32:04.535342+0000","last_active":"2024-04-05T23:32:04.540756+0000","last_peered":"2024-04-05T23:32:04.540756+0000","last_clean":"2024-04-05T23:32:04.540756+0000","last_became_active":"2024-04-05T23:32:04.535053+0000","last_became_peered":"2024-04-05T23:32:04.535053+0000","last_unstale":"2024-04-05T23:32:04.540756+0000","last_undegraded":"2024-04-05T23:32:04.540756+0000","last_fullsized":"2024-04-05T23:32:04.540756+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:19:20.683836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.426401+0000","last_change":"2024-04-05T23:32:04.001911+0000","last_active":"2024-04-05T23:32:10.426401+0000","last_peered":"2024-04-05T23:32:10.426401+0000","last_clean":"2024-04-05T23:32:10.426401+0000","last_became_active":"2024-04-05T23:32:04.001530+0000","last_became_peered":"2024-04-05T23:32:04.001530+0000","last_unstale":"2024-04-05T23:32:10.426401+0000","last_undegraded":"2024-04-05T23:32:10.426401+0000","last_fullsized":"2024-04-05T23:32:10.426401+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:00:05.891629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561195+0000","last_change":"2024-04-05T23:32:04.536455+0000","last_active":"2024-04-05T23:32:05.561195+0000","last_peered":"2024-04-05T23:32:05.561195+0000","last_clean":"2024-04-05T23:32:05.561195+0000","last_became_active":"2024-04-05T23:32:04.533994+0000","last_became_peered":"2024-04-05T23:32:04.533994+0000","last_unstale":"2024-04-05T23:32:05.561195+0000","last_undegraded":"2024-04-05T23:32:05.561195+0000","last_fullsized":"2024-04-05T23:32:05.561195+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:38:18.952823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.007724+0000","last_change":"2024-04-05T23:32:04.000708+0000","last_active":"2024-04-05T23:32:08.007724+0000","last_peered":"2024-04-05T23:32:08.007724+0000","last_clean":"2024-04-05T23:32:08.007724+0000","last_became_active":"2024-04-05T23:32:04.000354+0000","last_became_peered":"2024-04-05T23:32:04.000354+0000","last_unstale":"2024-04-05T23:32:08.007724+0000","last_undegraded":"2024-04-05T23:32:08.007724+0000","last_fullsized":"2024-04-05T23:32:08.007724+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:05:34.291607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562033+0000","last_change":"2024-04-05T23:32:04.532146+0000","last_active":"2024-04-05T23:32:05.562033+0000","last_peered":"2024-04-05T23:32:05.562033+0000","last_clean":"2024-04-05T23:32:05.562033+0000","last_became_active":"2024-04-05T23:32:04.531735+0000","last_became_peered":"2024-04-05T23:32:04.531735+0000","last_unstale":"2024-04-05T23:32:05.562033+0000","last_undegraded":"2024-04-05T23:32:05.562033+0000","last_fullsized":"2024-04-05T23:32:05.562033+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:52:44.898871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:11.416606+0000","last_change":"2024-04-05T23:32:04.000144+0000","last_active":"2024-04-05T23:32:11.416606+0000","last_peered":"2024-04-05T23:32:11.416606+0000","last_clean":"2024-04-05T23:32:11.416606+0000","last_became_active":"2024-04-05T23:32:03.999712+0000","last_became_peered":"2024-04-05T23:32:03.999712+0000","last_unstale":"2024-04-05T23:32:11.416606+0000","last_undegraded":"2024-04-05T23:32:11.416606+0000","last_fullsized":"2024-04-05T23:32:11.416606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:43.775018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.560698+0000","last_change":"2024-04-05T23:32:04.537623+0000","last_active":"2024-04-05T23:32:05.560698+0000","last_peered":"2024-04-05T23:32:05.560698+0000","last_clean":"2024-04-05T23:32:05.560698+0000","last_became_active":"2024-04-05T23:32:04.537399+0000","last_became_peered":"2024-04-05T23:32:04.537399+0000","last_unstale":"2024-04-05T23:32:05.560698+0000","last_undegraded":"2024-04-05T23:32:05.560698+0000","last_fullsized":"2024-04-05T23:32:05.560698+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:01:01.643133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.014307+0000","last_change":"2024-04-05T23:32:04.006806+0000","last_active":"2024-04-05T23:32:09.014307+0000","last_peered":"2024-04-05T23:32:09.014307+0000","last_clean":"2024-04-05T23:32:09.014307+0000","last_became_active":"2024-04-05T23:32:04.006397+0000","last_became_peered":"2024-04-05T23:32:04.006397+0000","last_unstale":"2024-04-05T23:32:09.014307+0000","last_undegraded":"2024-04-05T23:32:09.014307+0000","last_fullsized":"2024-04-05T23:32:09.014307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:42:01.758863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346287+0000","last_change":"2024-04-05T23:32:04.545952+0000","last_active":"2024-04-05T23:32:05.346287+0000","last_peered":"2024-04-05T23:32:05.346287+0000","last_clean":"2024-04-05T23:32:05.346287+0000","last_became_active":"2024-04-05T23:32:04.545528+0000","last_became_peered":"2024-04-05T23:32:04.545528+0000","last_unstale":"2024-04-05T23:32:05.346287+0000","last_undegraded":"2024-04-05T23:32:05.346287+0000","last_fullsized":"2024-04-05T23:32:05.346287+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:33:06.074173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.544361+0000","last_change":"2024-04-05T23:32:03.999841+0000","last_active":"2024-04-05T23:32:04.544361+0000","last_peered":"2024-04-05T23:32:04.544361+0000","last_clean":"2024-04-05T23:32:04.544361+0000","last_became_active":"2024-04-05T23:32:03.999418+0000","last_became_peered":"2024-04-05T23:32:03.999418+0000","last_unstale":"2024-04-05T23:32:04.544361+0000","last_undegraded":"2024-04-05T23:32:04.544361+0000","last_fullsized":"2024-04-05T23:32:04.544361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:00:41.452167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561536+0000","last_change":"2024-04-05T23:32:04.530012+0000","last_active":"2024-04-05T23:32:05.561536+0000","last_peered":"2024-04-05T23:32:05.561536+0000","last_clean":"2024-04-05T23:32:05.561536+0000","last_became_active":"2024-04-05T23:32:04.529628+0000","last_became_peered":"2024-04-05T23:32:04.529628+0000","last_unstale":"2024-04-05T23:32:05.561536+0000","last_undegraded":"2024-04-05T23:32:05.561536+0000","last_fullsized":"2024-04-05T23:32:05.561536+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:40:22.036170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.527314+0000","last_change":"2024-04-05T23:32:04.006157+0000","last_active":"2024-04-05T23:32:04.527314+0000","last_peered":"2024-04-05T23:32:04.527314+0000","last_clean":"2024-04-05T23:32:04.527314+0000","last_became_active":"2024-04-05T23:32:04.005853+0000","last_became_peered":"2024-04-05T23:32:04.005853+0000","last_unstale":"2024-04-05T23:32:04.527314+0000","last_undegraded":"2024-04-05T23:32:04.527314+0000","last_fullsized":"2024-04-05T23:32:04.527314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:48:39.849306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:07.154300+0000","last_change":"2024-04-05T23:32:04.547367+0000","last_active":"2024-04-05T23:32:07.154300+0000","last_peered":"2024-04-05T23:32:07.154300+0000","last_clean":"2024-04-05T23:32:07.154300+0000","last_became_active":"2024-04-05T23:32:04.547040+0000","last_became_peered":"2024-04-05T23:32:04.547040+0000","last_unstale":"2024-04-05T23:32:07.154300+0000","last_undegraded":"2024-04-05T23:32:07.154300+0000","last_fullsized":"2024-04-05T23:32:07.154300+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:41:41.998984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.426531+0000","last_change":"2024-04-05T23:32:04.009851+0000","last_active":"2024-04-05T23:32:10.426531+0000","last_peered":"2024-04-05T23:32:10.426531+0000","last_clean":"2024-04-05T23:32:10.426531+0000","last_became_active":"2024-04-05T23:32:04.009460+0000","last_became_peered":"2024-04-05T23:32:04.009460+0000","last_unstale":"2024-04-05T23:32:10.426531+0000","last_undegraded":"2024-04-05T23:32:10.426531+0000","last_fullsized":"2024-04-05T23:32:10.426531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:39:41.000439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561646+0000","last_change":"2024-04-05T23:32:04.549621+0000","last_active":"2024-04-05T23:32:05.561646+0000","last_peered":"2024-04-05T23:32:05.561646+0000","last_clean":"2024-04-05T23:32:05.561646+0000","last_became_active":"2024-04-05T23:32:04.549325+0000","last_became_peered":"2024-04-05T23:32:04.549325+0000","last_unstale":"2024-04-05T23:32:05.561646+0000","last_undegraded":"2024-04-05T23:32:05.561646+0000","last_fullsized":"2024-04-05T23:32:05.561646+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:11:07.101463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346275+0000","last_change":"2024-04-05T23:32:04.546768+0000","last_active":"2024-04-05T23:32:05.346275+0000","last_peered":"2024-04-05T23:32:05.346275+0000","last_clean":"2024-04-05T23:32:05.346275+0000","last_became_active":"2024-04-05T23:32:04.546471+0000","last_became_peered":"2024-04-05T23:32:04.546471+0000","last_unstale":"2024-04-05T23:32:05.346275+0000","last_undegraded":"2024-04-05T23:32:05.346275+0000","last_fullsized":"2024-04-05T23:32:05.346275+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:48:22.728976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.414884+0000","last_change":"2024-04-05T23:32:04.007093+0000","last_active":"2024-04-05T23:32:10.414884+0000","last_peered":"2024-04-05T23:32:10.414884+0000","last_clean":"2024-04-05T23:32:10.414884+0000","last_became_active":"2024-04-05T23:32:04.006836+0000","last_became_peered":"2024-04-05T23:32:04.006836+0000","last_unstale":"2024-04-05T23:32:10.414884+0000","last_undegraded":"2024-04-05T23:32:10.414884+0000","last_fullsized":"2024-04-05T23:32:10.414884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:20:42.489368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522825+0000","last_change":"2024-04-05T23:32:04.538204+0000","last_active":"2024-04-05T23:32:05.522825+0000","last_peered":"2024-04-05T23:32:05.522825+0000","last_clean":"2024-04-05T23:32:05.522825+0000","last_became_active":"2024-04-05T23:32:04.535254+0000","last_became_peered":"2024-04-05T23:32:04.535254+0000","last_unstale":"2024-04-05T23:32:05.522825+0000","last_undegraded":"2024-04-05T23:32:05.522825+0000","last_fullsized":"2024-04-05T23:32:05.522825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:06:01.059339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.425969+0000","last_change":"2024-04-05T23:32:03.997704+0000","last_active":"2024-04-05T23:32:10.425969+0000","last_peered":"2024-04-05T23:32:10.425969+0000","last_clean":"2024-04-05T23:32:10.425969+0000","last_became_active":"2024-04-05T23:32:03.997275+0000","last_became_peered":"2024-04-05T23:32:03.997275+0000","last_unstale":"2024-04-05T23:32:10.425969+0000","last_undegraded":"2024-04-05T23:32:10.425969+0000","last_fullsized":"2024-04-05T23:32:10.425969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:58.355104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":333,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.528158+0000","last_change":"2024-04-05T23:31:10.395041+0000","last_active":"2024-04-05T23:32:04.528158+0000","last_peered":"2024-04-05T23:32:04.528158+0000","last_clean":"2024-04-05T23:32:04.528158+0000","last_became_active":"2024-04-05T23:31:10.394479+0000","last_became_peered":"2024-04-05T23:31:10.394479+0000","last_unstale":"2024-04-05T23:32:04.528158+0000","last_undegraded":"2024-04-05T23:32:04.528158+0000","last_fullsized":"2024-04-05T23:32:04.528158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_clean_scrub_stamp":"2024-04-05T23:28:33.945697+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:54:10.683526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.560765+0000","last_change":"2024-04-05T23:32:04.537756+0000","last_active":"2024-04-05T23:32:05.560765+0000","last_peered":"2024-04-05T23:32:05.560765+0000","last_clean":"2024-04-05T23:32:05.560765+0000","last_became_active":"2024-04-05T23:32:04.536242+0000","last_became_peered":"2024-04-05T23:32:04.536242+0000","last_unstale":"2024-04-05T23:32:05.560765+0000","last_undegraded":"2024-04-05T23:32:05.560765+0000","last_fullsized":"2024-04-05T23:32:05.560765+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:41:07.817969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.012297+0000","last_change":"2024-04-05T23:32:04.005978+0000","last_active":"2024-04-05T23:32:09.012297+0000","last_peered":"2024-04-05T23:32:09.012297+0000","last_clean":"2024-04-05T23:32:09.012297+0000","last_became_active":"2024-04-05T23:32:04.005638+0000","last_became_peered":"2024-04-05T23:32:04.005638+0000","last_unstale":"2024-04-05T23:32:09.012297+0000","last_undegraded":"2024-04-05T23:32:09.012297+0000","last_fullsized":"2024-04-05T23:32:09.012297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:39:35.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.725108+0000","last_change":"2024-04-05T23:32:04.006727+0000","last_active":"2024-04-05T23:32:05.725108+0000","last_peered":"2024-04-05T23:32:05.725108+0000","last_clean":"2024-04-05T23:32:05.725108+0000","last_became_active":"2024-04-05T23:32:04.006414+0000","last_became_peered":"2024-04-05T23:32:04.006414+0000","last_unstale":"2024-04-05T23:32:05.725108+0000","last_undegraded":"2024-04-05T23:32:05.725108+0000","last_fullsized":"2024-04-05T23:32:05.725108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:41:28.661118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561450+0000","last_change":"2024-04-05T23:32:04.536893+0000","last_active":"2024-04-05T23:32:05.561450+0000","last_peered":"2024-04-05T23:32:05.561450+0000","last_clean":"2024-04-05T23:32:05.561450+0000","last_became_active":"2024-04-05T23:32:04.536629+0000","last_became_peered":"2024-04-05T23:32:04.536629+0000","last_unstale":"2024-04-05T23:32:05.561450+0000","last_undegraded":"2024-04-05T23:32:05.561450+0000","last_fullsized":"2024-04-05T23:32:05.561450+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:27:58.279768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'3","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.014072+0000","last_change":"2024-04-05T23:32:04.010202+0000","last_active":"2024-04-05T23:32:09.014072+0000","last_peered":"2024-04-05T23:32:09.014072+0000","last_clean":"2024-04-05T23:32:09.014072+0000","last_became_active":"2024-04-05T23:32:04.009889+0000","last_became_peered":"2024-04-05T23:32:04.009889+0000","last_unstale":"2024-04-05T23:32:09.014072+0000","last_undegraded":"2024-04-05T23:32:09.014072+0000","last_fullsized":"2024-04-05T23:32:09.014072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:12:13.121449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522648+0000","last_change":"2024-04-05T23:32:04.533576+0000","last_active":"2024-04-05T23:32:05.522648+0000","last_peered":"2024-04-05T23:32:05.522648+0000","last_clean":"2024-04-05T23:32:05.522648+0000","last_became_active":"2024-04-05T23:32:04.533066+0000","last_became_peered":"2024-04-05T23:32:04.533066+0000","last_unstale":"2024-04-05T23:32:05.522648+0000","last_undegraded":"2024-04-05T23:32:05.522648+0000","last_fullsized":"2024-04-05T23:32:05.522648+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:01:44.019210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.425968+0000","last_change":"2024-04-05T23:32:04.005096+0000","last_active":"2024-04-05T23:32:10.425968+0000","last_peered":"2024-04-05T23:32:10.425968+0000","last_clean":"2024-04-05T23:32:10.425968+0000","last_became_active":"2024-04-05T23:32:04.004702+0000","last_became_peered":"2024-04-05T23:32:04.004702+0000","last_unstale":"2024-04-05T23:32:10.425968+0000","last_undegraded":"2024-04-05T23:32:10.425968+0000","last_fullsized":"2024-04-05T23:32:10.425968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:33:09.934039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561473+0000","last_change":"2024-04-05T23:32:04.535623+0000","last_active":"2024-04-05T23:32:05.561473+0000","last_peered":"2024-04-05T23:32:05.561473+0000","last_clean":"2024-04-05T23:32:05.561473+0000","last_became_active":"2024-04-05T23:32:04.535223+0000","last_became_peered":"2024-04-05T23:32:04.535223+0000","last_unstale":"2024-04-05T23:32:05.561473+0000","last_undegraded":"2024-04-05T23:32:05.561473+0000","last_fullsized":"2024-04-05T23:32:05.561473+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:24:43.543119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.008998+0000","last_change":"2024-04-05T23:32:04.004097+0000","last_active":"2024-04-05T23:32:08.008998+0000","last_peered":"2024-04-05T23:32:08.008998+0000","last_clean":"2024-04-05T23:32:08.008998+0000","last_became_active":"2024-04-05T23:32:04.003713+0000","last_became_peered":"2024-04-05T23:32:04.003713+0000","last_unstale":"2024-04-05T23:32:08.008998+0000","last_undegraded":"2024-04-05T23:32:08.008998+0000","last_fullsized":"2024-04-05T23:32:08.008998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:47:43.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.540597+0000","last_change":"2024-04-05T23:32:04.538843+0000","last_active":"2024-04-05T23:32:04.540597+0000","last_peered":"2024-04-05T23:32:04.540597+0000","last_clean":"2024-04-05T23:32:04.540597+0000","last_became_active":"2024-04-05T23:32:04.538472+0000","last_became_peered":"2024-04-05T23:32:04.538472+0000","last_unstale":"2024-04-05T23:32:04.540597+0000","last_undegraded":"2024-04-05T23:32:04.540597+0000","last_fullsized":"2024-04-05T23:32:04.540597+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:40:40.863815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.006820+0000","last_change":"2024-04-05T23:32:04.013064+0000","last_active":"2024-04-05T23:32:08.006820+0000","last_peered":"2024-04-05T23:32:08.006820+0000","last_clean":"2024-04-05T23:32:08.006820+0000","last_became_active":"2024-04-05T23:32:04.012751+0000","last_became_peered":"2024-04-05T23:32:04.012751+0000","last_unstale":"2024-04-05T23:32:08.006820+0000","last_undegraded":"2024-04-05T23:32:08.006820+0000","last_fullsized":"2024-04-05T23:32:08.006820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:06:10.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561313+0000","last_change":"2024-04-05T23:32:04.529014+0000","last_active":"2024-04-05T23:32:05.561313+0000","last_peered":"2024-04-05T23:32:05.561313+0000","last_clean":"2024-04-05T23:32:05.561313+0000","last_became_active":"2024-04-05T23:32:04.528622+0000","last_became_peered":"2024-04-05T23:32:04.528622+0000","last_unstale":"2024-04-05T23:32:05.561313+0000","last_undegraded":"2024-04-05T23:32:05.561313+0000","last_fullsized":"2024-04-05T23:32:05.561313+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:46:26.390934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.877474+0000","last_change":"2024-04-05T23:32:04.002651+0000","last_active":"2024-04-05T23:32:10.877474+0000","last_peered":"2024-04-05T23:32:10.877474+0000","last_clean":"2024-04-05T23:32:10.877474+0000","last_became_active":"2024-04-05T23:32:04.002346+0000","last_became_peered":"2024-04-05T23:32:04.002346+0000","last_unstale":"2024-04-05T23:32:10.877474+0000","last_undegraded":"2024-04-05T23:32:10.877474+0000","last_fullsized":"2024-04-05T23:32:10.877474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:26:48.176254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522393+0000","last_change":"2024-04-05T23:32:04.538785+0000","last_active":"2024-04-05T23:32:05.522393+0000","last_peered":"2024-04-05T23:32:05.522393+0000","last_clean":"2024-04-05T23:32:05.522393+0000","last_became_active":"2024-04-05T23:32:04.538463+0000","last_became_peered":"2024-04-05T23:32:04.538463+0000","last_unstale":"2024-04-05T23:32:05.522393+0000","last_undegraded":"2024-04-05T23:32:05.522393+0000","last_fullsized":"2024-04-05T23:32:05.522393+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:44:38.941857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.006605+0000","last_change":"2024-04-05T23:32:04.003760+0000","last_active":"2024-04-05T23:32:08.006605+0000","last_peered":"2024-04-05T23:32:08.006605+0000","last_clean":"2024-04-05T23:32:08.006605+0000","last_became_active":"2024-04-05T23:32:04.003389+0000","last_became_peered":"2024-04-05T23:32:04.003389+0000","last_unstale":"2024-04-05T23:32:08.006605+0000","last_undegraded":"2024-04-05T23:32:08.006605+0000","last_fullsized":"2024-04-05T23:32:08.006605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:42:00.353651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561098+0000","last_change":"2024-04-05T23:32:04.540051+0000","last_active":"2024-04-05T23:32:05.561098+0000","last_peered":"2024-04-05T23:32:05.561098+0000","last_clean":"2024-04-05T23:32:05.561098+0000","last_became_active":"2024-04-05T23:32:04.539736+0000","last_became_peered":"2024-04-05T23:32:04.539736+0000","last_unstale":"2024-04-05T23:32:05.561098+0000","last_undegraded":"2024-04-05T23:32:05.561098+0000","last_fullsized":"2024-04-05T23:32:05.561098+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:54:14.086904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:11.450069+0000","last_change":"2024-04-05T23:32:04.006492+0000","last_active":"2024-04-05T23:32:11.450069+0000","last_peered":"2024-04-05T23:32:11.450069+0000","last_clean":"2024-04-05T23:32:11.450069+0000","last_became_active":"2024-04-05T23:32:04.006154+0000","last_became_peered":"2024-04-05T23:32:04.006154+0000","last_unstale":"2024-04-05T23:32:11.450069+0000","last_undegraded":"2024-04-05T23:32:11.450069+0000","last_fullsized":"2024-04-05T23:32:11.450069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:53:23.431400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562475+0000","last_change":"2024-04-05T23:32:04.540757+0000","last_active":"2024-04-05T23:32:05.562475+0000","last_peered":"2024-04-05T23:32:05.562475+0000","last_clean":"2024-04-05T23:32:05.562475+0000","last_became_active":"2024-04-05T23:32:04.540359+0000","last_became_peered":"2024-04-05T23:32:04.540359+0000","last_unstale":"2024-04-05T23:32:05.562475+0000","last_undegraded":"2024-04-05T23:32:05.562475+0000","last_fullsized":"2024-04-05T23:32:05.562475+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:43:52.533489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'2","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.409996+0000","last_change":"2024-04-05T23:32:04.002736+0000","last_active":"2024-04-05T23:32:09.409996+0000","last_peered":"2024-04-05T23:32:09.409996+0000","last_clean":"2024-04-05T23:32:09.409996+0000","last_became_active":"2024-04-05T23:32:04.002277+0000","last_became_peered":"2024-04-05T23:32:04.002277+0000","last_unstale":"2024-04-05T23:32:09.409996+0000","last_undegraded":"2024-04-05T23:32:09.409996+0000","last_fullsized":"2024-04-05T23:32:09.409996+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:56:39.001732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561684+0000","last_change":"2024-04-05T23:32:04.537076+0000","last_active":"2024-04-05T23:32:05.561684+0000","last_peered":"2024-04-05T23:32:05.561684+0000","last_clean":"2024-04-05T23:32:05.561684+0000","last_became_active":"2024-04-05T23:32:04.536707+0000","last_became_peered":"2024-04-05T23:32:04.536707+0000","last_unstale":"2024-04-05T23:32:05.561684+0000","last_undegraded":"2024-04-05T23:32:05.561684+0000","last_fullsized":"2024-04-05T23:32:05.561684+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:51:09.395408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.425335+0000","last_change":"2024-04-05T23:32:04.006122+0000","last_active":"2024-04-05T23:32:10.425335+0000","last_peered":"2024-04-05T23:32:10.425335+0000","last_clean":"2024-04-05T23:32:10.425335+0000","last_became_active":"2024-04-05T23:32:04.005683+0000","last_became_peered":"2024-04-05T23:32:04.005683+0000","last_unstale":"2024-04-05T23:32:10.425335+0000","last_undegraded":"2024-04-05T23:32:10.425335+0000","last_fullsized":"2024-04-05T23:32:10.425335+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:21:40.922614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561891+0000","last_change":"2024-04-05T23:32:04.533321+0000","last_active":"2024-04-05T23:32:05.561891+0000","last_peered":"2024-04-05T23:32:05.561891+0000","last_clean":"2024-04-05T23:32:05.561891+0000","last_became_active":"2024-04-05T23:32:04.532949+0000","last_became_peered":"2024-04-05T23:32:04.532949+0000","last_unstale":"2024-04-05T23:32:05.561891+0000","last_undegraded":"2024-04-05T23:32:05.561891+0000","last_fullsized":"2024-04-05T23:32:05.561891+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:31.500132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561404+0000","last_change":"2024-04-05T23:32:04.536508+0000","last_active":"2024-04-05T23:32:05.561404+0000","last_peered":"2024-04-05T23:32:05.561404+0000","last_clean":"2024-04-05T23:32:05.561404+0000","last_became_active":"2024-04-05T23:32:04.536142+0000","last_became_peered":"2024-04-05T23:32:04.536142+0000","last_unstale":"2024-04-05T23:32:05.561404+0000","last_undegraded":"2024-04-05T23:32:05.561404+0000","last_fullsized":"2024-04-05T23:32:05.561404+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:18:28.497807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561185+0000","last_change":"2024-04-05T23:32:04.545511+0000","last_active":"2024-04-05T23:32:05.561185+0000","last_peered":"2024-04-05T23:32:05.561185+0000","last_clean":"2024-04-05T23:32:05.561185+0000","last_became_active":"2024-04-05T23:32:04.545191+0000","last_became_peered":"2024-04-05T23:32:04.545191+0000","last_unstale":"2024-04-05T23:32:05.561185+0000","last_undegraded":"2024-04-05T23:32:05.561185+0000","last_fullsized":"2024-04-05T23:32:05.561185+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:17:06.431429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.560809+0000","last_change":"2024-04-05T23:32:04.549673+0000","last_active":"2024-04-05T23:32:05.560809+0000","last_peered":"2024-04-05T23:32:05.560809+0000","last_clean":"2024-04-05T23:32:05.560809+0000","last_became_active":"2024-04-05T23:32:04.549391+0000","last_became_peered":"2024-04-05T23:32:04.549391+0000","last_unstale":"2024-04-05T23:32:05.560809+0000","last_undegraded":"2024-04-05T23:32:05.560809+0000","last_fullsized":"2024-04-05T23:32:05.560809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:30:22.689107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561256+0000","last_change":"2024-04-05T23:32:04.535735+0000","last_active":"2024-04-05T23:32:05.561256+0000","last_peered":"2024-04-05T23:32:05.561256+0000","last_clean":"2024-04-05T23:32:05.561256+0000","last_became_active":"2024-04-05T23:32:04.534461+0000","last_became_peered":"2024-04-05T23:32:04.534461+0000","last_unstale":"2024-04-05T23:32:05.561256+0000","last_undegraded":"2024-04-05T23:32:05.561256+0000","last_fullsized":"2024-04-05T23:32:05.561256+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:37:41.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561517+0000","last_change":"2024-04-05T23:32:04.532340+0000","last_active":"2024-04-05T23:32:05.561517+0000","last_peered":"2024-04-05T23:32:05.561517+0000","last_clean":"2024-04-05T23:32:05.561517+0000","last_became_active":"2024-04-05T23:32:04.530053+0000","last_became_peered":"2024-04-05T23:32:04.530053+0000","last_unstale":"2024-04-05T23:32:05.561517+0000","last_undegraded":"2024-04-05T23:32:05.561517+0000","last_fullsized":"2024-04-05T23:32:05.561517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:16:11.168366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561662+0000","last_change":"2024-04-05T23:32:04.529972+0000","last_active":"2024-04-05T23:32:05.561662+0000","last_peered":"2024-04-05T23:32:05.561662+0000","last_clean":"2024-04-05T23:32:05.561662+0000","last_became_active":"2024-04-05T23:32:04.529551+0000","last_became_peered":"2024-04-05T23:32:04.529551+0000","last_unstale":"2024-04-05T23:32:05.561662+0000","last_undegraded":"2024-04-05T23:32:05.561662+0000","last_fullsized":"2024-04-05T23:32:05.561662+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:13.764971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561382+0000","last_change":"2024-04-05T23:32:04.529166+0000","last_active":"2024-04-05T23:32:05.561382+0000","last_peered":"2024-04-05T23:32:05.561382+0000","last_clean":"2024-04-05T23:32:05.561382+0000","last_became_active":"2024-04-05T23:32:04.528785+0000","last_became_peered":"2024-04-05T23:32:04.528785+0000","last_unstale":"2024-04-05T23:32:05.561382+0000","last_undegraded":"2024-04-05T23:32:05.561382+0000","last_fullsized":"2024-04-05T23:32:05.561382+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:29:43.964265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522551+0000","last_change":"2024-04-05T23:32:04.540631+0000","last_active":"2024-04-05T23:32:05.522551+0000","last_peered":"2024-04-05T23:32:05.522551+0000","last_clean":"2024-04-05T23:32:05.522551+0000","last_became_active":"2024-04-05T23:32:04.540327+0000","last_became_peered":"2024-04-05T23:32:04.540327+0000","last_unstale":"2024-04-05T23:32:05.522551+0000","last_undegraded":"2024-04-05T23:32:05.522551+0000","last_fullsized":"2024-04-05T23:32:05.522551+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:22:48.825324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346263+0000","last_change":"2024-04-05T23:32:04.544845+0000","last_active":"2024-04-05T23:32:05.346263+0000","last_peered":"2024-04-05T23:32:05.346263+0000","last_clean":"2024-04-05T23:32:05.346263+0000","last_became_active":"2024-04-05T23:32:04.544472+0000","last_became_peered":"2024-04-05T23:32:04.544472+0000","last_unstale":"2024-04-05T23:32:05.346263+0000","last_undegraded":"2024-04-05T23:32:05.346263+0000","last_fullsized":"2024-04-05T23:32:05.346263+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:34:09.529644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522160+0000","last_change":"2024-04-05T23:32:04.545303+0000","last_active":"2024-04-05T23:32:05.522160+0000","last_peered":"2024-04-05T23:32:05.522160+0000","last_clean":"2024-04-05T23:32:05.522160+0000","last_became_active":"2024-04-05T23:32:04.540192+0000","last_became_peered":"2024-04-05T23:32:04.540192+0000","last_unstale":"2024-04-05T23:32:05.522160+0000","last_undegraded":"2024-04-05T23:32:05.522160+0000","last_fullsized":"2024-04-05T23:32:05.522160+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:49:18.505440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390540+0000","last_change":"2024-04-05T23:32:04.534631+0000","last_active":"2024-04-05T23:32:05.390540+0000","last_peered":"2024-04-05T23:32:05.390540+0000","last_clean":"2024-04-05T23:32:05.390540+0000","last_became_active":"2024-04-05T23:32:04.534330+0000","last_became_peered":"2024-04-05T23:32:04.534330+0000","last_unstale":"2024-04-05T23:32:05.390540+0000","last_undegraded":"2024-04-05T23:32:05.390540+0000","last_fullsized":"2024-04-05T23:32:05.390540+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:20:47.592174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561028+0000","last_change":"2024-04-05T23:32:04.531459+0000","last_active":"2024-04-05T23:32:05.561028+0000","last_peered":"2024-04-05T23:32:05.561028+0000","last_clean":"2024-04-05T23:32:05.561028+0000","last_became_active":"2024-04-05T23:32:04.530828+0000","last_became_peered":"2024-04-05T23:32:04.530828+0000","last_unstale":"2024-04-05T23:32:05.561028+0000","last_undegraded":"2024-04-05T23:32:05.561028+0000","last_fullsized":"2024-04-05T23:32:05.561028+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:51:34.090030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561489+0000","last_change":"2024-04-05T23:32:04.527711+0000","last_active":"2024-04-05T23:32:05.561489+0000","last_peered":"2024-04-05T23:32:05.561489+0000","last_clean":"2024-04-05T23:32:05.561489+0000","last_became_active":"2024-04-05T23:32:04.527374+0000","last_became_peered":"2024-04-05T23:32:04.527374+0000","last_unstale":"2024-04-05T23:32:05.561489+0000","last_undegraded":"2024-04-05T23:32:05.561489+0000","last_fullsized":"2024-04-05T23:32:05.561489+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:15:10.764340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":26317,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":38,"num_read_kb":34,"num_write":175,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":81428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":412499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]}]},{"osd":10,"up_from":67,"seq":287762808847,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1507328,"data_stored":1012283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52200000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.70499999999999996}]},{"osd":3,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.67400000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.73299999999999998}]},{"osd":5,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.749}]},{"osd":6,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.77100000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.79300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]}]},{"osd":9,"up_from":62,"seq":266287972370,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":412845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.89600000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.72499999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.58699999999999997}]},{"osd":4,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.052}]},{"osd":5,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.82999999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.91900000000000004}]},{"osd":7,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.874},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.004}]},{"osd":8,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.88700000000000001}]},{"osd":10,"last update":"Fri Apr 5 23:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.90500000000000003}]}]},{"osd":8,"up_from":55,"seq":236223201303,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":412687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.59299999999999997}]},{"osd":1,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.495},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.434},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.46200000000000002}]},{"osd":6,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.67100000000000004}]},{"osd":7,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.622},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.69199999999999995}]},{"osd":9,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.68600000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.751},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":7,"up_from":48,"seq":206158430234,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1478656,"data_stored":1005427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.76400000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.35999999999999999}]},{"osd":2,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.56200000000000006}]},{"osd":4,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.34000000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.378},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51700000000000002}]},{"osd":6,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.68100000000000005}]},{"osd":8,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.443},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.69499999999999995}]},{"osd":9,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Apr 5 23:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.622},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":6,"up_from":42,"seq":180388626462,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":412462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47799999999999998,"15min":0.47499999999999998},"min":{"1min":0.34000000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73499999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.433},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.505,"15min":0.5},"min":{"1min":0.36399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.79600000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.70199999999999996}]},{"osd":1,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.45900000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90400000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.627,"15min":0.627},"min":{"1min":0.39500000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92400000000000004,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.41799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85599999999999998,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.89600000000000002}]},{"osd":3,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78500000000000003,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.754},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67400000000000004,"15min":0.66900000000000004},"min":{"1min":0.47299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.92800000000000005,"5min":1.246,"15min":1.246},"last":0.65200000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.50600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.90200000000000002,"5min":1.095,"15min":1.095},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.48599999999999999}]},{"osd":5,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.49099999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.016,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.73899999999999999},"min":{"1min":0.54900000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0509999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.74199999999999999}]},{"osd":7,"last update":"Fri Apr 5 23:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.55400000000000005}]},{"osd":8,"last update":"Fri Apr 5 23:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.79100000000000004}]},{"osd":9,"last update":"Fri Apr 5 23:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.85099999999999998}]},{"osd":10,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.81999999999999995}]}]},{"osd":1,"up_from":12,"seq":51539607601,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":860160,"data_stored":413352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.498},"min":{"1min":0.35099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.65400000000000003,"15min":0.68899999999999995},"min":{"1min":0.32700000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.748,"5min":4.4199999999999999,"15min":4.4199999999999999},"last":0.45900000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52100000000000002,"15min":0.50900000000000001},"min":{"1min":0.32600000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.751},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.52600000000000002,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.85199999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.56299999999999994,"15min":0.53400000000000003},"min":{"1min":0.44700000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.627},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44400000000000001,"15min":0.442},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65700000000000003,"15min":0.65300000000000002},"min":{"1min":0.44900000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.89100000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.68700000000000006,"15min":0.67900000000000005},"min":{"1min":0.5,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65100000000000002,"15min":0.64800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81000000000000005}]},{"osd":9,"last update":"Fri Apr 5 23:31:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.69399999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36080,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709072,"statfs":{"total":95995035648,"available":95958089728,"internally_reserved":0,"allocated":892928,"data_stored":414191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 23:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.52300000000000002,"15min":0.499},"min":{"1min":0.41899999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.51600000000000001,"15min":0.49199999999999999},"min":{"1min":0.41899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Apr 5 23:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.59999999999999998,"15min":0.57799999999999996},"min":{"1min":0.53800000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.68700000000000006,"15min":0.59999999999999998},"min":{"1min":0.49099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1379999999999999,"5min":7.2060000000000004,"15min":7.2060000000000004},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Apr 5 23:30:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.54200000000000004,"15min":0.52800000000000002},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68100000000000005}]},{"osd":4,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.92700000000000005,"15min":0.96099999999999997},"min":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.2929999999999999,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.67400000000000004,"15min":0.65500000000000003},"min":{"1min":0.56100000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.621}]},{"osd":5,"last update":"Fri Apr 5 23:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.67700000000000005,"15min":0.66400000000000003},"min":{"1min":0.51800000000000002,"5min":0.434,"15min":0.434},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.872,"5min":0.72899999999999998,"15min":0.70499999999999996},"min":{"1min":0.60299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.65600000000000003}]},{"osd":6,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.88100000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.153,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.77800000000000002,"15min":0.77500000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.70699999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.77300000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.81599999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:32:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.83099999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]}]},{"osd":2,"up_from":17,"seq":73014444079,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":909312,"data_stored":434513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.624}]},{"osd":1,"last update":"Fri Apr 5 23:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75600000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54100000000000004,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":0.874,"15min":0.874},"last":0.66600000000000004}]},{"osd":3,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55500000000000005,"15min":0.53800000000000003},"min":{"1min":0.37,"5min":0.311,"15min":0.311},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Fri Apr 5 23:31:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.67600000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.54500000000000004},"min":{"1min":0.43099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40999999999999998,"15min":0.40799999999999997},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68200000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.48099999999999998,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78500000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93899999999999995,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73199999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.438},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.47599999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.746},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.68700000000000006}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]},{"osd":3,"up_from":23,"seq":98784247849,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36048,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709104,"statfs":{"total":95995035648,"available":95958122496,"internally_reserved":0,"allocated":856064,"data_stored":411892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.45100000000000001},"min":{"1min":0.33500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.018,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51000000000000001,"15min":0.46300000000000002},"min":{"1min":0.379,"5min":0.313,"15min":0.313},"max":{"1min":1.0860000000000001,"5min":4.96,"15min":4.96},"last":0.996}]},{"osd":1,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64000000000000001,"15min":0.60599999999999998},"min":{"1min":0.44400000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91200000000000003,"5min":4.5590000000000002,"15min":4.5590000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.61599999999999999,"15min":0.59599999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71599999999999997,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.72799999999999998,"15min":0.67800000000000005},"min":{"1min":0.51400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.028,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71099999999999997,"15min":0.67300000000000004},"min":{"1min":0.57199999999999995,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94399999999999995,"5min":4.6710000000000003,"15min":4.6710000000000003},"last":0.67500000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.69699999999999995,"15min":0.66300000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.8529999999999998,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.69699999999999995,"15min":0.66000000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":4.6289999999999996,"5min":4.6289999999999996,"15min":4.6289999999999996},"last":1.028}]},{"osd":5,"last update":"Fri Apr 5 23:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.73699999999999999,"15min":0.69299999999999995},"min":{"1min":0.57699999999999996,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.76900000000000002,"15min":0.72299999999999998},"min":{"1min":0.61299999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":4.8929999999999998,"5min":4.8929999999999998,"15min":4.8929999999999998},"last":0.73599999999999999}]},{"osd":6,"last update":"Fri Apr 5 23:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.878,"15min":0.88300000000000001},"min":{"1min":0.63400000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.228,"5min":4.8159999999999998,"15min":4.8159999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.78600000000000003,"15min":0.81399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":5.0979999999999999,"15min":5.0979999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.9980000000000002,"5min":4.9980000000000002,"15min":4.9980000000000002},"last":1.2929999999999999},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":4.7290000000000001,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.80500000000000005}]},{"osd":8,"last update":"Fri Apr 5 23:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":4.7569999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.0300000000000002,"5min":5.0300000000000002,"15min":5.0300000000000002},"last":0.72199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.874},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.84299999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":412440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.35799999999999998},"min":{"1min":0.29499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.49299999999999999,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38100000000000001,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.503,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66600000000000004,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54200000000000004,"15min":0.54100000000000004},"min":{"1min":0.41199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.66700000000000004,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.59599999999999997}]},{"osd":3,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.55800000000000005},"min":{"1min":0.44,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56000000000000005,"15min":0.55500000000000005},"min":{"1min":0.38,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56599999999999995}]},{"osd":5,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.47399999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.502},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.61299999999999999,"15min":0.60099999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.58399999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.61299999999999999,"15min":0.59499999999999997},"min":{"1min":0.435,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.72999999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.436}]},{"osd":8,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.751}]},{"osd":9,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.71299999999999997}]},{"osd":10,"last update":"Fri Apr 5 23:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.84299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":5,"up_from":35,"seq":150323855393,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1486848,"data_stored":1004619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.38600000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55900000000000005,"15min":0.55500000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.434}]},{"osd":1,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.75,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77100000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63600000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.51100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.5}]},{"osd":3,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.437,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.42399999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73999999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59199999999999997}]},{"osd":6,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65700000000000003,"15min":0.65900000000000003},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.78900000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77900000000000003,"5min":1.153,"15min":1.153},"last":0.60299999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.624}]},{"osd":8,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.72499999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.754},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.71299999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":45207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":22537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T23:32:14.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-05T23:32:14.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.645+0000 7f0c7fe00700 1 -- 172.21.15.16:0/2390019460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78073940 msgr2=0x7f0c78073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.645+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/2390019460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78073940 0x7f0c78073dc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009990 tx=0x7f0c64009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.645+0000 7f0c7fe00700 1 -- 172.21.15.16:0/2390019460 shutdown_connections 2024-04-05T23:32:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.645+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/2390019460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c78108c70 0x7f0c7810b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.645+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/2390019460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c78074f20 0x7f0c78073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.645+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/2390019460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78073940 0x7f0c78073dc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.645+0000 7f0c7fe00700 1 -- 172.21.15.16:0/2390019460 >> 172.21.15.16:0/2390019460 conn(0x7f0c780fc320 msgr2=0x7f0c780fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.645+0000 7f0c7fe00700 1 -- 172.21.15.16:0/2390019460 shutdown_connections 2024-04-05T23:32:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.645+0000 7f0c7fe00700 1 -- 172.21.15.16:0/2390019460 wait complete. 2024-04-05T23:32:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.646+0000 7f0c7fe00700 1 Processor -- start 2024-04-05T23:32:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.646+0000 7f0c7fe00700 1 -- start start 2024-04-05T23:32:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c78073940 0x7f0c78198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c78074f20 0x7f0c781994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78108c70 0x7f0c781a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7fe00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781a0ac0 con 0x7f0c78108c70 2024-04-05T23:32:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7fe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781a0c00 con 0x7f0c78074f20 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7fe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781a0d90 con 0x7f0c78073940 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78108c70 0x7f0c781a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c78074f20 0x7f0c781994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78108c70 0x7f0c781a04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48500/0 (socket says 172.21.15.16:48500) 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7de00700 1 -- 172.21.15.16:0/4013477415 learned_addr learned my addr 172.21.15.16:0/4013477415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7d400700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c78073940 0x7f0c78198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7de00700 1 -- 172.21.15.16:0/4013477415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c78073940 msgr2=0x7f0c78198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7de00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c78073940 0x7f0c78198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7de00700 1 -- 172.21.15.16:0/4013477415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c78074f20 msgr2=0x7f0c781994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7de00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c78074f20 0x7f0c781994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7de00700 1 -- 172.21.15.16:0/4013477415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c640095f0 con 0x7f0c78108c70 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.647+0000 7f0c7ca00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c78074f20 0x7f0c781994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.648+0000 7f0c7de00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78108c70 0x7f0c781a04f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000ef30 tx=0x7f0c7000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.648+0000 7f0c76000700 1 -- 172.21.15.16:0/4013477415 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c70018070 con 0x7f0c78108c70 2024-04-05T23:32:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.648+0000 7f0c76000700 1 -- 172.21.15.16:0/4013477415 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c7000f040 con 0x7f0c78108c70 2024-04-05T23:32:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.648+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c781a1010 con 0x7f0c78108c70 2024-04-05T23:32:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.648+0000 7f0c76000700 1 -- 172.21.15.16:0/4013477415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7000cd00 con 0x7f0c78108c70 2024-04-05T23:32:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.648+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c781a1520 con 0x7f0c78108c70 2024-04-05T23:32:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.649+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c78107a30 con 0x7f0c78108c70 2024-04-05T23:32:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.651+0000 7f0c76000700 1 -- 172.21.15.16:0/4013477415 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f0c70003a00 con 0x7f0c78108c70 2024-04-05T23:32:14.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.652+0000 7f0c76000700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0c600a11b0 0x7f0c600a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:14.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.653+0000 7f0c76000700 1 -- 172.21.15.16:0/4013477415 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0c700cc4f0 con 0x7f0c78108c70 2024-04-05T23:32:14.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.653+0000 7f0c7d400700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0c600a11b0 0x7f0c600a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:14.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.653+0000 7f0c7d400700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0c600a11b0 0x7f0c600a3670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009c60 tx=0x7f0c6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.657+0000 7f0c76000700 1 -- 172.21.15.16:0/4013477415 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0c70095c60 con 0x7f0c78108c70 2024-04-05T23:32:14.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:14 smithi027 ceph-mon[78597]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:14.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:14 smithi027 ceph-mon[78597]: pgmap v237: 97 pgs: 97 active+clean; 602 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9.2 KiB/s wr, 24 op/s 2024-04-05T23:32:14.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:14 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3552222405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:32:14.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:14 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4205277373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:14.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:14 smithi016 ceph-mon[38248]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:14.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:14 smithi016 ceph-mon[38248]: pgmap v237: 97 pgs: 97 active+clean; 602 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9.2 KiB/s wr, 24 op/s 2024-04-05T23:32:14.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:14 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3552222405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:32:14.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:14 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4205277373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:14.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:14 smithi130 ceph-mon[79417]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:32:14.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:14 smithi130 ceph-mon[79417]: pgmap v237: 97 pgs: 97 active+clean; 602 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9.2 KiB/s wr, 24 op/s 2024-04-05T23:32:14.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:14 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3552222405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:32:14.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:14 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4205277373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.870+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c7802ce00 con 0x7f0c600a11b0 2024-04-05T23:32:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.880+0000 7f0c76000700 1 -- 172.21.15.16:0/4013477415 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+333654 (secure 0 0 0) 0x7f0c7802ce00 con 0x7f0c600a11b0 2024-04-05T23:32:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0c600a11b0 msgr2=0x7f0c600a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0c600a11b0 0x7f0c600a3670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009c60 tx=0x7f0c6c009260 comp rx=0 tx=0).stop 2024-04-05T23:32:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78108c70 msgr2=0x7f0c781a04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78108c70 0x7f0c781a04f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000ef30 tx=0x7f0c7000c3a0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 shutdown_connections 2024-04-05T23:32:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0c600a11b0 0x7f0c600a3670 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c78108c70 0x7f0c781a04f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c78073940 0x7f0c78198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 --2- 172.21.15.16:0/4013477415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c78074f20 0x7f0c781994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 >> 172.21.15.16:0/4013477415 conn(0x7f0c780fc320 msgr2=0x7f0c78104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 shutdown_connections 2024-04-05T23:32:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:14.885+0000 7f0c7fe00700 1 -- 172.21.15.16:0/4013477415 wait complete. 2024-04-05T23:32:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-05T23:32:14.896 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-04-05T23:32:13.293797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616802,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":174,"num_read_kb":153,"num_write":357,"num_write_kb":3019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364896,"kb_used_data":12028,"kb_used_omap":18,"kb_used_meta":352749,"kb_avail":1124576928,"statfs":{"total":1151940427776,"available":1151566774272,"internally_reserved":0,"allocated":12316672,"data_stored":6759210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":361215301},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26434,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":38,"num_read_kb":34,"num_write":176,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.779254"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561860+0000","last_change":"2024-04-05T23:32:04.536879+0000","last_active":"2024-04-05T23:32:05.561860+0000","last_peered":"2024-04-05T23:32:05.561860+0000","last_clean":"2024-04-05T23:32:05.561860+0000","last_became_active":"2024-04-05T23:32:04.536183+0000","last_became_peered":"2024-04-05T23:32:04.536183+0000","last_unstale":"2024-04-05T23:32:05.561860+0000","last_undegraded":"2024-04-05T23:32:05.561860+0000","last_fullsized":"2024-04-05T23:32:05.561860+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:13:22.589464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561585+0000","last_change":"2024-04-05T23:32:04.540791+0000","last_active":"2024-04-05T23:32:05.561585+0000","last_peered":"2024-04-05T23:32:05.561585+0000","last_clean":"2024-04-05T23:32:05.561585+0000","last_became_active":"2024-04-05T23:32:04.540455+0000","last_became_peered":"2024-04-05T23:32:04.540455+0000","last_unstale":"2024-04-05T23:32:05.561585+0000","last_undegraded":"2024-04-05T23:32:05.561585+0000","last_fullsized":"2024-04-05T23:32:05.561585+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:06:19.090672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561467+0000","last_change":"2024-04-05T23:32:04.543428+0000","last_active":"2024-04-05T23:32:05.561467+0000","last_peered":"2024-04-05T23:32:05.561467+0000","last_clean":"2024-04-05T23:32:05.561467+0000","last_became_active":"2024-04-05T23:32:04.543071+0000","last_became_peered":"2024-04-05T23:32:04.543071+0000","last_unstale":"2024-04-05T23:32:05.561467+0000","last_undegraded":"2024-04-05T23:32:05.561467+0000","last_fullsized":"2024-04-05T23:32:05.561467+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:35:44.911749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561794+0000","last_change":"2024-04-05T23:32:04.531444+0000","last_active":"2024-04-05T23:32:05.561794+0000","last_peered":"2024-04-05T23:32:05.561794+0000","last_clean":"2024-04-05T23:32:05.561794+0000","last_became_active":"2024-04-05T23:32:04.529556+0000","last_became_peered":"2024-04-05T23:32:04.529556+0000","last_unstale":"2024-04-05T23:32:05.561794+0000","last_undegraded":"2024-04-05T23:32:05.561794+0000","last_fullsized":"2024-04-05T23:32:05.561794+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:48.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.537368+0000","last_change":"2024-04-05T23:32:04.529565+0000","last_active":"2024-04-05T23:32:04.537368+0000","last_peered":"2024-04-05T23:32:04.537368+0000","last_clean":"2024-04-05T23:32:04.537368+0000","last_became_active":"2024-04-05T23:32:04.529226+0000","last_became_peered":"2024-04-05T23:32:04.529226+0000","last_unstale":"2024-04-05T23:32:04.537368+0000","last_undegraded":"2024-04-05T23:32:04.537368+0000","last_fullsized":"2024-04-05T23:32:04.537368+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:44:25.208996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562159+0000","last_change":"2024-04-05T23:32:04.546260+0000","last_active":"2024-04-05T23:32:05.562159+0000","last_peered":"2024-04-05T23:32:05.562159+0000","last_clean":"2024-04-05T23:32:05.562159+0000","last_became_active":"2024-04-05T23:32:04.545851+0000","last_became_peered":"2024-04-05T23:32:04.545851+0000","last_unstale":"2024-04-05T23:32:05.562159+0000","last_undegraded":"2024-04-05T23:32:05.562159+0000","last_fullsized":"2024-04-05T23:32:05.562159+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:59:16.735635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.560777+0000","last_change":"2024-04-05T23:32:04.536348+0000","last_active":"2024-04-05T23:32:05.560777+0000","last_peered":"2024-04-05T23:32:05.560777+0000","last_clean":"2024-04-05T23:32:05.560777+0000","last_became_active":"2024-04-05T23:32:04.536015+0000","last_became_peered":"2024-04-05T23:32:04.536015+0000","last_unstale":"2024-04-05T23:32:05.560777+0000","last_undegraded":"2024-04-05T23:32:05.560777+0000","last_fullsized":"2024-04-05T23:32:05.560777+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:02:24.897957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522127+0000","last_change":"2024-04-05T23:32:04.532528+0000","last_active":"2024-04-05T23:32:05.522127+0000","last_peered":"2024-04-05T23:32:05.522127+0000","last_clean":"2024-04-05T23:32:05.522127+0000","last_became_active":"2024-04-05T23:32:04.532104+0000","last_became_peered":"2024-04-05T23:32:04.532104+0000","last_unstale":"2024-04-05T23:32:05.522127+0000","last_undegraded":"2024-04-05T23:32:05.522127+0000","last_fullsized":"2024-04-05T23:32:05.522127+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:06.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390633+0000","last_change":"2024-04-05T23:32:04.531223+0000","last_active":"2024-04-05T23:32:05.390633+0000","last_peered":"2024-04-05T23:32:05.390633+0000","last_clean":"2024-04-05T23:32:05.390633+0000","last_became_active":"2024-04-05T23:32:04.530863+0000","last_became_peered":"2024-04-05T23:32:04.530863+0000","last_unstale":"2024-04-05T23:32:05.390633+0000","last_undegraded":"2024-04-05T23:32:05.390633+0000","last_fullsized":"2024-04-05T23:32:05.390633+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:03:35.912984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562452+0000","last_change":"2024-04-05T23:32:04.541481+0000","last_active":"2024-04-05T23:32:05.562452+0000","last_peered":"2024-04-05T23:32:05.562452+0000","last_clean":"2024-04-05T23:32:05.562452+0000","last_became_active":"2024-04-05T23:32:04.541133+0000","last_became_peered":"2024-04-05T23:32:04.541133+0000","last_unstale":"2024-04-05T23:32:05.562452+0000","last_undegraded":"2024-04-05T23:32:05.562452+0000","last_fullsized":"2024-04-05T23:32:05.562452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:01:22.825587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562374+0000","last_change":"2024-04-05T23:32:04.542390+0000","last_active":"2024-04-05T23:32:05.562374+0000","last_peered":"2024-04-05T23:32:05.562374+0000","last_clean":"2024-04-05T23:32:05.562374+0000","last_became_active":"2024-04-05T23:32:04.542026+0000","last_became_peered":"2024-04-05T23:32:04.542026+0000","last_unstale":"2024-04-05T23:32:05.562374+0000","last_undegraded":"2024-04-05T23:32:05.562374+0000","last_fullsized":"2024-04-05T23:32:05.562374+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:15:42.243807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561754+0000","last_change":"2024-04-05T23:32:04.540096+0000","last_active":"2024-04-05T23:32:05.561754+0000","last_peered":"2024-04-05T23:32:05.561754+0000","last_clean":"2024-04-05T23:32:05.561754+0000","last_became_active":"2024-04-05T23:32:04.539764+0000","last_became_peered":"2024-04-05T23:32:04.539764+0000","last_unstale":"2024-04-05T23:32:05.561754+0000","last_undegraded":"2024-04-05T23:32:05.561754+0000","last_fullsized":"2024-04-05T23:32:05.561754+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:57:36.442393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561070+0000","last_change":"2024-04-05T23:32:04.540839+0000","last_active":"2024-04-05T23:32:05.561070+0000","last_peered":"2024-04-05T23:32:05.561070+0000","last_clean":"2024-04-05T23:32:05.561070+0000","last_became_active":"2024-04-05T23:32:04.540511+0000","last_became_peered":"2024-04-05T23:32:04.540511+0000","last_unstale":"2024-04-05T23:32:05.561070+0000","last_undegraded":"2024-04-05T23:32:05.561070+0000","last_fullsized":"2024-04-05T23:32:05.561070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:58:24.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561331+0000","last_change":"2024-04-05T23:32:04.539222+0000","last_active":"2024-04-05T23:32:05.561331+0000","last_peered":"2024-04-05T23:32:05.561331+0000","last_clean":"2024-04-05T23:32:05.561331+0000","last_became_active":"2024-04-05T23:32:04.538906+0000","last_became_peered":"2024-04-05T23:32:04.538906+0000","last_unstale":"2024-04-05T23:32:05.561331+0000","last_undegraded":"2024-04-05T23:32:05.561331+0000","last_fullsized":"2024-04-05T23:32:05.561331+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:05:24.429153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561895+0000","last_change":"2024-04-05T23:32:04.544155+0000","last_active":"2024-04-05T23:32:05.561895+0000","last_peered":"2024-04-05T23:32:05.561895+0000","last_clean":"2024-04-05T23:32:05.561895+0000","last_became_active":"2024-04-05T23:32:04.543852+0000","last_became_peered":"2024-04-05T23:32:04.543852+0000","last_unstale":"2024-04-05T23:32:05.561895+0000","last_undegraded":"2024-04-05T23:32:05.561895+0000","last_fullsized":"2024-04-05T23:32:05.561895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:18:34.353542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390483+0000","last_change":"2024-04-05T23:32:04.533772+0000","last_active":"2024-04-05T23:32:05.390483+0000","last_peered":"2024-04-05T23:32:05.390483+0000","last_clean":"2024-04-05T23:32:05.390483+0000","last_became_active":"2024-04-05T23:32:04.533416+0000","last_became_peered":"2024-04-05T23:32:04.533416+0000","last_unstale":"2024-04-05T23:32:05.390483+0000","last_undegraded":"2024-04-05T23:32:05.390483+0000","last_fullsized":"2024-04-05T23:32:05.390483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:36:55.072349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561961+0000","last_change":"2024-04-05T23:32:04.551902+0000","last_active":"2024-04-05T23:32:05.561961+0000","last_peered":"2024-04-05T23:32:05.561961+0000","last_clean":"2024-04-05T23:32:05.561961+0000","last_became_active":"2024-04-05T23:32:04.551595+0000","last_became_peered":"2024-04-05T23:32:04.551595+0000","last_unstale":"2024-04-05T23:32:05.561961+0000","last_undegraded":"2024-04-05T23:32:05.561961+0000","last_fullsized":"2024-04-05T23:32:05.561961+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:14:46.648789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.546726+0000","last_change":"2024-04-05T23:32:04.542967+0000","last_active":"2024-04-05T23:32:04.546726+0000","last_peered":"2024-04-05T23:32:04.546726+0000","last_clean":"2024-04-05T23:32:04.546726+0000","last_became_active":"2024-04-05T23:32:04.541767+0000","last_became_peered":"2024-04-05T23:32:04.541767+0000","last_unstale":"2024-04-05T23:32:04.546726+0000","last_undegraded":"2024-04-05T23:32:04.546726+0000","last_fullsized":"2024-04-05T23:32:04.546726+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:27:46.150249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561256+0000","last_change":"2024-04-05T23:32:04.549512+0000","last_active":"2024-04-05T23:32:05.561256+0000","last_peered":"2024-04-05T23:32:05.561256+0000","last_clean":"2024-04-05T23:32:05.561256+0000","last_became_active":"2024-04-05T23:32:04.549184+0000","last_became_peered":"2024-04-05T23:32:04.549184+0000","last_unstale":"2024-04-05T23:32:05.561256+0000","last_undegraded":"2024-04-05T23:32:05.561256+0000","last_fullsized":"2024-04-05T23:32:05.561256+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:48:24.186408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.014599+0000","last_change":"2024-04-05T23:32:04.005188+0000","last_active":"2024-04-05T23:32:09.014599+0000","last_peered":"2024-04-05T23:32:09.014599+0000","last_clean":"2024-04-05T23:32:09.014599+0000","last_became_active":"2024-04-05T23:32:04.004818+0000","last_became_peered":"2024-04-05T23:32:04.004818+0000","last_unstale":"2024-04-05T23:32:09.014599+0000","last_undegraded":"2024-04-05T23:32:09.014599+0000","last_fullsized":"2024-04-05T23:32:09.014599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:49:41.146761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522767+0000","last_change":"2024-04-05T23:32:04.544881+0000","last_active":"2024-04-05T23:32:05.522767+0000","last_peered":"2024-04-05T23:32:05.522767+0000","last_clean":"2024-04-05T23:32:05.522767+0000","last_became_active":"2024-04-05T23:32:04.542149+0000","last_became_peered":"2024-04-05T23:32:04.542149+0000","last_unstale":"2024-04-05T23:32:05.522767+0000","last_undegraded":"2024-04-05T23:32:05.522767+0000","last_fullsized":"2024-04-05T23:32:05.522767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:28:51.713053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.549110+0000","last_change":"2024-04-05T23:32:04.011450+0000","last_active":"2024-04-05T23:32:04.549110+0000","last_peered":"2024-04-05T23:32:04.549110+0000","last_clean":"2024-04-05T23:32:04.549110+0000","last_became_active":"2024-04-05T23:32:04.009607+0000","last_became_peered":"2024-04-05T23:32:04.009607+0000","last_unstale":"2024-04-05T23:32:04.549110+0000","last_undegraded":"2024-04-05T23:32:04.549110+0000","last_fullsized":"2024-04-05T23:32:04.549110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:16:45.483108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561707+0000","last_change":"2024-04-05T23:32:04.550370+0000","last_active":"2024-04-05T23:32:05.561707+0000","last_peered":"2024-04-05T23:32:05.561707+0000","last_clean":"2024-04-05T23:32:05.561707+0000","last_became_active":"2024-04-05T23:32:04.549980+0000","last_became_peered":"2024-04-05T23:32:04.549980+0000","last_unstale":"2024-04-05T23:32:05.561707+0000","last_undegraded":"2024-04-05T23:32:05.561707+0000","last_fullsized":"2024-04-05T23:32:05.561707+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:44:30.704366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.011377+0000","last_change":"2024-04-05T23:32:04.004169+0000","last_active":"2024-04-05T23:32:09.011377+0000","last_peered":"2024-04-05T23:32:09.011377+0000","last_clean":"2024-04-05T23:32:09.011377+0000","last_became_active":"2024-04-05T23:32:04.003646+0000","last_became_peered":"2024-04-05T23:32:04.003646+0000","last_unstale":"2024-04-05T23:32:09.011377+0000","last_undegraded":"2024-04-05T23:32:09.011377+0000","last_fullsized":"2024-04-05T23:32:09.011377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:57:02.781815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390563+0000","last_change":"2024-04-05T23:32:04.542066+0000","last_active":"2024-04-05T23:32:05.390563+0000","last_peered":"2024-04-05T23:32:05.390563+0000","last_clean":"2024-04-05T23:32:05.390563+0000","last_became_active":"2024-04-05T23:32:04.540735+0000","last_became_peered":"2024-04-05T23:32:04.540735+0000","last_unstale":"2024-04-05T23:32:05.390563+0000","last_undegraded":"2024-04-05T23:32:05.390563+0000","last_fullsized":"2024-04-05T23:32:05.390563+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:02:03.595888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.544085+0000","last_change":"2024-04-05T23:32:03.999943+0000","last_active":"2024-04-05T23:32:04.544085+0000","last_peered":"2024-04-05T23:32:04.544085+0000","last_clean":"2024-04-05T23:32:04.544085+0000","last_became_active":"2024-04-05T23:32:03.999518+0000","last_became_peered":"2024-04-05T23:32:03.999518+0000","last_unstale":"2024-04-05T23:32:04.544085+0000","last_undegraded":"2024-04-05T23:32:04.544085+0000","last_fullsized":"2024-04-05T23:32:04.544085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:27:52.486637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346372+0000","last_change":"2024-04-05T23:32:04.545780+0000","last_active":"2024-04-05T23:32:05.346372+0000","last_peered":"2024-04-05T23:32:05.346372+0000","last_clean":"2024-04-05T23:32:05.346372+0000","last_became_active":"2024-04-05T23:32:04.545279+0000","last_became_peered":"2024-04-05T23:32:04.545279+0000","last_unstale":"2024-04-05T23:32:05.346372+0000","last_undegraded":"2024-04-05T23:32:05.346372+0000","last_fullsized":"2024-04-05T23:32:05.346372+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:10.577280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.411051+0000","last_change":"2024-04-05T23:32:04.004189+0000","last_active":"2024-04-05T23:32:09.411051+0000","last_peered":"2024-04-05T23:32:09.411051+0000","last_clean":"2024-04-05T23:32:09.411051+0000","last_became_active":"2024-04-05T23:32:04.003841+0000","last_became_peered":"2024-04-05T23:32:04.003841+0000","last_unstale":"2024-04-05T23:32:09.411051+0000","last_undegraded":"2024-04-05T23:32:09.411051+0000","last_fullsized":"2024-04-05T23:32:09.411051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:33:26.898657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.537327+0000","last_change":"2024-04-05T23:32:04.529538+0000","last_active":"2024-04-05T23:32:04.537327+0000","last_peered":"2024-04-05T23:32:04.537327+0000","last_clean":"2024-04-05T23:32:04.537327+0000","last_became_active":"2024-04-05T23:32:04.529231+0000","last_became_peered":"2024-04-05T23:32:04.529231+0000","last_unstale":"2024-04-05T23:32:04.537327+0000","last_undegraded":"2024-04-05T23:32:04.537327+0000","last_fullsized":"2024-04-05T23:32:04.537327+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:49:15.608966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.421236+0000","last_change":"2024-04-05T23:32:04.002763+0000","last_active":"2024-04-05T23:32:09.421236+0000","last_peered":"2024-04-05T23:32:09.421236+0000","last_clean":"2024-04-05T23:32:09.421236+0000","last_became_active":"2024-04-05T23:32:04.002410+0000","last_became_peered":"2024-04-05T23:32:04.002410+0000","last_unstale":"2024-04-05T23:32:09.421236+0000","last_undegraded":"2024-04-05T23:32:09.421236+0000","last_fullsized":"2024-04-05T23:32:09.421236+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:15.193813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346346+0000","last_change":"2024-04-05T23:32:04.545381+0000","last_active":"2024-04-05T23:32:05.346346+0000","last_peered":"2024-04-05T23:32:05.346346+0000","last_clean":"2024-04-05T23:32:05.346346+0000","last_became_active":"2024-04-05T23:32:04.545128+0000","last_became_peered":"2024-04-05T23:32:04.545128+0000","last_unstale":"2024-04-05T23:32:05.346346+0000","last_undegraded":"2024-04-05T23:32:05.346346+0000","last_fullsized":"2024-04-05T23:32:05.346346+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:45:21.931160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.014686+0000","last_change":"2024-04-05T23:32:04.001937+0000","last_active":"2024-04-05T23:32:09.014686+0000","last_peered":"2024-04-05T23:32:09.014686+0000","last_clean":"2024-04-05T23:32:09.014686+0000","last_became_active":"2024-04-05T23:32:04.001508+0000","last_became_peered":"2024-04-05T23:32:04.001508+0000","last_unstale":"2024-04-05T23:32:09.014686+0000","last_undegraded":"2024-04-05T23:32:09.014686+0000","last_fullsized":"2024-04-05T23:32:09.014686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:09:20.682084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522193+0000","last_change":"2024-04-05T23:32:04.531576+0000","last_active":"2024-04-05T23:32:05.522193+0000","last_peered":"2024-04-05T23:32:05.522193+0000","last_clean":"2024-04-05T23:32:05.522193+0000","last_became_active":"2024-04-05T23:32:04.531234+0000","last_became_peered":"2024-04-05T23:32:04.531234+0000","last_unstale":"2024-04-05T23:32:05.522193+0000","last_undegraded":"2024-04-05T23:32:05.522193+0000","last_fullsized":"2024-04-05T23:32:05.522193+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:21:57.769861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'2","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.008501+0000","last_change":"2024-04-05T23:32:04.001570+0000","last_active":"2024-04-05T23:32:08.008501+0000","last_peered":"2024-04-05T23:32:08.008501+0000","last_clean":"2024-04-05T23:32:08.008501+0000","last_became_active":"2024-04-05T23:32:04.001224+0000","last_became_peered":"2024-04-05T23:32:04.001224+0000","last_unstale":"2024-04-05T23:32:08.008501+0000","last_undegraded":"2024-04-05T23:32:08.008501+0000","last_fullsized":"2024-04-05T23:32:08.008501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:44:41.070205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.546699+0000","last_change":"2024-04-05T23:32:04.542029+0000","last_active":"2024-04-05T23:32:04.546699+0000","last_peered":"2024-04-05T23:32:04.546699+0000","last_clean":"2024-04-05T23:32:04.546699+0000","last_became_active":"2024-04-05T23:32:04.541684+0000","last_became_peered":"2024-04-05T23:32:04.541684+0000","last_unstale":"2024-04-05T23:32:04.546699+0000","last_undegraded":"2024-04-05T23:32:04.546699+0000","last_fullsized":"2024-04-05T23:32:04.546699+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:07:06.456917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:11.414830+0000","last_change":"2024-04-05T23:32:04.004986+0000","last_active":"2024-04-05T23:32:11.414830+0000","last_peered":"2024-04-05T23:32:11.414830+0000","last_clean":"2024-04-05T23:32:11.414830+0000","last_became_active":"2024-04-05T23:32:04.004606+0000","last_became_peered":"2024-04-05T23:32:04.004606+0000","last_unstale":"2024-04-05T23:32:11.414830+0000","last_undegraded":"2024-04-05T23:32:11.414830+0000","last_fullsized":"2024-04-05T23:32:11.414830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:23:41.060500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390610+0000","last_change":"2024-04-05T23:32:04.541334+0000","last_active":"2024-04-05T23:32:05.390610+0000","last_peered":"2024-04-05T23:32:05.390610+0000","last_clean":"2024-04-05T23:32:05.390610+0000","last_became_active":"2024-04-05T23:32:04.540967+0000","last_became_peered":"2024-04-05T23:32:04.540967+0000","last_unstale":"2024-04-05T23:32:05.390610+0000","last_undegraded":"2024-04-05T23:32:05.390610+0000","last_fullsized":"2024-04-05T23:32:05.390610+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:16:33.045172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.425394+0000","last_change":"2024-04-05T23:32:04.003655+0000","last_active":"2024-04-05T23:32:10.425394+0000","last_peered":"2024-04-05T23:32:10.425394+0000","last_clean":"2024-04-05T23:32:10.425394+0000","last_became_active":"2024-04-05T23:32:04.003312+0000","last_became_peered":"2024-04-05T23:32:04.003312+0000","last_unstale":"2024-04-05T23:32:10.425394+0000","last_undegraded":"2024-04-05T23:32:10.425394+0000","last_fullsized":"2024-04-05T23:32:10.425394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:11:49.770921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561707+0000","last_change":"2024-04-05T23:32:04.533477+0000","last_active":"2024-04-05T23:32:05.561707+0000","last_peered":"2024-04-05T23:32:05.561707+0000","last_clean":"2024-04-05T23:32:05.561707+0000","last_became_active":"2024-04-05T23:32:04.533109+0000","last_became_peered":"2024-04-05T23:32:04.533109+0000","last_unstale":"2024-04-05T23:32:05.561707+0000","last_undegraded":"2024-04-05T23:32:05.561707+0000","last_fullsized":"2024-04-05T23:32:05.561707+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:35:08.383911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.411791+0000","last_change":"2024-04-05T23:32:04.000697+0000","last_active":"2024-04-05T23:32:09.411791+0000","last_peered":"2024-04-05T23:32:09.411791+0000","last_clean":"2024-04-05T23:32:09.411791+0000","last_became_active":"2024-04-05T23:32:04.000377+0000","last_became_peered":"2024-04-05T23:32:04.000377+0000","last_unstale":"2024-04-05T23:32:09.411791+0000","last_undegraded":"2024-04-05T23:32:09.411791+0000","last_fullsized":"2024-04-05T23:32:09.411791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:55:11.227637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390587+0000","last_change":"2024-04-05T23:32:04.528939+0000","last_active":"2024-04-05T23:32:05.390587+0000","last_peered":"2024-04-05T23:32:05.390587+0000","last_clean":"2024-04-05T23:32:05.390587+0000","last_became_active":"2024-04-05T23:32:04.528610+0000","last_became_peered":"2024-04-05T23:32:04.528610+0000","last_unstale":"2024-04-05T23:32:05.390587+0000","last_undegraded":"2024-04-05T23:32:05.390587+0000","last_fullsized":"2024-04-05T23:32:05.390587+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:08:35.685691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'17","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:11.450020+0000","last_change":"2024-04-05T23:32:04.007322+0000","last_active":"2024-04-05T23:32:11.450020+0000","last_peered":"2024-04-05T23:32:11.450020+0000","last_clean":"2024-04-05T23:32:11.450020+0000","last_became_active":"2024-04-05T23:32:04.006971+0000","last_became_peered":"2024-04-05T23:32:04.006971+0000","last_unstale":"2024-04-05T23:32:11.450020+0000","last_undegraded":"2024-04-05T23:32:11.450020+0000","last_fullsized":"2024-04-05T23:32:11.450020+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:04:38.413482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20849,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":12,"num_write":19,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.540756+0000","last_change":"2024-04-05T23:32:04.535342+0000","last_active":"2024-04-05T23:32:04.540756+0000","last_peered":"2024-04-05T23:32:04.540756+0000","last_clean":"2024-04-05T23:32:04.540756+0000","last_became_active":"2024-04-05T23:32:04.535053+0000","last_became_peered":"2024-04-05T23:32:04.535053+0000","last_unstale":"2024-04-05T23:32:04.540756+0000","last_undegraded":"2024-04-05T23:32:04.540756+0000","last_fullsized":"2024-04-05T23:32:04.540756+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:19:20.683836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.426401+0000","last_change":"2024-04-05T23:32:04.001911+0000","last_active":"2024-04-05T23:32:10.426401+0000","last_peered":"2024-04-05T23:32:10.426401+0000","last_clean":"2024-04-05T23:32:10.426401+0000","last_became_active":"2024-04-05T23:32:04.001530+0000","last_became_peered":"2024-04-05T23:32:04.001530+0000","last_unstale":"2024-04-05T23:32:10.426401+0000","last_undegraded":"2024-04-05T23:32:10.426401+0000","last_fullsized":"2024-04-05T23:32:10.426401+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:00:05.891629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561195+0000","last_change":"2024-04-05T23:32:04.536455+0000","last_active":"2024-04-05T23:32:05.561195+0000","last_peered":"2024-04-05T23:32:05.561195+0000","last_clean":"2024-04-05T23:32:05.561195+0000","last_became_active":"2024-04-05T23:32:04.533994+0000","last_became_peered":"2024-04-05T23:32:04.533994+0000","last_unstale":"2024-04-05T23:32:05.561195+0000","last_undegraded":"2024-04-05T23:32:05.561195+0000","last_fullsized":"2024-04-05T23:32:05.561195+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:38:18.952823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.007724+0000","last_change":"2024-04-05T23:32:04.000708+0000","last_active":"2024-04-05T23:32:08.007724+0000","last_peered":"2024-04-05T23:32:08.007724+0000","last_clean":"2024-04-05T23:32:08.007724+0000","last_became_active":"2024-04-05T23:32:04.000354+0000","last_became_peered":"2024-04-05T23:32:04.000354+0000","last_unstale":"2024-04-05T23:32:08.007724+0000","last_undegraded":"2024-04-05T23:32:08.007724+0000","last_fullsized":"2024-04-05T23:32:08.007724+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:05:34.291607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562033+0000","last_change":"2024-04-05T23:32:04.532146+0000","last_active":"2024-04-05T23:32:05.562033+0000","last_peered":"2024-04-05T23:32:05.562033+0000","last_clean":"2024-04-05T23:32:05.562033+0000","last_became_active":"2024-04-05T23:32:04.531735+0000","last_became_peered":"2024-04-05T23:32:04.531735+0000","last_unstale":"2024-04-05T23:32:05.562033+0000","last_undegraded":"2024-04-05T23:32:05.562033+0000","last_fullsized":"2024-04-05T23:32:05.562033+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:52:44.898871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:11.416606+0000","last_change":"2024-04-05T23:32:04.000144+0000","last_active":"2024-04-05T23:32:11.416606+0000","last_peered":"2024-04-05T23:32:11.416606+0000","last_clean":"2024-04-05T23:32:11.416606+0000","last_became_active":"2024-04-05T23:32:03.999712+0000","last_became_peered":"2024-04-05T23:32:03.999712+0000","last_unstale":"2024-04-05T23:32:11.416606+0000","last_undegraded":"2024-04-05T23:32:11.416606+0000","last_fullsized":"2024-04-05T23:32:11.416606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:43.775018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.560698+0000","last_change":"2024-04-05T23:32:04.537623+0000","last_active":"2024-04-05T23:32:05.560698+0000","last_peered":"2024-04-05T23:32:05.560698+0000","last_clean":"2024-04-05T23:32:05.560698+0000","last_became_active":"2024-04-05T23:32:04.537399+0000","last_became_peered":"2024-04-05T23:32:04.537399+0000","last_unstale":"2024-04-05T23:32:05.560698+0000","last_undegraded":"2024-04-05T23:32:05.560698+0000","last_fullsized":"2024-04-05T23:32:05.560698+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:01:01.643133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.014307+0000","last_change":"2024-04-05T23:32:04.006806+0000","last_active":"2024-04-05T23:32:09.014307+0000","last_peered":"2024-04-05T23:32:09.014307+0000","last_clean":"2024-04-05T23:32:09.014307+0000","last_became_active":"2024-04-05T23:32:04.006397+0000","last_became_peered":"2024-04-05T23:32:04.006397+0000","last_unstale":"2024-04-05T23:32:09.014307+0000","last_undegraded":"2024-04-05T23:32:09.014307+0000","last_fullsized":"2024-04-05T23:32:09.014307+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:42:01.758863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346287+0000","last_change":"2024-04-05T23:32:04.545952+0000","last_active":"2024-04-05T23:32:05.346287+0000","last_peered":"2024-04-05T23:32:05.346287+0000","last_clean":"2024-04-05T23:32:05.346287+0000","last_became_active":"2024-04-05T23:32:04.545528+0000","last_became_peered":"2024-04-05T23:32:04.545528+0000","last_unstale":"2024-04-05T23:32:05.346287+0000","last_undegraded":"2024-04-05T23:32:05.346287+0000","last_fullsized":"2024-04-05T23:32:05.346287+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:33:06.074173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.544361+0000","last_change":"2024-04-05T23:32:03.999841+0000","last_active":"2024-04-05T23:32:04.544361+0000","last_peered":"2024-04-05T23:32:04.544361+0000","last_clean":"2024-04-05T23:32:04.544361+0000","last_became_active":"2024-04-05T23:32:03.999418+0000","last_became_peered":"2024-04-05T23:32:03.999418+0000","last_unstale":"2024-04-05T23:32:04.544361+0000","last_undegraded":"2024-04-05T23:32:04.544361+0000","last_fullsized":"2024-04-05T23:32:04.544361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:00:41.452167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561536+0000","last_change":"2024-04-05T23:32:04.530012+0000","last_active":"2024-04-05T23:32:05.561536+0000","last_peered":"2024-04-05T23:32:05.561536+0000","last_clean":"2024-04-05T23:32:05.561536+0000","last_became_active":"2024-04-05T23:32:04.529628+0000","last_became_peered":"2024-04-05T23:32:04.529628+0000","last_unstale":"2024-04-05T23:32:05.561536+0000","last_undegraded":"2024-04-05T23:32:05.561536+0000","last_fullsized":"2024-04-05T23:32:05.561536+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:40:22.036170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.527314+0000","last_change":"2024-04-05T23:32:04.006157+0000","last_active":"2024-04-05T23:32:04.527314+0000","last_peered":"2024-04-05T23:32:04.527314+0000","last_clean":"2024-04-05T23:32:04.527314+0000","last_became_active":"2024-04-05T23:32:04.005853+0000","last_became_peered":"2024-04-05T23:32:04.005853+0000","last_unstale":"2024-04-05T23:32:04.527314+0000","last_undegraded":"2024-04-05T23:32:04.527314+0000","last_fullsized":"2024-04-05T23:32:04.527314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:48:39.849306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:07.154300+0000","last_change":"2024-04-05T23:32:04.547367+0000","last_active":"2024-04-05T23:32:07.154300+0000","last_peered":"2024-04-05T23:32:07.154300+0000","last_clean":"2024-04-05T23:32:07.154300+0000","last_became_active":"2024-04-05T23:32:04.547040+0000","last_became_peered":"2024-04-05T23:32:04.547040+0000","last_unstale":"2024-04-05T23:32:07.154300+0000","last_undegraded":"2024-04-05T23:32:07.154300+0000","last_fullsized":"2024-04-05T23:32:07.154300+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:41:41.998984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.426531+0000","last_change":"2024-04-05T23:32:04.009851+0000","last_active":"2024-04-05T23:32:10.426531+0000","last_peered":"2024-04-05T23:32:10.426531+0000","last_clean":"2024-04-05T23:32:10.426531+0000","last_became_active":"2024-04-05T23:32:04.009460+0000","last_became_peered":"2024-04-05T23:32:04.009460+0000","last_unstale":"2024-04-05T23:32:10.426531+0000","last_undegraded":"2024-04-05T23:32:10.426531+0000","last_fullsized":"2024-04-05T23:32:10.426531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:39:41.000439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561646+0000","last_change":"2024-04-05T23:32:04.549621+0000","last_active":"2024-04-05T23:32:05.561646+0000","last_peered":"2024-04-05T23:32:05.561646+0000","last_clean":"2024-04-05T23:32:05.561646+0000","last_became_active":"2024-04-05T23:32:04.549325+0000","last_became_peered":"2024-04-05T23:32:04.549325+0000","last_unstale":"2024-04-05T23:32:05.561646+0000","last_undegraded":"2024-04-05T23:32:05.561646+0000","last_fullsized":"2024-04-05T23:32:05.561646+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:11:07.101463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346275+0000","last_change":"2024-04-05T23:32:04.546768+0000","last_active":"2024-04-05T23:32:05.346275+0000","last_peered":"2024-04-05T23:32:05.346275+0000","last_clean":"2024-04-05T23:32:05.346275+0000","last_became_active":"2024-04-05T23:32:04.546471+0000","last_became_peered":"2024-04-05T23:32:04.546471+0000","last_unstale":"2024-04-05T23:32:05.346275+0000","last_undegraded":"2024-04-05T23:32:05.346275+0000","last_fullsized":"2024-04-05T23:32:05.346275+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:48:22.728976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.414884+0000","last_change":"2024-04-05T23:32:04.007093+0000","last_active":"2024-04-05T23:32:10.414884+0000","last_peered":"2024-04-05T23:32:10.414884+0000","last_clean":"2024-04-05T23:32:10.414884+0000","last_became_active":"2024-04-05T23:32:04.006836+0000","last_became_peered":"2024-04-05T23:32:04.006836+0000","last_unstale":"2024-04-05T23:32:10.414884+0000","last_undegraded":"2024-04-05T23:32:10.414884+0000","last_fullsized":"2024-04-05T23:32:10.414884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:20:42.489368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522825+0000","last_change":"2024-04-05T23:32:04.538204+0000","last_active":"2024-04-05T23:32:05.522825+0000","last_peered":"2024-04-05T23:32:05.522825+0000","last_clean":"2024-04-05T23:32:05.522825+0000","last_became_active":"2024-04-05T23:32:04.535254+0000","last_became_peered":"2024-04-05T23:32:04.535254+0000","last_unstale":"2024-04-05T23:32:05.522825+0000","last_undegraded":"2024-04-05T23:32:05.522825+0000","last_fullsized":"2024-04-05T23:32:05.522825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:06:01.059339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.425969+0000","last_change":"2024-04-05T23:32:03.997704+0000","last_active":"2024-04-05T23:32:10.425969+0000","last_peered":"2024-04-05T23:32:10.425969+0000","last_clean":"2024-04-05T23:32:10.425969+0000","last_became_active":"2024-04-05T23:32:03.997275+0000","last_became_peered":"2024-04-05T23:32:03.997275+0000","last_unstale":"2024-04-05T23:32:10.425969+0000","last_undegraded":"2024-04-05T23:32:10.425969+0000","last_fullsized":"2024-04-05T23:32:10.425969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:58.355104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":333,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.528158+0000","last_change":"2024-04-05T23:31:10.395041+0000","last_active":"2024-04-05T23:32:04.528158+0000","last_peered":"2024-04-05T23:32:04.528158+0000","last_clean":"2024-04-05T23:32:04.528158+0000","last_became_active":"2024-04-05T23:31:10.394479+0000","last_became_peered":"2024-04-05T23:31:10.394479+0000","last_unstale":"2024-04-05T23:32:04.528158+0000","last_undegraded":"2024-04-05T23:32:04.528158+0000","last_fullsized":"2024-04-05T23:32:04.528158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_clean_scrub_stamp":"2024-04-05T23:28:33.945697+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:54:10.683526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.560765+0000","last_change":"2024-04-05T23:32:04.537756+0000","last_active":"2024-04-05T23:32:05.560765+0000","last_peered":"2024-04-05T23:32:05.560765+0000","last_clean":"2024-04-05T23:32:05.560765+0000","last_became_active":"2024-04-05T23:32:04.536242+0000","last_became_peered":"2024-04-05T23:32:04.536242+0000","last_unstale":"2024-04-05T23:32:05.560765+0000","last_undegraded":"2024-04-05T23:32:05.560765+0000","last_fullsized":"2024-04-05T23:32:05.560765+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:41:07.817969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.012297+0000","last_change":"2024-04-05T23:32:04.005978+0000","last_active":"2024-04-05T23:32:09.012297+0000","last_peered":"2024-04-05T23:32:09.012297+0000","last_clean":"2024-04-05T23:32:09.012297+0000","last_became_active":"2024-04-05T23:32:04.005638+0000","last_became_peered":"2024-04-05T23:32:04.005638+0000","last_unstale":"2024-04-05T23:32:09.012297+0000","last_undegraded":"2024-04-05T23:32:09.012297+0000","last_fullsized":"2024-04-05T23:32:09.012297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:39:35.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.725108+0000","last_change":"2024-04-05T23:32:04.006727+0000","last_active":"2024-04-05T23:32:05.725108+0000","last_peered":"2024-04-05T23:32:05.725108+0000","last_clean":"2024-04-05T23:32:05.725108+0000","last_became_active":"2024-04-05T23:32:04.006414+0000","last_became_peered":"2024-04-05T23:32:04.006414+0000","last_unstale":"2024-04-05T23:32:05.725108+0000","last_undegraded":"2024-04-05T23:32:05.725108+0000","last_fullsized":"2024-04-05T23:32:05.725108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:41:28.661118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561450+0000","last_change":"2024-04-05T23:32:04.536893+0000","last_active":"2024-04-05T23:32:05.561450+0000","last_peered":"2024-04-05T23:32:05.561450+0000","last_clean":"2024-04-05T23:32:05.561450+0000","last_became_active":"2024-04-05T23:32:04.536629+0000","last_became_peered":"2024-04-05T23:32:04.536629+0000","last_unstale":"2024-04-05T23:32:05.561450+0000","last_undegraded":"2024-04-05T23:32:05.561450+0000","last_fullsized":"2024-04-05T23:32:05.561450+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:27:58.279768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'3","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.014072+0000","last_change":"2024-04-05T23:32:04.010202+0000","last_active":"2024-04-05T23:32:09.014072+0000","last_peered":"2024-04-05T23:32:09.014072+0000","last_clean":"2024-04-05T23:32:09.014072+0000","last_became_active":"2024-04-05T23:32:04.009889+0000","last_became_peered":"2024-04-05T23:32:04.009889+0000","last_unstale":"2024-04-05T23:32:09.014072+0000","last_undegraded":"2024-04-05T23:32:09.014072+0000","last_fullsized":"2024-04-05T23:32:09.014072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:12:13.121449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522648+0000","last_change":"2024-04-05T23:32:04.533576+0000","last_active":"2024-04-05T23:32:05.522648+0000","last_peered":"2024-04-05T23:32:05.522648+0000","last_clean":"2024-04-05T23:32:05.522648+0000","last_became_active":"2024-04-05T23:32:04.533066+0000","last_became_peered":"2024-04-05T23:32:04.533066+0000","last_unstale":"2024-04-05T23:32:05.522648+0000","last_undegraded":"2024-04-05T23:32:05.522648+0000","last_fullsized":"2024-04-05T23:32:05.522648+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:01:44.019210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.425968+0000","last_change":"2024-04-05T23:32:04.005096+0000","last_active":"2024-04-05T23:32:10.425968+0000","last_peered":"2024-04-05T23:32:10.425968+0000","last_clean":"2024-04-05T23:32:10.425968+0000","last_became_active":"2024-04-05T23:32:04.004702+0000","last_became_peered":"2024-04-05T23:32:04.004702+0000","last_unstale":"2024-04-05T23:32:10.425968+0000","last_undegraded":"2024-04-05T23:32:10.425968+0000","last_fullsized":"2024-04-05T23:32:10.425968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:33:09.934039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561473+0000","last_change":"2024-04-05T23:32:04.535623+0000","last_active":"2024-04-05T23:32:05.561473+0000","last_peered":"2024-04-05T23:32:05.561473+0000","last_clean":"2024-04-05T23:32:05.561473+0000","last_became_active":"2024-04-05T23:32:04.535223+0000","last_became_peered":"2024-04-05T23:32:04.535223+0000","last_unstale":"2024-04-05T23:32:05.561473+0000","last_undegraded":"2024-04-05T23:32:05.561473+0000","last_fullsized":"2024-04-05T23:32:05.561473+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:24:43.543119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.008998+0000","last_change":"2024-04-05T23:32:04.004097+0000","last_active":"2024-04-05T23:32:08.008998+0000","last_peered":"2024-04-05T23:32:08.008998+0000","last_clean":"2024-04-05T23:32:08.008998+0000","last_became_active":"2024-04-05T23:32:04.003713+0000","last_became_peered":"2024-04-05T23:32:04.003713+0000","last_unstale":"2024-04-05T23:32:08.008998+0000","last_undegraded":"2024-04-05T23:32:08.008998+0000","last_fullsized":"2024-04-05T23:32:08.008998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:47:43.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:04.540597+0000","last_change":"2024-04-05T23:32:04.538843+0000","last_active":"2024-04-05T23:32:04.540597+0000","last_peered":"2024-04-05T23:32:04.540597+0000","last_clean":"2024-04-05T23:32:04.540597+0000","last_became_active":"2024-04-05T23:32:04.538472+0000","last_became_peered":"2024-04-05T23:32:04.538472+0000","last_unstale":"2024-04-05T23:32:04.540597+0000","last_undegraded":"2024-04-05T23:32:04.540597+0000","last_fullsized":"2024-04-05T23:32:04.540597+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:40:40.863815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.006820+0000","last_change":"2024-04-05T23:32:04.013064+0000","last_active":"2024-04-05T23:32:08.006820+0000","last_peered":"2024-04-05T23:32:08.006820+0000","last_clean":"2024-04-05T23:32:08.006820+0000","last_became_active":"2024-04-05T23:32:04.012751+0000","last_became_peered":"2024-04-05T23:32:04.012751+0000","last_unstale":"2024-04-05T23:32:08.006820+0000","last_undegraded":"2024-04-05T23:32:08.006820+0000","last_fullsized":"2024-04-05T23:32:08.006820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:06:10.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561313+0000","last_change":"2024-04-05T23:32:04.529014+0000","last_active":"2024-04-05T23:32:05.561313+0000","last_peered":"2024-04-05T23:32:05.561313+0000","last_clean":"2024-04-05T23:32:05.561313+0000","last_became_active":"2024-04-05T23:32:04.528622+0000","last_became_peered":"2024-04-05T23:32:04.528622+0000","last_unstale":"2024-04-05T23:32:05.561313+0000","last_undegraded":"2024-04-05T23:32:05.561313+0000","last_fullsized":"2024-04-05T23:32:05.561313+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:46:26.390934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.877474+0000","last_change":"2024-04-05T23:32:04.002651+0000","last_active":"2024-04-05T23:32:10.877474+0000","last_peered":"2024-04-05T23:32:10.877474+0000","last_clean":"2024-04-05T23:32:10.877474+0000","last_became_active":"2024-04-05T23:32:04.002346+0000","last_became_peered":"2024-04-05T23:32:04.002346+0000","last_unstale":"2024-04-05T23:32:10.877474+0000","last_undegraded":"2024-04-05T23:32:10.877474+0000","last_fullsized":"2024-04-05T23:32:10.877474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:26:48.176254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522393+0000","last_change":"2024-04-05T23:32:04.538785+0000","last_active":"2024-04-05T23:32:05.522393+0000","last_peered":"2024-04-05T23:32:05.522393+0000","last_clean":"2024-04-05T23:32:05.522393+0000","last_became_active":"2024-04-05T23:32:04.538463+0000","last_became_peered":"2024-04-05T23:32:04.538463+0000","last_unstale":"2024-04-05T23:32:05.522393+0000","last_undegraded":"2024-04-05T23:32:05.522393+0000","last_fullsized":"2024-04-05T23:32:05.522393+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:44:38.941857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:08.006605+0000","last_change":"2024-04-05T23:32:04.003760+0000","last_active":"2024-04-05T23:32:08.006605+0000","last_peered":"2024-04-05T23:32:08.006605+0000","last_clean":"2024-04-05T23:32:08.006605+0000","last_became_active":"2024-04-05T23:32:04.003389+0000","last_became_peered":"2024-04-05T23:32:04.003389+0000","last_unstale":"2024-04-05T23:32:08.006605+0000","last_undegraded":"2024-04-05T23:32:08.006605+0000","last_fullsized":"2024-04-05T23:32:08.006605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:42:00.353651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561098+0000","last_change":"2024-04-05T23:32:04.540051+0000","last_active":"2024-04-05T23:32:05.561098+0000","last_peered":"2024-04-05T23:32:05.561098+0000","last_clean":"2024-04-05T23:32:05.561098+0000","last_became_active":"2024-04-05T23:32:04.539736+0000","last_became_peered":"2024-04-05T23:32:04.539736+0000","last_unstale":"2024-04-05T23:32:05.561098+0000","last_undegraded":"2024-04-05T23:32:05.561098+0000","last_fullsized":"2024-04-05T23:32:05.561098+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:54:14.086904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:11.450069+0000","last_change":"2024-04-05T23:32:04.006492+0000","last_active":"2024-04-05T23:32:11.450069+0000","last_peered":"2024-04-05T23:32:11.450069+0000","last_clean":"2024-04-05T23:32:11.450069+0000","last_became_active":"2024-04-05T23:32:04.006154+0000","last_became_peered":"2024-04-05T23:32:04.006154+0000","last_unstale":"2024-04-05T23:32:11.450069+0000","last_undegraded":"2024-04-05T23:32:11.450069+0000","last_fullsized":"2024-04-05T23:32:11.450069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:53:23.431400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.562475+0000","last_change":"2024-04-05T23:32:04.540757+0000","last_active":"2024-04-05T23:32:05.562475+0000","last_peered":"2024-04-05T23:32:05.562475+0000","last_clean":"2024-04-05T23:32:05.562475+0000","last_became_active":"2024-04-05T23:32:04.540359+0000","last_became_peered":"2024-04-05T23:32:04.540359+0000","last_unstale":"2024-04-05T23:32:05.562475+0000","last_undegraded":"2024-04-05T23:32:05.562475+0000","last_fullsized":"2024-04-05T23:32:05.562475+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:43:52.533489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'2","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:09.409996+0000","last_change":"2024-04-05T23:32:04.002736+0000","last_active":"2024-04-05T23:32:09.409996+0000","last_peered":"2024-04-05T23:32:09.409996+0000","last_clean":"2024-04-05T23:32:09.409996+0000","last_became_active":"2024-04-05T23:32:04.002277+0000","last_became_peered":"2024-04-05T23:32:04.002277+0000","last_unstale":"2024-04-05T23:32:09.409996+0000","last_undegraded":"2024-04-05T23:32:09.409996+0000","last_fullsized":"2024-04-05T23:32:09.409996+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:56:39.001732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561684+0000","last_change":"2024-04-05T23:32:04.537076+0000","last_active":"2024-04-05T23:32:05.561684+0000","last_peered":"2024-04-05T23:32:05.561684+0000","last_clean":"2024-04-05T23:32:05.561684+0000","last_became_active":"2024-04-05T23:32:04.536707+0000","last_became_peered":"2024-04-05T23:32:04.536707+0000","last_unstale":"2024-04-05T23:32:05.561684+0000","last_undegraded":"2024-04-05T23:32:05.561684+0000","last_fullsized":"2024-04-05T23:32:05.561684+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:51:09.395408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:10.425335+0000","last_change":"2024-04-05T23:32:04.006122+0000","last_active":"2024-04-05T23:32:10.425335+0000","last_peered":"2024-04-05T23:32:10.425335+0000","last_clean":"2024-04-05T23:32:10.425335+0000","last_became_active":"2024-04-05T23:32:04.005683+0000","last_became_peered":"2024-04-05T23:32:04.005683+0000","last_unstale":"2024-04-05T23:32:10.425335+0000","last_undegraded":"2024-04-05T23:32:10.425335+0000","last_fullsized":"2024-04-05T23:32:10.425335+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:21:40.922614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561891+0000","last_change":"2024-04-05T23:32:04.533321+0000","last_active":"2024-04-05T23:32:05.561891+0000","last_peered":"2024-04-05T23:32:05.561891+0000","last_clean":"2024-04-05T23:32:05.561891+0000","last_became_active":"2024-04-05T23:32:04.532949+0000","last_became_peered":"2024-04-05T23:32:04.532949+0000","last_unstale":"2024-04-05T23:32:05.561891+0000","last_undegraded":"2024-04-05T23:32:05.561891+0000","last_fullsized":"2024-04-05T23:32:05.561891+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:31.500132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561404+0000","last_change":"2024-04-05T23:32:04.536508+0000","last_active":"2024-04-05T23:32:05.561404+0000","last_peered":"2024-04-05T23:32:05.561404+0000","last_clean":"2024-04-05T23:32:05.561404+0000","last_became_active":"2024-04-05T23:32:04.536142+0000","last_became_peered":"2024-04-05T23:32:04.536142+0000","last_unstale":"2024-04-05T23:32:05.561404+0000","last_undegraded":"2024-04-05T23:32:05.561404+0000","last_fullsized":"2024-04-05T23:32:05.561404+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:18:28.497807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561185+0000","last_change":"2024-04-05T23:32:04.545511+0000","last_active":"2024-04-05T23:32:05.561185+0000","last_peered":"2024-04-05T23:32:05.561185+0000","last_clean":"2024-04-05T23:32:05.561185+0000","last_became_active":"2024-04-05T23:32:04.545191+0000","last_became_peered":"2024-04-05T23:32:04.545191+0000","last_unstale":"2024-04-05T23:32:05.561185+0000","last_undegraded":"2024-04-05T23:32:05.561185+0000","last_fullsized":"2024-04-05T23:32:05.561185+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:17:06.431429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.560809+0000","last_change":"2024-04-05T23:32:04.549673+0000","last_active":"2024-04-05T23:32:05.560809+0000","last_peered":"2024-04-05T23:32:05.560809+0000","last_clean":"2024-04-05T23:32:05.560809+0000","last_became_active":"2024-04-05T23:32:04.549391+0000","last_became_peered":"2024-04-05T23:32:04.549391+0000","last_unstale":"2024-04-05T23:32:05.560809+0000","last_undegraded":"2024-04-05T23:32:05.560809+0000","last_fullsized":"2024-04-05T23:32:05.560809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:30:22.689107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561256+0000","last_change":"2024-04-05T23:32:04.535735+0000","last_active":"2024-04-05T23:32:05.561256+0000","last_peered":"2024-04-05T23:32:05.561256+0000","last_clean":"2024-04-05T23:32:05.561256+0000","last_became_active":"2024-04-05T23:32:04.534461+0000","last_became_peered":"2024-04-05T23:32:04.534461+0000","last_unstale":"2024-04-05T23:32:05.561256+0000","last_undegraded":"2024-04-05T23:32:05.561256+0000","last_fullsized":"2024-04-05T23:32:05.561256+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:37:41.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561517+0000","last_change":"2024-04-05T23:32:04.532340+0000","last_active":"2024-04-05T23:32:05.561517+0000","last_peered":"2024-04-05T23:32:05.561517+0000","last_clean":"2024-04-05T23:32:05.561517+0000","last_became_active":"2024-04-05T23:32:04.530053+0000","last_became_peered":"2024-04-05T23:32:04.530053+0000","last_unstale":"2024-04-05T23:32:05.561517+0000","last_undegraded":"2024-04-05T23:32:05.561517+0000","last_fullsized":"2024-04-05T23:32:05.561517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:16:11.168366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561662+0000","last_change":"2024-04-05T23:32:04.529972+0000","last_active":"2024-04-05T23:32:05.561662+0000","last_peered":"2024-04-05T23:32:05.561662+0000","last_clean":"2024-04-05T23:32:05.561662+0000","last_became_active":"2024-04-05T23:32:04.529551+0000","last_became_peered":"2024-04-05T23:32:04.529551+0000","last_unstale":"2024-04-05T23:32:05.561662+0000","last_undegraded":"2024-04-05T23:32:05.561662+0000","last_fullsized":"2024-04-05T23:32:05.561662+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:13.764971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561382+0000","last_change":"2024-04-05T23:32:04.529166+0000","last_active":"2024-04-05T23:32:05.561382+0000","last_peered":"2024-04-05T23:32:05.561382+0000","last_clean":"2024-04-05T23:32:05.561382+0000","last_became_active":"2024-04-05T23:32:04.528785+0000","last_became_peered":"2024-04-05T23:32:04.528785+0000","last_unstale":"2024-04-05T23:32:05.561382+0000","last_undegraded":"2024-04-05T23:32:05.561382+0000","last_fullsized":"2024-04-05T23:32:05.561382+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:29:43.964265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522551+0000","last_change":"2024-04-05T23:32:04.540631+0000","last_active":"2024-04-05T23:32:05.522551+0000","last_peered":"2024-04-05T23:32:05.522551+0000","last_clean":"2024-04-05T23:32:05.522551+0000","last_became_active":"2024-04-05T23:32:04.540327+0000","last_became_peered":"2024-04-05T23:32:04.540327+0000","last_unstale":"2024-04-05T23:32:05.522551+0000","last_undegraded":"2024-04-05T23:32:05.522551+0000","last_fullsized":"2024-04-05T23:32:05.522551+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:22:48.825324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.346263+0000","last_change":"2024-04-05T23:32:04.544845+0000","last_active":"2024-04-05T23:32:05.346263+0000","last_peered":"2024-04-05T23:32:05.346263+0000","last_clean":"2024-04-05T23:32:05.346263+0000","last_became_active":"2024-04-05T23:32:04.544472+0000","last_became_peered":"2024-04-05T23:32:04.544472+0000","last_unstale":"2024-04-05T23:32:05.346263+0000","last_undegraded":"2024-04-05T23:32:05.346263+0000","last_fullsized":"2024-04-05T23:32:05.346263+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:34:09.529644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.522160+0000","last_change":"2024-04-05T23:32:04.545303+0000","last_active":"2024-04-05T23:32:05.522160+0000","last_peered":"2024-04-05T23:32:05.522160+0000","last_clean":"2024-04-05T23:32:05.522160+0000","last_became_active":"2024-04-05T23:32:04.540192+0000","last_became_peered":"2024-04-05T23:32:04.540192+0000","last_unstale":"2024-04-05T23:32:05.522160+0000","last_undegraded":"2024-04-05T23:32:05.522160+0000","last_fullsized":"2024-04-05T23:32:05.522160+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:49:18.505440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.390540+0000","last_change":"2024-04-05T23:32:04.534631+0000","last_active":"2024-04-05T23:32:05.390540+0000","last_peered":"2024-04-05T23:32:05.390540+0000","last_clean":"2024-04-05T23:32:05.390540+0000","last_became_active":"2024-04-05T23:32:04.534330+0000","last_became_peered":"2024-04-05T23:32:04.534330+0000","last_unstale":"2024-04-05T23:32:05.390540+0000","last_undegraded":"2024-04-05T23:32:05.390540+0000","last_fullsized":"2024-04-05T23:32:05.390540+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:20:47.592174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561028+0000","last_change":"2024-04-05T23:32:04.531459+0000","last_active":"2024-04-05T23:32:05.561028+0000","last_peered":"2024-04-05T23:32:05.561028+0000","last_clean":"2024-04-05T23:32:05.561028+0000","last_became_active":"2024-04-05T23:32:04.530828+0000","last_became_peered":"2024-04-05T23:32:04.530828+0000","last_unstale":"2024-04-05T23:32:05.561028+0000","last_undegraded":"2024-04-05T23:32:05.561028+0000","last_fullsized":"2024-04-05T23:32:05.561028+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:51:34.090030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-05T23:32:05.561489+0000","last_change":"2024-04-05T23:32:04.527711+0000","last_active":"2024-04-05T23:32:05.561489+0000","last_peered":"2024-04-05T23:32:05.561489+0000","last_clean":"2024-04-05T23:32:05.561489+0000","last_became_active":"2024-04-05T23:32:04.527374+0000","last_became_peered":"2024-04-05T23:32:04.527374+0000","last_unstale":"2024-04-05T23:32:05.561489+0000","last_undegraded":"2024-04-05T23:32:05.561489+0000","last_fullsized":"2024-04-05T23:32:05.561489+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:15:10.764340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":26317,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":38,"num_read_kb":34,"num_write":175,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":81428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":412499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]}]},{"osd":10,"up_from":67,"seq":287762808847,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1507328,"data_stored":1012283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52200000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.70499999999999996}]},{"osd":3,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.67400000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.73299999999999998}]},{"osd":5,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.749}]},{"osd":6,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.77100000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.79300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]}]},{"osd":9,"up_from":62,"seq":266287972370,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":412845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.89600000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.72499999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.58699999999999997}]},{"osd":4,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.052}]},{"osd":5,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.82999999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.91900000000000004}]},{"osd":7,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.874},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.004}]},{"osd":8,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.88700000000000001}]},{"osd":10,"last update":"Fri Apr 5 23:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.90500000000000003}]}]},{"osd":8,"up_from":55,"seq":236223201303,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":412687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.59299999999999997}]},{"osd":1,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.495},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.434},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.46200000000000002}]},{"osd":6,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.67100000000000004}]},{"osd":7,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.622},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.69199999999999995}]},{"osd":9,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.68600000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.751},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":7,"up_from":48,"seq":206158430234,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1478656,"data_stored":1005427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.76400000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.35999999999999999}]},{"osd":2,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.56200000000000006}]},{"osd":4,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.34000000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.378},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51700000000000002}]},{"osd":6,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.68100000000000005}]},{"osd":8,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.443},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.69499999999999995}]},{"osd":9,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Apr 5 23:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.622},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":6,"up_from":42,"seq":180388626462,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":412462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47799999999999998,"15min":0.47499999999999998},"min":{"1min":0.34000000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73499999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.433},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.505,"15min":0.5},"min":{"1min":0.36399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.79600000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.70199999999999996}]},{"osd":1,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.45900000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90400000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.627,"15min":0.627},"min":{"1min":0.39500000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92400000000000004,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.41799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85599999999999998,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.89600000000000002}]},{"osd":3,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78500000000000003,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.754},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67400000000000004,"15min":0.66900000000000004},"min":{"1min":0.47299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.92800000000000005,"5min":1.246,"15min":1.246},"last":0.65200000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.50600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.90200000000000002,"5min":1.095,"15min":1.095},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.48599999999999999}]},{"osd":5,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.49099999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.016,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.73899999999999999},"min":{"1min":0.54900000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0509999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.74199999999999999}]},{"osd":7,"last update":"Fri Apr 5 23:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.55400000000000005}]},{"osd":8,"last update":"Fri Apr 5 23:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.79100000000000004}]},{"osd":9,"last update":"Fri Apr 5 23:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.85099999999999998}]},{"osd":10,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.81999999999999995}]}]},{"osd":1,"up_from":12,"seq":51539607601,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":860160,"data_stored":413352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.498},"min":{"1min":0.35099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.65400000000000003,"15min":0.68899999999999995},"min":{"1min":0.32700000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.748,"5min":4.4199999999999999,"15min":4.4199999999999999},"last":0.45900000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52100000000000002,"15min":0.50900000000000001},"min":{"1min":0.32600000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.751},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.52600000000000002,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.85199999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.56299999999999994,"15min":0.53400000000000003},"min":{"1min":0.44700000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.627},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44400000000000001,"15min":0.442},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65700000000000003,"15min":0.65300000000000002},"min":{"1min":0.44900000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.89100000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.68700000000000006,"15min":0.67900000000000005},"min":{"1min":0.5,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65100000000000002,"15min":0.64800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81000000000000005}]},{"osd":9,"last update":"Fri Apr 5 23:31:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.69399999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36080,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709072,"statfs":{"total":95995035648,"available":95958089728,"internally_reserved":0,"allocated":892928,"data_stored":414191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 23:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.52300000000000002,"15min":0.499},"min":{"1min":0.41899999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.51600000000000001,"15min":0.49199999999999999},"min":{"1min":0.41899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Apr 5 23:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.59999999999999998,"15min":0.57799999999999996},"min":{"1min":0.53800000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.68700000000000006,"15min":0.59999999999999998},"min":{"1min":0.49099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1379999999999999,"5min":7.2060000000000004,"15min":7.2060000000000004},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Apr 5 23:30:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.54200000000000004,"15min":0.52800000000000002},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68100000000000005}]},{"osd":4,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.92700000000000005,"15min":0.96099999999999997},"min":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.2929999999999999,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.67400000000000004,"15min":0.65500000000000003},"min":{"1min":0.56100000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.621}]},{"osd":5,"last update":"Fri Apr 5 23:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.67700000000000005,"15min":0.66400000000000003},"min":{"1min":0.51800000000000002,"5min":0.434,"15min":0.434},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.872,"5min":0.72899999999999998,"15min":0.70499999999999996},"min":{"1min":0.60299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.65600000000000003}]},{"osd":6,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.88100000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.153,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.77800000000000002,"15min":0.77500000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.70699999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.77300000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.81599999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:32:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.83099999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]}]},{"osd":2,"up_from":17,"seq":73014444079,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":909312,"data_stored":434513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.624}]},{"osd":1,"last update":"Fri Apr 5 23:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75600000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54100000000000004,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":0.874,"15min":0.874},"last":0.66600000000000004}]},{"osd":3,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55500000000000005,"15min":0.53800000000000003},"min":{"1min":0.37,"5min":0.311,"15min":0.311},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Fri Apr 5 23:31:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.67600000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.54500000000000004},"min":{"1min":0.43099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40999999999999998,"15min":0.40799999999999997},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68200000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.48099999999999998,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78500000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93899999999999995,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73199999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:31:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.438},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.47599999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.746},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.68700000000000006}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]},{"osd":3,"up_from":23,"seq":98784247849,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36048,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709104,"statfs":{"total":95995035648,"available":95958122496,"internally_reserved":0,"allocated":856064,"data_stored":411892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.45100000000000001},"min":{"1min":0.33500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.018,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51000000000000001,"15min":0.46300000000000002},"min":{"1min":0.379,"5min":0.313,"15min":0.313},"max":{"1min":1.0860000000000001,"5min":4.96,"15min":4.96},"last":0.996}]},{"osd":1,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64000000000000001,"15min":0.60599999999999998},"min":{"1min":0.44400000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91200000000000003,"5min":4.5590000000000002,"15min":4.5590000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.61599999999999999,"15min":0.59599999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71599999999999997,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.72799999999999998,"15min":0.67800000000000005},"min":{"1min":0.51400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.028,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71099999999999997,"15min":0.67300000000000004},"min":{"1min":0.57199999999999995,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94399999999999995,"5min":4.6710000000000003,"15min":4.6710000000000003},"last":0.67500000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.69699999999999995,"15min":0.66300000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.8529999999999998,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.69699999999999995,"15min":0.66000000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":4.6289999999999996,"5min":4.6289999999999996,"15min":4.6289999999999996},"last":1.028}]},{"osd":5,"last update":"Fri Apr 5 23:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.73699999999999999,"15min":0.69299999999999995},"min":{"1min":0.57699999999999996,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.76900000000000002,"15min":0.72299999999999998},"min":{"1min":0.61299999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":4.8929999999999998,"5min":4.8929999999999998,"15min":4.8929999999999998},"last":0.73599999999999999}]},{"osd":6,"last update":"Fri Apr 5 23:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.878,"15min":0.88300000000000001},"min":{"1min":0.63400000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.228,"5min":4.8159999999999998,"15min":4.8159999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.78600000000000003,"15min":0.81399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":5.0979999999999999,"15min":5.0979999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.9980000000000002,"5min":4.9980000000000002,"15min":4.9980000000000002},"last":1.2929999999999999},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":4.7290000000000001,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.80500000000000005}]},{"osd":8,"last update":"Fri Apr 5 23:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":4.7569999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.0300000000000002,"5min":5.0300000000000002,"15min":5.0300000000000002},"last":0.72199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.874},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.84299999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":412440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.35799999999999998},"min":{"1min":0.29499999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.49299999999999999,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38100000000000001,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.503,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66600000000000004,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54200000000000004,"15min":0.54100000000000004},"min":{"1min":0.41199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.66700000000000004,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.59599999999999997}]},{"osd":3,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.55800000000000005},"min":{"1min":0.44,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56000000000000005,"15min":0.55500000000000005},"min":{"1min":0.38,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56599999999999995}]},{"osd":5,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.47399999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.502},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.61299999999999999,"15min":0.60099999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.58399999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.61299999999999999,"15min":0.59499999999999997},"min":{"1min":0.435,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.72999999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.436}]},{"osd":8,"last update":"Fri Apr 5 23:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.751}]},{"osd":9,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.71299999999999997}]},{"osd":10,"last update":"Fri Apr 5 23:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.84299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":5,"up_from":35,"seq":150323855393,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1486848,"data_stored":1004619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.38600000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55900000000000005,"15min":0.55500000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.434}]},{"osd":1,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.75,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77100000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63600000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.51100000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.5}]},{"osd":3,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.437,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.42399999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73999999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59199999999999997}]},{"osd":6,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65700000000000003,"15min":0.65900000000000003},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.78900000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77900000000000003,"5min":1.153,"15min":1.153},"last":0.60299999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.624}]},{"osd":8,"last update":"Fri Apr 5 23:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.72499999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.754},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.71299999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":45207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":22537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T23:32:14.902 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-05T23:32:14.902 INFO:tasks.cephadm:Setting up client nodes... 2024-04-05T23:32:14.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-05T23:32:15.298 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:15.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:15 smithi027 ceph-mon[78597]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:15.756 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:15 smithi016 ceph-mon[38248]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:15.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:15 smithi130 ceph-mon[79417]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:16.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.442+0000 7ff630800700 1 -- 172.21.15.16:0/3113885627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628100f10 msgr2=0x7ff628101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:16.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.442+0000 7ff630800700 1 --2- 172.21.15.16:0/3113885627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628100f10 0x7ff628101330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009990 tx=0x7ff61c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.443+0000 7ff630800700 1 -- 172.21.15.16:0/3113885627 shutdown_connections 2024-04-05T23:32:16.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.443+0000 7ff630800700 1 --2- 172.21.15.16:0/3113885627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628100f10 0x7ff628101330 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.443+0000 7ff630800700 1 --2- 172.21.15.16:0/3113885627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628102a30 0x7ff628109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.443+0000 7ff630800700 1 --2- 172.21.15.16:0/3113885627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628102070 0x7ff6281024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.443+0000 7ff630800700 1 -- 172.21.15.16:0/3113885627 >> 172.21.15.16:0/3113885627 conn(0x7ff6280fc4b0 msgr2=0x7ff6280fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.443+0000 7ff630800700 1 -- 172.21.15.16:0/3113885627 shutdown_connections 2024-04-05T23:32:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.443+0000 7ff630800700 1 -- 172.21.15.16:0/3113885627 wait complete. 2024-04-05T23:32:16.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:16 smithi016 ceph-mon[38248]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.444+0000 7ff630800700 1 Processor -- start 2024-04-05T23:32:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.444+0000 7ff630800700 1 -- start start 2024-04-05T23:32:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff630800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100f10 0x7ff62819d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff630800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102070 0x7ff62819d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff630800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628102a30 0x7ff6281a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff630800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6281a4fb0 con 0x7ff628102070 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff630800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6281a50f0 con 0x7ff628100f10 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff630800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6281a5280 con 0x7ff628102a30 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100f10 0x7ff62819d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100f10 0x7ff62819d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:46984/0 (socket says 172.21.15.16:46984) 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102070 0x7ff62819d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62de00700 1 -- 172.21.15.16:0/431648709 learned_addr learned my addr 172.21.15.16:0/431648709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62e800700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628102a30 0x7ff6281a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62de00700 1 -- 172.21.15.16:0/431648709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628102a30 msgr2=0x7ff6281a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62de00700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628102a30 0x7ff6281a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62de00700 1 -- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102070 msgr2=0x7ff62819d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62de00700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102070 0x7ff62819d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62de00700 1 -- 172.21.15.16:0/431648709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c0095f0 con 0x7ff628100f10 2024-04-05T23:32:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.445+0000 7ff62d400700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102070 0x7ff62819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.446+0000 7ff62de00700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100f10 0x7ff62819d470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0059d0 tx=0x7ff61c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.446+0000 7ff622a00700 1 -- 172.21.15.16:0/431648709 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c01d070 con 0x7ff628100f10 2024-04-05T23:32:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.446+0000 7ff622a00700 1 -- 172.21.15.16:0/431648709 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff61c004020 con 0x7ff628100f10 2024-04-05T23:32:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.446+0000 7ff622a00700 1 -- 172.21.15.16:0/431648709 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c022c20 con 0x7ff628100f10 2024-04-05T23:32:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.446+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6281a5500 con 0x7ff628100f10 2024-04-05T23:32:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.446+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6281a5910 con 0x7ff628100f10 2024-04-05T23:32:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.448+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff628106370 con 0x7ff628100f10 2024-04-05T23:32:16.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.449+0000 7ff622a00700 1 -- 172.21.15.16:0/431648709 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7ff61c003b60 con 0x7ff628100f10 2024-04-05T23:32:16.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.450+0000 7ff622a00700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff6100a1230 0x7ff6100a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:16.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.451+0000 7ff622a00700 1 -- 172.21.15.16:0/431648709 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff61c0cdeb0 con 0x7ff628100f10 2024-04-05T23:32:16.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.451+0000 7ff62d400700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff6100a1230 0x7ff6100a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:16.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.451+0000 7ff62d400700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff6100a1230 0x7ff6100a36f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff618010420 tx=0x7ff6180073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:16.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.455+0000 7ff622a00700 1 -- 172.21.15.16:0/431648709 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff61c097620 con 0x7ff628100f10 2024-04-05T23:32:16.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:16 smithi027 ceph-mon[78597]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:16.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:16 smithi027 ceph-mon[78597]: pgmap v238: 97 pgs: 97 active+clean; 603 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8.5 KiB/s wr, 26 op/s 2024-04-05T23:32:16.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.753+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ff62810c950 con 0x7ff628100f10 2024-04-05T23:32:16.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.759+0000 7ff622a00700 1 -- 172.21.15.16:0/431648709 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7ff61c09ac40 con 0x7ff628100f10 2024-04-05T23:32:16.751 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-04-05T23:32:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAAihBm2FgALRAASdrv5hT0DyX3heZ32mT3hQ== 2024-04-05T23:32:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff6100a1230 msgr2=0x7ff6100a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff6100a1230 0x7ff6100a36f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff618010420 tx=0x7ff6180073d0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100f10 msgr2=0x7ff62819d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100f10 0x7ff62819d470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0059d0 tx=0x7ff61c005740 comp rx=0 tx=0).stop 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 shutdown_connections 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff6100a1230 0x7ff6100a36f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628102a30 0x7ff6281a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100f10 0x7ff62819d470 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 --2- 172.21.15.16:0/431648709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102070 0x7ff62819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 >> 172.21.15.16:0/431648709 conn(0x7ff6280fc4b0 msgr2=0x7ff6281034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 shutdown_connections 2024-04-05T23:32:16.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:16.762+0000 7ff630800700 1 -- 172.21.15.16:0/431648709 wait complete. 2024-04-05T23:32:16.776 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:16 smithi016 ceph-mon[38248]: pgmap v238: 97 pgs: 97 active+clean; 603 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8.5 KiB/s wr, 26 op/s 2024-04-05T23:32:16.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:16 smithi130 ceph-mon[79417]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:16.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:16 smithi130 ceph-mon[79417]: pgmap v238: 97 pgs: 97 active+clean; 603 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8.5 KiB/s wr, 26 op/s 2024-04-05T23:32:17.124 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:32:17.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-05T23:32:17.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-05T23:32:17.221 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-05T23:32:17.221 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-05T23:32:17.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-05T23:32:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:17.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:17 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/431648709' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T23:32:17.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:17 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T23:32:17.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:17 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T23:32:17.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:17 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/431648709' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T23:32:17.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:17 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T23:32:17.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:17 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T23:32:17.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:17 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/431648709' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T23:32:17.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:17 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T23:32:17.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:17 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T23:32:18.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:18 smithi027 ceph-mon[78597]: pgmap v239: 97 pgs: 97 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.1 KiB/s wr, 26 op/s 2024-04-05T23:32:18.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.749+0000 7f5024000700 1 -- 172.21.15.16:0/145206973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 msgr2=0x7f501c10ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:18.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.749+0000 7f5024000700 1 --2- 172.21.15.16:0/145206973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 0x7f501c10ddb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f501400b0a0 tx=0x7f501400b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:18.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.749+0000 7f5024000700 1 -- 172.21.15.16:0/145206973 shutdown_connections 2024-04-05T23:32:18.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.749+0000 7f5024000700 1 --2- 172.21.15.16:0/145206973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 0x7f501c10ddb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:18.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.749+0000 7f5024000700 1 --2- 172.21.15.16:0/145206973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f501c106f70 0x7f501c107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:18.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.749+0000 7f5024000700 1 --2- 172.21.15.16:0/145206973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f501c100f30 0x7f501c101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:18.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.749+0000 7f5024000700 1 -- 172.21.15.16:0/145206973 >> 172.21.15.16:0/145206973 conn(0x7f501c0780d0 msgr2=0x7f501c0784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:18.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.750+0000 7f5024000700 1 -- 172.21.15.16:0/145206973 shutdown_connections 2024-04-05T23:32:18.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.750+0000 7f5024000700 1 -- 172.21.15.16:0/145206973 wait complete. 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.751+0000 7f5024000700 1 Processor -- start 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.751+0000 7f5024000700 1 -- start start 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.751+0000 7f5024000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f501c100f30 0x7f501c199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.751+0000 7f5024000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 0x7f501c199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.751+0000 7f5024000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f501c106f70 0x7f501c19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.751+0000 7f5024000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c19e570 con 0x7f501c1018d0 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.751+0000 7f5024000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c19e6e0 con 0x7f501c106f70 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.751+0000 7f5024000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c19e8a0 con 0x7f501c100f30 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.751+0000 7f5020c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 0x7f501c199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5020c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 0x7f501c199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48544/0 (socket says 172.21.15.16:48544) 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5020c00700 1 -- 172.21.15.16:0/1805183092 learned_addr learned my addr 172.21.15.16:0/1805183092 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5021600700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f501c100f30 0x7f501c199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5022000700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f501c106f70 0x7f501c19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5020c00700 1 -- 172.21.15.16:0/1805183092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f501c100f30 msgr2=0x7f501c199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5020c00700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f501c100f30 0x7f501c199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5020c00700 1 -- 172.21.15.16:0/1805183092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f501c106f70 msgr2=0x7f501c19df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5020c00700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f501c106f70 0x7f501c19df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5020c00700 1 -- 172.21.15.16:0/1805183092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5014009d00 con 0x7f501c1018d0 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5020c00700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 0x7f501c199bb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f500800d980 tx=0x7f500800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f501a000700 1 -- 172.21.15.16:0/1805183092 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008004220 con 0x7f501c1018d0 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f501a000700 1 -- 172.21.15.16:0/1805183092 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5008007640 con 0x7f501c1018d0 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501c19eb80 con 0x7f501c1018d0 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.752+0000 7f501a000700 1 -- 172.21.15.16:0/1805183092 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008005690 con 0x7f501c1018d0 2024-04-05T23:32:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.753+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501c1a63e0 con 0x7f501c1018d0 2024-04-05T23:32:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.754+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501c102620 con 0x7f501c1018d0 2024-04-05T23:32:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.755+0000 7f501a000700 1 -- 172.21.15.16:0/1805183092 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f5008004750 con 0x7f501c1018d0 2024-04-05T23:32:18.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.757+0000 7f501a000700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f50040a11b0 0x7f50040a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.757+0000 7f501a000700 1 -- 172.21.15.16:0/1805183092 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f50080cc730 con 0x7f501c1018d0 2024-04-05T23:32:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.757+0000 7f5021600700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f50040a11b0 0x7f50040a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.758+0000 7f5021600700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f50040a11b0 0x7f50040a3670 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5010009be0 tx=0x7f5010009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:18.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:18.761+0000 7f501a000700 1 -- 172.21.15.16:0/1805183092 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5008095e10 con 0x7f501c1018d0 2024-04-05T23:32:18.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:18 smithi016 ceph-mon[38248]: pgmap v239: 97 pgs: 97 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.1 KiB/s wr, 26 op/s 2024-04-05T23:32:18.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:18 smithi130 ceph-mon[79417]: pgmap v239: 97 pgs: 97 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.1 KiB/s wr, 26 op/s 2024-04-05T23:32:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.032+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f501c04ebb0 con 0x7f501c1018d0 2024-04-05T23:32:19.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.038+0000 7f501a000700 1 -- 172.21.15.16:0/1805183092 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260285 (secure 0 0 0) 0x7f5008016090 con 0x7f501c1018d0 2024-04-05T23:32:19.033 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f50040a11b0 msgr2=0x7f50040a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f50040a11b0 0x7f50040a3670 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5010009be0 tx=0x7f5010009260 comp rx=0 tx=0).stop 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 msgr2=0x7f501c199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 0x7f501c199bb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f500800d980 tx=0x7f500800dc90 comp rx=0 tx=0).stop 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 shutdown_connections 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f50040a11b0 0x7f50040a3670 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f501c106f70 0x7f501c19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f501c100f30 0x7f501c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 --2- 172.21.15.16:0/1805183092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501c1018d0 0x7f501c199bb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.042+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 >> 172.21.15.16:0/1805183092 conn(0x7f501c0780d0 msgr2=0x7f501c0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.043+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 shutdown_connections 2024-04-05T23:32:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:19.043+0000 7f5024000700 1 -- 172.21.15.16:0/1805183092 wait complete. 2024-04-05T23:32:19.431 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1565223355},{"type":"v1","addr":"172.21.15.16:6801","nonce":1565223355}]},"active_addr":"172.21.15.16:6801/1565223355","active_change":"2024-04-05T23:26:37.227432+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24107,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4184690359}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":985211078}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1740984099}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4157043736}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2846240830}]}]} 2024-04-05T23:32:19.436 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-05T23:32:19.436 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-05T23:32:19.436 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-05T23:32:19.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:19 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1805183092' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-05T23:32:19.805 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:19.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:19 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1805183092' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-05T23:32:19.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:19 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1805183092' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-05T23:32:20.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:20 smithi016 ceph-mon[38248]: pgmap v240: 97 pgs: 97 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.7 KiB/s wr, 25 op/s 2024-04-05T23:32:20.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:20 smithi027 ceph-mon[78597]: pgmap v240: 97 pgs: 97 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.7 KiB/s wr, 25 op/s 2024-04-05T23:32:20.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:20 smithi130 ceph-mon[79417]: pgmap v240: 97 pgs: 97 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.7 KiB/s wr, 25 op/s 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.880+0000 7f9792e00700 1 -- 172.21.15.16:0/2335285748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 msgr2=0x7f978c10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.880+0000 7f9792e00700 1 --2- 172.21.15.16:0/2335285748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 0x7f978c10d480 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f97780099c0 tx=0x7f9778009cd0 comp rx=0 tx=0).stop 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.880+0000 7f9792e00700 1 -- 172.21.15.16:0/2335285748 shutdown_connections 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.880+0000 7f9792e00700 1 --2- 172.21.15.16:0/2335285748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f978c10d9c0 0x7f978c10fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.880+0000 7f9792e00700 1 --2- 172.21.15.16:0/2335285748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c100910 0x7f978c100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.880+0000 7f9792e00700 1 --2- 172.21.15.16:0/2335285748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 0x7f978c10d480 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.880+0000 7f9792e00700 1 -- 172.21.15.16:0/2335285748 >> 172.21.15.16:0/2335285748 conn(0x7f978c075ec0 msgr2=0x7f978c0762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.880+0000 7f9792e00700 1 -- 172.21.15.16:0/2335285748 shutdown_connections 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.880+0000 7f9792e00700 1 -- 172.21.15.16:0/2335285748 wait complete. 2024-04-05T23:32:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.881+0000 7f9792e00700 1 Processor -- start 2024-04-05T23:32:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.881+0000 7f9792e00700 1 -- start start 2024-04-05T23:32:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f9792e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f978c100910 0x7f978c199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f9792e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 0x7f978c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f9792e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c10d9c0 0x7f978c19de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f9792e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c19e440 con 0x7f978c101230 2024-04-05T23:32:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f9792e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c19e5b0 con 0x7f978c10d9c0 2024-04-05T23:32:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f9792e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c19e770 con 0x7f978c100910 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f978b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 0x7f978c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f978b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 0x7f978c199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48560/0 (socket says 172.21.15.16:48560) 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f978b400700 1 -- 172.21.15.16:0/1601533004 learned_addr learned my addr 172.21.15.16:0/1601533004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f978be00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f978c100910 0x7f978c199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f9790e00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c10d9c0 0x7f978c19de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f978b400700 1 -- 172.21.15.16:0/1601533004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f978c100910 msgr2=0x7f978c199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f978b400700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f978c100910 0x7f978c199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f978b400700 1 -- 172.21.15.16:0/1601533004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c10d9c0 msgr2=0x7f978c19de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f978b400700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c10d9c0 0x7f978c19de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f978b400700 1 -- 172.21.15.16:0/1601533004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97780095f0 con 0x7f978c101230 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.882+0000 7f9790e00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c10d9c0 0x7f978c19de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.883+0000 7f978b400700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 0x7f978c199ad0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9778005b40 tx=0x7f977800ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.883+0000 7f9788c00700 1 -- 172.21.15.16:0/1601533004 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977801d070 con 0x7f978c101230 2024-04-05T23:32:20.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.883+0000 7f9788c00700 1 -- 172.21.15.16:0/1601533004 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f977800be70 con 0x7f978c101230 2024-04-05T23:32:20.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.883+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c19e9f0 con 0x7f978c101230 2024-04-05T23:32:20.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.883+0000 7f9788c00700 1 -- 172.21.15.16:0/1601533004 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977800fb70 con 0x7f978c101230 2024-04-05T23:32:20.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.883+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c1a62b0 con 0x7f978c101230 2024-04-05T23:32:20.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.884+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c108be0 con 0x7f978c101230 2024-04-05T23:32:20.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.886+0000 7f9788c00700 1 -- 172.21.15.16:0/1601533004 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f977802b440 con 0x7f978c101230 2024-04-05T23:32:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.887+0000 7f9788c00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f97700a1230 0x7f97700a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.887+0000 7f9788c00700 1 -- 172.21.15.16:0/1601533004 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f97780cd600 con 0x7f978c101230 2024-04-05T23:32:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.887+0000 7f978be00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f97700a1230 0x7f97700a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:20.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.888+0000 7f978be00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f97700a1230 0x7f97700a36f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f977c009c60 tx=0x7f977c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:20.891+0000 7f9788c00700 1 -- 172.21.15.16:0/1601533004 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9778096d70 con 0x7f978c101230 2024-04-05T23:32:21.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.106+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f978c100420 con 0x7f978c101230 2024-04-05T23:32:21.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.107+0000 7f9788c00700 1 -- 172.21.15.16:0/1601533004 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+22084 (secure 0 0 0) 0x7f977809a390 con 0x7f978c101230 2024-04-05T23:32:21.100 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:21.100 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":79,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:05.514031+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f97700a1230 msgr2=0x7f97700a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f97700a1230 0x7f97700a36f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f977c009c60 tx=0x7f977c009260 comp rx=0 tx=0).stop 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 msgr2=0x7f978c199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 0x7f978c199ad0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9778005b40 tx=0x7f977800ba30 comp rx=0 tx=0).stop 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 shutdown_connections 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f97700a1230 0x7f97700a36f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c10d9c0 0x7f978c19de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f978c100910 0x7f978c199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 --2- 172.21.15.16:0/1601533004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f978c101230 0x7f978c199ad0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 >> 172.21.15.16:0/1601533004 conn(0x7f978c075ec0 msgr2=0x7f978c10f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 shutdown_connections 2024-04-05T23:32:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:21.110+0000 7f9792e00700 1 -- 172.21.15.16:0/1601533004 wait complete. 2024-04-05T23:32:21.498 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-05T23:32:21.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-05T23:32:21.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:21 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1601533004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:21.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:21 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1601533004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:21.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:21 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1601533004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:21.865 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:22.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:22 smithi027 ceph-mon[78597]: pgmap v241: 97 pgs: 97 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.1 KiB/s wr, 26 op/s 2024-04-05T23:32:22.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:22 smithi016 ceph-mon[38248]: pgmap v241: 97 pgs: 97 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.1 KiB/s wr, 26 op/s 2024-04-05T23:32:22.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:22 smithi130 ceph-mon[79417]: pgmap v241: 97 pgs: 97 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.1 KiB/s wr, 26 op/s 2024-04-05T23:32:23.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.013+0000 7f0853e00700 1 -- 172.21.15.16:0/1837226311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f085410d530 msgr2=0x7f085410f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:23.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.013+0000 7f0853e00700 1 --2- 172.21.15.16:0/1837226311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f085410d530 0x7f085410f9a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f084c009f90 tx=0x7f084c00b330 comp rx=0 tx=0).stop 2024-04-05T23:32:23.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.013+0000 7f0853e00700 1 -- 172.21.15.16:0/1837226311 shutdown_connections 2024-04-05T23:32:23.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.013+0000 7f0853e00700 1 --2- 172.21.15.16:0/1837226311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f085410d530 0x7f085410f9a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.013+0000 7f0853e00700 1 --2- 172.21.15.16:0/1837226311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0854073310 0x7f08540736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.013+0000 7f0853e00700 1 --2- 172.21.15.16:0/1837226311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854073c30 0x7f085410cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.013+0000 7f0853e00700 1 -- 172.21.15.16:0/1837226311 >> 172.21.15.16:0/1837226311 conn(0x7f08540fc840 msgr2=0x7f08540fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:23.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.014+0000 7f0853e00700 1 -- 172.21.15.16:0/1837226311 shutdown_connections 2024-04-05T23:32:23.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.014+0000 7f0853e00700 1 -- 172.21.15.16:0/1837226311 wait complete. 2024-04-05T23:32:23.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0853e00700 1 Processor -- start 2024-04-05T23:32:23.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0853e00700 1 -- start start 2024-04-05T23:32:23.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0853e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0854073310 0x7f0854199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:23.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0853e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854073c30 0x7f0854199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:23.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0853e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f085410d530 0x7f085419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:23.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0853e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085419e330 con 0x7f0854073310 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0853e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085419e4a0 con 0x7f085410d530 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0853e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085419e660 con 0x7f0854073c30 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0852000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854073c30 0x7f0854199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0852a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0854073310 0x7f0854199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0852000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854073c30 0x7f0854199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43096/0 (socket says 172.21.15.16:43096) 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.015+0000 7f0852000700 1 -- 172.21.15.16:0/2865317616 learned_addr learned my addr 172.21.15.16:0/2865317616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f0853400700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f085410d530 0x7f085419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f0852000700 1 -- 172.21.15.16:0/2865317616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f085410d530 msgr2=0x7f085419dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f0852000700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f085410d530 0x7f085419dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f0852000700 1 -- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0854073310 msgr2=0x7f0854199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f0852000700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0854073310 0x7f0854199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f0852000700 1 -- 172.21.15.16:0/2865317616 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c009c40 con 0x7f0854073c30 2024-04-05T23:32:23.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f0852a00700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0854073310 0x7f0854199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:23.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f0852000700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854073c30 0x7f0854199970 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f084000e9c0 tx=0x7f084000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:23.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f084b400700 1 -- 172.21.15.16:0/2865317616 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840004a30 con 0x7f0854073c30 2024-04-05T23:32:23.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f084b400700 1 -- 172.21.15.16:0/2865317616 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f084000cc80 con 0x7f0854073c30 2024-04-05T23:32:23.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085419e940 con 0x7f0854073c30 2024-04-05T23:32:23.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.016+0000 7f084b400700 1 -- 172.21.15.16:0/2865317616 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08400054e0 con 0x7f0854073c30 2024-04-05T23:32:23.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.017+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08541a61a0 con 0x7f0854073c30 2024-04-05T23:32:23.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.018+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0854108740 con 0x7f0854073c30 2024-04-05T23:32:23.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.020+0000 7f084b400700 1 -- 172.21.15.16:0/2865317616 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f0840004b90 con 0x7f0854073c30 2024-04-05T23:32:23.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.021+0000 7f084b400700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f08380a11b0 0x7f08380a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:23.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.021+0000 7f084b400700 1 -- 172.21.15.16:0/2865317616 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0840014070 con 0x7f0854073c30 2024-04-05T23:32:23.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.021+0000 7f0852a00700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f08380a11b0 0x7f08380a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:23.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.022+0000 7f0852a00700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f08380a11b0 0x7f08380a3670 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0844011490 tx=0x7f0844009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:23.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.023+0000 7f084b400700 1 -- 172.21.15.16:0/2865317616 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f08400964b0 con 0x7f0854073c30 2024-04-05T23:32:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.238+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08540749c0 con 0x7f0854073c30 2024-04-05T23:32:23.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.239+0000 7f084b400700 1 -- 172.21.15.16:0/2865317616 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+22084 (secure 0 0 0) 0x7f0840099ad0 con 0x7f0854073c30 2024-04-05T23:32:23.232 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:23.232 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":79,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:05.514031+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:32:23.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f08380a11b0 msgr2=0x7f08380a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:23.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f08380a11b0 0x7f08380a3670 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0844011490 tx=0x7f0844009260 comp rx=0 tx=0).stop 2024-04-05T23:32:23.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854073c30 msgr2=0x7f0854199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:23.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854073c30 0x7f0854199970 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f084000e9c0 tx=0x7f084000ecd0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 shutdown_connections 2024-04-05T23:32:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f08380a11b0 0x7f08380a3670 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0854073310 0x7f0854199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f085410d530 0x7f085419dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 --2- 172.21.15.16:0/2865317616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854073c30 0x7f0854199970 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 >> 172.21.15.16:0/2865317616 conn(0x7f08540fc840 msgr2=0x7f08540fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 shutdown_connections 2024-04-05T23:32:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:23.242+0000 7f0853e00700 1 -- 172.21.15.16:0/2865317616 wait complete. 2024-04-05T23:32:23.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:23 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2865317616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:23.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-05T23:32:23.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-05T23:32:23.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-05T23:32:23.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-05T23:32:23.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-05T23:32:23.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-05T23:32:23.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-05T23:32:23.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-05T23:32:23.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.8 flush_pg_stats 2024-04-05T23:32:23.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.9 flush_pg_stats 2024-04-05T23:32:23.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.10 flush_pg_stats 2024-04-05T23:32:23.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph tell osd.11 flush_pg_stats 2024-04-05T23:32:23.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:23 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2865317616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:23.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:23 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2865317616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:24.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:24 smithi027 ceph-mon[78597]: pgmap v242: 97 pgs: 97 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 22 op/s 2024-04-05T23:32:24.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:24 smithi016 ceph-mon[38248]: pgmap v242: 97 pgs: 97 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 22 op/s 2024-04-05T23:32:24.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:24 smithi130 ceph-mon[79417]: pgmap v242: 97 pgs: 97 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 22 op/s 2024-04-05T23:32:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:25.507 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:25.559 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:25.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:25 smithi016 ceph-mon[38248]: pgmap v243: 97 pgs: 97 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 3.1 KiB/s wr, 14 op/s 2024-04-05T23:32:25.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:25 smithi130 ceph-mon[79417]: pgmap v243: 97 pgs: 97 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 3.1 KiB/s wr, 14 op/s 2024-04-05T23:32:25.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:25 smithi027 ceph-mon[78597]: pgmap v243: 97 pgs: 97 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 3.1 KiB/s wr, 14 op/s 2024-04-05T23:32:26.367 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:26.369 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:26.830 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:26.830 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:27.259 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.571+0000 7f24d0400700 1 -- 172.21.15.16:0/195012846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8100f30 msgr2=0x7f24c8101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.571+0000 7f24d0400700 1 --2- 172.21.15.16:0/195012846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8100f30 0x7f24c8101390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009990 tx=0x7f24b4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.571+0000 7f24d0400700 1 -- 172.21.15.16:0/195012846 shutdown_connections 2024-04-05T23:32:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.571+0000 7f24d0400700 1 --2- 172.21.15.16:0/195012846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c81018d0 0x7f24c810ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.571+0000 7f24d0400700 1 --2- 172.21.15.16:0/195012846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c8106f70 0x7f24c8107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.571+0000 7f24d0400700 1 --2- 172.21.15.16:0/195012846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8100f30 0x7f24c8101390 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.571+0000 7f24d0400700 1 -- 172.21.15.16:0/195012846 >> 172.21.15.16:0/195012846 conn(0x7f24c80780d0 msgr2=0x7f24c80784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.571+0000 7f24d0400700 1 -- 172.21.15.16:0/195012846 shutdown_connections 2024-04-05T23:32:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.572+0000 7f24d0400700 1 -- 172.21.15.16:0/195012846 wait complete. 2024-04-05T23:32:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.572+0000 7f24d0400700 1 Processor -- start 2024-04-05T23:32:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24d0400700 1 -- start start 2024-04-05T23:32:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24d0400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c8100f30 0x7f24c8197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24d0400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c81018d0 0x7f24c8197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24d0400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8106f70 0x7f24c819bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24d0400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c819c200 con 0x7f24c8106f70 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24d0400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c819c370 con 0x7f24c81018d0 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24d0400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c819c530 con 0x7f24c8100f30 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24cd000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c81018d0 0x7f24c8197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24ce400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8106f70 0x7f24c819bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24cda00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c8100f30 0x7f24c8197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24cd000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c81018d0 0x7f24c8197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:46848/0 (socket says 172.21.15.16:46848) 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.573+0000 7f24cd000700 1 -- 172.21.15.16:0/3630401769 learned_addr learned my addr 172.21.15.16:0/3630401769 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24ce400700 1 -- 172.21.15.16:0/3630401769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c8100f30 msgr2=0x7f24c8197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24ce400700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c8100f30 0x7f24c8197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24ce400700 1 -- 172.21.15.16:0/3630401769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c81018d0 msgr2=0x7f24c8197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24ce400700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c81018d0 0x7f24c8197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24ce400700 1 -- 172.21.15.16:0/3630401769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b40095f0 con 0x7f24c8106f70 2024-04-05T23:32:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24cd000700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c81018d0 0x7f24c8197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24cda00700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c8100f30 0x7f24c8197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24ce400700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8106f70 0x7f24c819bbf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f24c000efc0 tx=0x7f24c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24c6000700 1 -- 172.21.15.16:0/3630401769 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0018070 con 0x7f24c8106f70 2024-04-05T23:32:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c819c7b0 con 0x7f24c8106f70 2024-04-05T23:32:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24c6000700 1 -- 172.21.15.16:0/3630401769 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24c000f040 con 0x7f24c8106f70 2024-04-05T23:32:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24c6000700 1 -- 172.21.15.16:0/3630401769 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c000cc80 con 0x7f24c8106f70 2024-04-05T23:32:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.574+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c81a4070 con 0x7f24c8106f70 2024-04-05T23:32:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.575+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f24c8102620 con 0x7f24c8106f70 2024-04-05T23:32:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.578+0000 7f24c6000700 1 -- 172.21.15.16:0/3630401769 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f24c0003a00 con 0x7f24c8106f70 2024-04-05T23:32:27.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.580+0000 7f24c6000700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f24b00a11b0 0x7f24b00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:27.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.580+0000 7f24c6000700 1 -- 172.21.15.16:0/3630401769 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f24c00cce20 con 0x7f24c8106f70 2024-04-05T23:32:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.580+0000 7f24cda00700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f24b00a11b0 0x7f24b00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.580+0000 7f24c6000700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] conn(0x7f24b00aa990 0x7f24b00acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.580+0000 7f24c6000700 1 -- 172.21.15.16:0/3630401769 --> [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b00ad460 con 0x7f24b00aa990 2024-04-05T23:32:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.581+0000 7f24c6000700 1 -- 172.21.15.16:0/3630401769 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24c00cd190 con 0x7f24c8106f70 2024-04-05T23:32:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.581+0000 7f24cda00700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f24b00a11b0 0x7f24b00a3670 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f24bc007f50 tx=0x7f24bc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.581+0000 7f24cd000700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] conn(0x7f24b00aa990 0x7f24b00acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.582+0000 7f24cd000700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] conn(0x7f24b00aa990 0x7f24b00acdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:27.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.583+0000 7f24c6000700 1 -- 172.21.15.16:0/3630401769 <== osd.2 v2:172.21.15.130:6800/3222862878 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f24b00ad460 con 0x7f24b00aa990 2024-04-05T23:32:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.600+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 --> [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f24c8111070 con 0x7f24b00aa990 2024-04-05T23:32:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.601+0000 7f24c6000700 1 -- 172.21.15.16:0/3630401769 <== osd.2 v2:172.21.15.130:6800/3222862878 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f24c8111070 con 0x7f24b00aa990 2024-04-05T23:32:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.601+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 >> [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] conn(0x7f24b00aa990 msgr2=0x7f24b00acdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.601+0000 7f24d0400700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] conn(0x7f24b00aa990 0x7f24b00acdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.601+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f24b00a11b0 msgr2=0x7f24b00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.601+0000 7f24d0400700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f24b00a11b0 0x7f24b00a3670 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f24bc007f50 tx=0x7f24bc0073d0 comp rx=0 tx=0).stop 2024-04-05T23:32:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.601+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8106f70 msgr2=0x7f24c819bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.601+0000 7f24d0400700 1 --2- 172.21.15.16:0/3630401769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8106f70 0x7f24c819bbf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f24c000efc0 tx=0x7f24c000c3a0 comp rx=0 tx=0).stop 2024-04-05T23:32:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.601+0000 7f24ce400700 1 -- 172.21.15.16:0/3630401769 reap_dead start 2024-04-05T23:32:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.602+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 shutdown_connections 2024-04-05T23:32:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.602+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 >> 172.21.15.16:0/3630401769 conn(0x7f24c80780d0 msgr2=0x7f24c80ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.602+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 shutdown_connections 2024-04-05T23:32:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:27.602+0000 7f24d0400700 1 -- 172.21.15.16:0/3630401769 wait complete. 2024-04-05T23:32:28.089 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:28.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.294+0000 7fe534c00700 1 -- 172.21.15.16:0/2956229158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5301018d0 msgr2=0x7fe53010ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:28.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.294+0000 7fe534c00700 1 --2- 172.21.15.16:0/2956229158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5301018d0 0x7fe53010ddb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe528009f90 tx=0x7fe52800b330 comp rx=0 tx=0).stop 2024-04-05T23:32:28.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.294+0000 7fe534c00700 1 -- 172.21.15.16:0/2956229158 shutdown_connections 2024-04-05T23:32:28.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.294+0000 7fe534c00700 1 --2- 172.21.15.16:0/2956229158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5301018d0 0x7fe53010ddb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:28.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.294+0000 7fe534c00700 1 --2- 172.21.15.16:0/2956229158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe530106f70 0x7fe530107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:28.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.294+0000 7fe534c00700 1 --2- 172.21.15.16:0/2956229158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530100f30 0x7fe530101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:28.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.294+0000 7fe534c00700 1 -- 172.21.15.16:0/2956229158 >> 172.21.15.16:0/2956229158 conn(0x7fe5300780d0 msgr2=0x7fe5300784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:28.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.294+0000 7fe534c00700 1 -- 172.21.15.16:0/2956229158 shutdown_connections 2024-04-05T23:32:28.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.295+0000 7fe534c00700 1 -- 172.21.15.16:0/2956229158 wait complete. 2024-04-05T23:32:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.295+0000 7fe534c00700 1 Processor -- start 2024-04-05T23:32:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe534c00700 1 -- start start 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe534c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe530100f30 0x7fe5301996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe534c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301018d0 0x7fe530199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe534c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe530106f70 0x7fe53019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe534c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53019e5a0 con 0x7fe5301018d0 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe534c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53019e710 con 0x7fe530100f30 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe534c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53019e8d0 con 0x7fe530106f70 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe52de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe530100f30 0x7fe5301996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe52d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301018d0 0x7fe530199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe52de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe530100f30 0x7fe5301996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:46868/0 (socket says 172.21.15.16:46868) 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.296+0000 7fe52de00700 1 -- 172.21.15.16:0/3939488488 learned_addr learned my addr 172.21.15.16:0/3939488488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe52e800700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe530106f70 0x7fe53019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe52d400700 1 -- 172.21.15.16:0/3939488488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe530106f70 msgr2=0x7fe53019df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe52d400700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe530106f70 0x7fe53019df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe52d400700 1 -- 172.21.15.16:0/3939488488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe530100f30 msgr2=0x7fe5301996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe52d400700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe530100f30 0x7fe5301996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe52d400700 1 -- 172.21.15.16:0/3939488488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c0095f0 con 0x7fe5301018d0 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe52d400700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301018d0 0x7fe530199be0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00eb90 tx=0x7fe51c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe526a00700 1 -- 172.21.15.16:0/3939488488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c009970 con 0x7fe5301018d0 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe528009c40 con 0x7fe5301018d0 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe526a00700 1 -- 172.21.15.16:0/3939488488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe51c0044e0 con 0x7fe5301018d0 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.297+0000 7fe526a00700 1 -- 172.21.15.16:0/3939488488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c003e50 con 0x7fe5301018d0 2024-04-05T23:32:28.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.298+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5301a6240 con 0x7fe5301018d0 2024-04-05T23:32:28.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.298+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe5301110f0 con 0x7fe5301018d0 2024-04-05T23:32:28.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.300+0000 7fe526a00700 1 -- 172.21.15.16:0/3939488488 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fe51c003fb0 con 0x7fe5301018d0 2024-04-05T23:32:28.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.302+0000 7fe526a00700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe5140a1230 0x7fe5140a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.302+0000 7fe526a00700 1 -- 172.21.15.16:0/3939488488 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe51c014070 con 0x7fe5301018d0 2024-04-05T23:32:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.302+0000 7fe52de00700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe5140a1230 0x7fe5140a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.302+0000 7fe526a00700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] conn(0x7fe5140aaa00 0x7fe5140ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:28.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.302+0000 7fe526a00700 1 -- 172.21.15.16:0/3939488488 --> [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5140ad4d0 con 0x7fe5140aaa00 2024-04-05T23:32:28.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.302+0000 7fe526a00700 1 -- 172.21.15.16:0/3939488488 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe51c0cd280 con 0x7fe5301018d0 2024-04-05T23:32:28.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.302+0000 7fe52e800700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] conn(0x7fe5140aaa00 0x7fe5140ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:28.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.302+0000 7fe52de00700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe5140a1230 0x7fe5140a36f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe520009ca0 tx=0x7fe520009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:28.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.303+0000 7fe52e800700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] conn(0x7fe5140aaa00 0x7fe5140ace20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:28.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.304+0000 7fe526a00700 1 -- 172.21.15.16:0/3939488488 <== osd.6 v2:172.21.15.16:6818/2729842886 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe5140ad4d0 con 0x7fe5140aaa00 2024-04-05T23:32:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.320+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 --> [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe530061560 con 0x7fe5140aaa00 2024-04-05T23:32:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe526a00700 1 -- 172.21.15.16:0/3939488488 <== osd.6 v2:172.21.15.16:6818/2729842886 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe530061560 con 0x7fe5140aaa00 2024-04-05T23:32:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] conn(0x7fe5140aaa00 msgr2=0x7fe5140ace20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe534c00700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] conn(0x7fe5140aaa00 0x7fe5140ace20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe5140a1230 msgr2=0x7fe5140a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe534c00700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe5140a1230 0x7fe5140a36f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe520009ca0 tx=0x7fe520009260 comp rx=0 tx=0).stop 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301018d0 msgr2=0x7fe530199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe534c00700 1 --2- 172.21.15.16:0/3939488488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301018d0 0x7fe530199be0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00eb90 tx=0x7fe51c00eea0 comp rx=0 tx=0).stop 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe52e800700 1 -- 172.21.15.16:0/3939488488 reap_dead start 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 shutdown_connections 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 >> 172.21.15.16:0/3939488488 conn(0x7fe5300780d0 msgr2=0x7fe53010c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.321+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 shutdown_connections 2024-04-05T23:32:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:28.322+0000 7fe534c00700 1 -- 172.21.15.16:0/3939488488 wait complete. 2024-04-05T23:32:28.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:28 smithi016 ceph-mon[38248]: pgmap v244: 97 pgs: 97 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.4 KiB/s wr, 13 op/s 2024-04-05T23:32:28.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:28 smithi027 ceph-mon[78597]: pgmap v244: 97 pgs: 97 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.4 KiB/s wr, 13 op/s 2024-04-05T23:32:28.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:28 smithi130 ceph-mon[79417]: pgmap v244: 97 pgs: 97 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.4 KiB/s wr, 13 op/s 2024-04-05T23:32:29.766 INFO:teuthology.orchestra.run.smithi016.stdout:73014444082 2024-04-05T23:32:29.766 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-05T23:32:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.892+0000 7f39cc400700 1 -- 172.21.15.16:0/1263408967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c4073650 msgr2=0x7f39c4073ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.892+0000 7f39cc400700 1 --2- 172.21.15.16:0/1263408967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c4073650 0x7f39c4073ad0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009990 tx=0x7f39b0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.892+0000 7f39cc400700 1 -- 172.21.15.16:0/1263408967 shutdown_connections 2024-04-05T23:32:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.892+0000 7f39cc400700 1 --2- 172.21.15.16:0/1263408967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c4111b70 0x7f39c4113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.892+0000 7f39cc400700 1 --2- 172.21.15.16:0/1263408967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c4074f40 0x7f39c4073110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.892+0000 7f39cc400700 1 --2- 172.21.15.16:0/1263408967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c4073650 0x7f39c4073ad0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:29.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.892+0000 7f39cc400700 1 -- 172.21.15.16:0/1263408967 >> 172.21.15.16:0/1263408967 conn(0x7f39c40780d0 msgr2=0x7f39c40784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.892+0000 7f39cc400700 1 -- 172.21.15.16:0/1263408967 shutdown_connections 2024-04-05T23:32:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.892+0000 7f39cc400700 1 -- 172.21.15.16:0/1263408967 wait complete. 2024-04-05T23:32:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.894+0000 7f39cc400700 1 Processor -- start 2024-04-05T23:32:29.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.894+0000 7f39cc400700 1 -- start start 2024-04-05T23:32:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.894+0000 7f39cc400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c4073650 0x7f39c41a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.894+0000 7f39cc400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c4074f40 0x7f39c41a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.894+0000 7f39cc400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c4111b70 0x7f39c419d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.894+0000 7f39cc400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c419de90 con 0x7f39c4111b70 2024-04-05T23:32:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.894+0000 7f39cc400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c419e000 con 0x7f39c4074f40 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.894+0000 7f39cc400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c419e1c0 con 0x7f39c4073650 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c4073650 0x7f39c41a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c4073650 0x7f39c41a36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43146/0 (socket says 172.21.15.16:43146) 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9a00700 1 -- 172.21.15.16:0/2138151968 learned_addr learned my addr 172.21.15.16:0/2138151968 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39ca400700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c4111b70 0x7f39c419d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9000700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c4074f40 0x7f39c41a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9a00700 1 -- 172.21.15.16:0/2138151968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c4074f40 msgr2=0x7f39c41a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9a00700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c4074f40 0x7f39c41a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9a00700 1 -- 172.21.15.16:0/2138151968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c4111b70 msgr2=0x7f39c419d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9a00700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c4111b70 0x7f39c419d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9a00700 1 -- 172.21.15.16:0/2138151968 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b00095f0 con 0x7f39c4073650 2024-04-05T23:32:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39ca400700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39c4111b70 0x7f39c419d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.895+0000 7f39c9000700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c4074f40 0x7f39c41a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.896+0000 7f39c9a00700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c4073650 0x7f39c41a36e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39b800e9a0 tx=0x7f39b800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.896+0000 7f39c2000700 1 -- 172.21.15.16:0/2138151968 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b8009790 con 0x7f39c4073650 2024-04-05T23:32:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.896+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c419e4a0 con 0x7f39c4073650 2024-04-05T23:32:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.896+0000 7f39c2000700 1 -- 172.21.15.16:0/2138151968 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39b8010430 con 0x7f39c4073650 2024-04-05T23:32:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.896+0000 7f39c2000700 1 -- 172.21.15.16:0/2138151968 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b8018680 con 0x7f39c4073650 2024-04-05T23:32:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.896+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c41aa6d0 con 0x7f39c4073650 2024-04-05T23:32:29.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.897+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f39c4117780 con 0x7f39c4073650 2024-04-05T23:32:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.901+0000 7f39c2000700 1 -- 172.21.15.16:0/2138151968 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f39b80105a0 con 0x7f39c4073650 2024-04-05T23:32:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.902+0000 7f39c2000700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f39ac0a1230 0x7f39ac0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.902+0000 7f39c2000700 1 -- 172.21.15.16:0/2138151968 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f39b8014070 con 0x7f39c4073650 2024-04-05T23:32:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.902+0000 7f39c9000700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f39ac0a1230 0x7f39ac0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:29.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.903+0000 7f39c2000700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] conn(0x7f39ac0aaa10 0x7f39ac0ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.903+0000 7f39c2000700 1 -- 172.21.15.16:0/2138151968 --> [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39ac0ad4e0 con 0x7f39ac0aaa10 2024-04-05T23:32:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.903+0000 7f39c2000700 1 -- 172.21.15.16:0/2138151968 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39b80cd160 con 0x7f39c4073650 2024-04-05T23:32:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.903+0000 7f39c9000700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f39ac0a1230 0x7f39ac0a36f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f39c419ec40 tx=0x7f39b000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.903+0000 7f39ca400700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] conn(0x7f39ac0aaa10 0x7f39ac0ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:29.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.904+0000 7f39ca400700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] conn(0x7f39ac0aaa10 0x7f39ac0ace30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.905+0000 7f39c2000700 1 -- 172.21.15.16:0/2138151968 <== osd.10 v2:172.21.15.27:6824/15853906 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f39ac0ad4e0 con 0x7f39ac0aaa10 2024-04-05T23:32:29.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.921+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 --> [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f39c41104b0 con 0x7f39ac0aaa10 2024-04-05T23:32:29.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.922+0000 7f39c2000700 1 -- 172.21.15.16:0/2138151968 <== osd.10 v2:172.21.15.27:6824/15853906 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f39c41104b0 con 0x7f39ac0aaa10 2024-04-05T23:32:29.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.922+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 >> [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] conn(0x7f39ac0aaa10 msgr2=0x7f39ac0ace30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.922+0000 7f39cc400700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] conn(0x7f39ac0aaa10 0x7f39ac0ace30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.922+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f39ac0a1230 msgr2=0x7f39ac0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.922+0000 7f39cc400700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f39ac0a1230 0x7f39ac0a36f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f39c419ec40 tx=0x7f39b000b260 comp rx=0 tx=0).stop 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.922+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c4073650 msgr2=0x7f39c41a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.922+0000 7f39cc400700 1 --2- 172.21.15.16:0/2138151968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c4073650 0x7f39c41a36e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39b800e9a0 tx=0x7f39b800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.922+0000 7f39ca400700 1 -- 172.21.15.16:0/2138151968 reap_dead start 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.923+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 shutdown_connections 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.923+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 >> 172.21.15.16:0/2138151968 conn(0x7f39c40780d0 msgr2=0x7f39c410fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.923+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 shutdown_connections 2024-04-05T23:32:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:29.923+0000 7f39cc400700 1 -- 172.21.15.16:0/2138151968 wait complete. 2024-04-05T23:32:30.023 INFO:teuthology.orchestra.run.smithi016.stdout:180388626466 2024-04-05T23:32:30.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-05T23:32:30.498 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:30 smithi016 ceph-mon[38248]: pgmap v245: 97 pgs: 97 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-05T23:32:30.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:30 smithi027 ceph-mon[78597]: pgmap v245: 97 pgs: 97 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-05T23:32:30.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:30 smithi130 ceph-mon[79417]: pgmap v245: 97 pgs: 97 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-05T23:32:31.710 INFO:teuthology.orchestra.run.smithi016.stdout:287762808851 2024-04-05T23:32:31.711 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.10 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.420+0000 7fde4f200700 1 -- 172.21.15.16:0/2731779381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde50102860 msgr2=0x7fde50106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.420+0000 7fde4f200700 1 --2- 172.21.15.16:0/2731779381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde50102860 0x7fde50106d10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0099c0 tx=0x7fde3c009cd0 comp rx=0 tx=0).stop 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.420+0000 7fde4f200700 1 -- 172.21.15.16:0/2731779381 shutdown_connections 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.420+0000 7fde4f200700 1 --2- 172.21.15.16:0/2731779381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde50107440 0x7fde5010ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.420+0000 7fde4f200700 1 --2- 172.21.15.16:0/2731779381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde50101eb0 0x7fde50102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.420+0000 7fde4f200700 1 --2- 172.21.15.16:0/2731779381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde50102860 0x7fde50106d10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.420+0000 7fde4f200700 1 -- 172.21.15.16:0/2731779381 >> 172.21.15.16:0/2731779381 conn(0x7fde500fb720 msgr2=0x7fde500fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.420+0000 7fde4f200700 1 -- 172.21.15.16:0/2731779381 shutdown_connections 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.420+0000 7fde4f200700 1 -- 172.21.15.16:0/2731779381 wait complete. 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.421+0000 7fde4f200700 1 Processor -- start 2024-04-05T23:32:32.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.421+0000 7fde4f200700 1 -- start start 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.421+0000 7fde4f200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde50101eb0 0x7fde501016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.421+0000 7fde4f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde50102860 0x7fde500ffb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4f200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde50107440 0x7fde50100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4f200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde501008e0 con 0x7fde50107440 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde501a9cb0 con 0x7fde50102860 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4f200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde501a9df0 con 0x7fde50101eb0 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde50107440 0x7fde50100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde50107440 0x7fde50100150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54512/0 (socket says 172.21.15.16:54512) 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4e800700 1 -- 172.21.15.16:0/2269924962 learned_addr learned my addr 172.21.15.16:0/2269924962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4e800700 1 -- 172.21.15.16:0/2269924962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde50101eb0 msgr2=0x7fde501016b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4e800700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde50101eb0 0x7fde501016b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4e800700 1 -- 172.21.15.16:0/2269924962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde50102860 msgr2=0x7fde500ffb10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-05T23:32:32.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4e800700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde50102860 0x7fde500ffb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.422+0000 7fde4e800700 1 -- 172.21.15.16:0/2269924962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c0095f0 con 0x7fde50107440 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.423+0000 7fde4e800700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde50107440 0x7fde50100150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fde48007ee0 tx=0x7fde4800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.423+0000 7fde46a00700 1 -- 172.21.15.16:0/2269924962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48017070 con 0x7fde50107440 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.423+0000 7fde46a00700 1 -- 172.21.15.16:0/2269924962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde4800f040 con 0x7fde50107440 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.423+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde501aa080 con 0x7fde50107440 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.423+0000 7fde46a00700 1 -- 172.21.15.16:0/2269924962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48005740 con 0x7fde50107440 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.423+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde501aa5d0 con 0x7fde50107440 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.424+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde50111090 con 0x7fde50107440 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.426+0000 7fde46a00700 1 -- 172.21.15.16:0/2269924962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fde480058a0 con 0x7fde50107440 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.428+0000 7fde46a00700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fde340a1230 0x7fde340a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.428+0000 7fde46a00700 1 -- 172.21.15.16:0/2269924962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fde480cc820 con 0x7fde50107440 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.428+0000 7fde4de00700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fde340a1230 0x7fde340a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.428+0000 7fde46a00700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] conn(0x7fde340aaa10 0x7fde340ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:32.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.428+0000 7fde46a00700 1 -- 172.21.15.16:0/2269924962 --> [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde340ad4e0 con 0x7fde340aaa10 2024-04-05T23:32:32.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.428+0000 7fde46a00700 1 -- 172.21.15.16:0/2269924962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde480ccb90 con 0x7fde50107440 2024-04-05T23:32:32.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.429+0000 7fde4d400700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] conn(0x7fde340aaa10 0x7fde340ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:32.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.429+0000 7fde4de00700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fde340a1230 0x7fde340a36f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde5006a310 tx=0x7fde4000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:32.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.429+0000 7fde4d400700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] conn(0x7fde340aaa10 0x7fde340ace30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:32.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.430+0000 7fde46a00700 1 -- 172.21.15.16:0/2269924962 <== osd.4 v2:172.21.15.27:6808/1591043365 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fde340ad4e0 con 0x7fde340aaa10 2024-04-05T23:32:32.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.452+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 --> [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fde50061560 con 0x7fde340aaa10 2024-04-05T23:32:32.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.452+0000 7fde46a00700 1 -- 172.21.15.16:0/2269924962 <== osd.4 v2:172.21.15.27:6808/1591043365 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fde50061560 con 0x7fde340aaa10 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 >> [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] conn(0x7fde340aaa10 msgr2=0x7fde340ace30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] conn(0x7fde340aaa10 0x7fde340ace30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fde340a1230 msgr2=0x7fde340a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fde340a1230 0x7fde340a36f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde5006a310 tx=0x7fde4000b410 comp rx=0 tx=0).stop 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde50107440 msgr2=0x7fde50100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 --2- 172.21.15.16:0/2269924962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde50107440 0x7fde50100150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fde48007ee0 tx=0x7fde4800d3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4e800700 1 -- 172.21.15.16:0/2269924962 reap_dead start 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 shutdown_connections 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 >> 172.21.15.16:0/2269924962 conn(0x7fde500fb720 msgr2=0x7fde50105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 shutdown_connections 2024-04-05T23:32:32.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:31.453+0000 7fde4f200700 1 -- 172.21.15.16:0/2269924962 wait complete. 2024-04-05T23:32:32.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:32 smithi016 ceph-mon[38248]: pgmap v246: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-05T23:32:32.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:32 smithi027 ceph-mon[78597]: pgmap v246: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-05T23:32:32.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:32 smithi130 ceph-mon[79417]: pgmap v246: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-05T23:32:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.729+0000 7fdd86a00700 1 -- 172.21.15.16:0/3053872799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd80101900 msgr2=0x7fdd80105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.729+0000 7fdd86a00700 1 --2- 172.21.15.16:0/3053872799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd80101900 0x7fdd80105950 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009990 tx=0x7fdd6c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.729+0000 7fdd86a00700 1 -- 172.21.15.16:0/3053872799 shutdown_connections 2024-04-05T23:32:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.729+0000 7fdd86a00700 1 --2- 172.21.15.16:0/3053872799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd80106030 0x7fdd8010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.729+0000 7fdd86a00700 1 --2- 172.21.15.16:0/3053872799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd80100f50 0x7fdd80101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.730+0000 7fdd86a00700 1 --2- 172.21.15.16:0/3053872799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd80101900 0x7fdd80105950 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.730+0000 7fdd86a00700 1 -- 172.21.15.16:0/3053872799 >> 172.21.15.16:0/3053872799 conn(0x7fdd800fc840 msgr2=0x7fdd800fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.730+0000 7fdd86a00700 1 -- 172.21.15.16:0/3053872799 shutdown_connections 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.730+0000 7fdd86a00700 1 -- 172.21.15.16:0/3053872799 wait complete. 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.731+0000 7fdd86a00700 1 Processor -- start 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.731+0000 7fdd86a00700 1 -- start start 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.731+0000 7fdd86a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd80100f50 0x7fdd80199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.731+0000 7fdd86a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd80101900 0x7fdd80199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.731+0000 7fdd86a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd80106030 0x7fdd8019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.731+0000 7fdd86a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8019e330 con 0x7fdd80101900 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.731+0000 7fdd86a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8019e4a0 con 0x7fdd80106030 2024-04-05T23:32:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.731+0000 7fdd86a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8019e660 con 0x7fdd80100f50 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.731+0000 7fdd84c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd80101900 0x7fdd80199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd84c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd80101900 0x7fdd80199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39370/0 (socket says 172.21.15.16:39370) 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd84c00700 1 -- 172.21.15.16:0/1000435880 learned_addr learned my addr 172.21.15.16:0/1000435880 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd85600700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd80100f50 0x7fdd80199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd86000700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd80106030 0x7fdd8019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd84c00700 1 -- 172.21.15.16:0/1000435880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd80100f50 msgr2=0x7fdd80199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd84c00700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd80100f50 0x7fdd80199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd84c00700 1 -- 172.21.15.16:0/1000435880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd80106030 msgr2=0x7fdd8019dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd84c00700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd80106030 0x7fdd8019dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd84c00700 1 -- 172.21.15.16:0/1000435880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c0095f0 con 0x7fdd80101900 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd84c00700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd80101900 0x7fdd80199970 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c000c00 tx=0x7fdd6c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.732+0000 7fdd7e000700 1 -- 172.21.15.16:0/1000435880 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c01d070 con 0x7fdd80101900 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.733+0000 7fdd7e000700 1 -- 172.21.15.16:0/1000435880 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd6c021410 con 0x7fdd80101900 2024-04-05T23:32:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.733+0000 7fdd7e000700 1 -- 172.21.15.16:0/1000435880 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c017650 con 0x7fdd80101900 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.733+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8019e8e0 con 0x7fdd80101900 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.733+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd801a5fd0 con 0x7fdd80101900 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.733+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdd80110f30 con 0x7fdd80101900 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.736+0000 7fdd7e000700 1 -- 172.21.15.16:0/1000435880 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fdd6c021580 con 0x7fdd80101900 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.737+0000 7fdd7e000700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fdd680a1230 0x7fdd680a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.737+0000 7fdd7e000700 1 -- 172.21.15.16:0/1000435880 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdd6c0ce5b0 con 0x7fdd80101900 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.737+0000 7fdd85600700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fdd680a1230 0x7fdd680a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.737+0000 7fdd7e000700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] conn(0x7fdd680aaa10 0x7fdd680ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.738+0000 7fdd7e000700 1 -- 172.21.15.16:0/1000435880 --> [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd680ad4e0 con 0x7fdd680aaa10 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.738+0000 7fdd7e000700 1 -- 172.21.15.16:0/1000435880 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd6c0ce920 con 0x7fdd80101900 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.738+0000 7fdd86000700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] conn(0x7fdd680aaa10 0x7fdd680ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.738+0000 7fdd85600700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fdd680a1230 0x7fdd680a36f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009c60 tx=0x7fdd74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.738+0000 7fdd86000700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] conn(0x7fdd680aaa10 0x7fdd680ace30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.739+0000 7fdd7e000700 1 -- 172.21.15.16:0/1000435880 <== osd.1 v2:172.21.15.27:6800/1426362237 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdd680ad4e0 con 0x7fdd680aaa10 2024-04-05T23:32:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.756+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 --> [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdd800743e0 con 0x7fdd680aaa10 2024-04-05T23:32:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.756+0000 7fdd7e000700 1 -- 172.21.15.16:0/1000435880 <== osd.1 v2:172.21.15.27:6800/1426362237 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdd800743e0 con 0x7fdd680aaa10 2024-04-05T23:32:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 >> [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] conn(0x7fdd680aaa10 msgr2=0x7fdd680ace30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] conn(0x7fdd680aaa10 0x7fdd680ace30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fdd680a1230 msgr2=0x7fdd680a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fdd680a1230 0x7fdd680a36f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009c60 tx=0x7fdd74009260 comp rx=0 tx=0).stop 2024-04-05T23:32:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd80101900 msgr2=0x7fdd80199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 --2- 172.21.15.16:0/1000435880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd80101900 0x7fdd80199970 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c000c00 tx=0x7fdd6c009580 comp rx=0 tx=0).stop 2024-04-05T23:32:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86000700 1 -- 172.21.15.16:0/1000435880 reap_dead start 2024-04-05T23:32:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 shutdown_connections 2024-04-05T23:32:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 >> 172.21.15.16:0/1000435880 conn(0x7fdd800fc840 msgr2=0x7fdd800fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 shutdown_connections 2024-04-05T23:32:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.757+0000 7fdd86a00700 1 -- 172.21.15.16:0/1000435880 wait complete. 2024-04-05T23:32:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.924+0000 7f2baea00700 1 -- 172.21.15.16:0/642756595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ba8068dd0 msgr2=0x7f2ba8069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.924+0000 7f2baea00700 1 --2- 172.21.15.16:0/642756595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ba8068dd0 0x7f2ba8069250 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b94009990 tx=0x7f2b94009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.924+0000 7f2baea00700 1 -- 172.21.15.16:0/642756595 shutdown_connections 2024-04-05T23:32:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.924+0000 7f2baea00700 1 --2- 172.21.15.16:0/642756595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8103910 0x7f2ba8103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.924+0000 7f2baea00700 1 --2- 172.21.15.16:0/642756595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba8105e00 0x7f2ba8068890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.924+0000 7f2baea00700 1 --2- 172.21.15.16:0/642756595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ba8068dd0 0x7f2ba8069250 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.924+0000 7f2baea00700 1 -- 172.21.15.16:0/642756595 >> 172.21.15.16:0/642756595 conn(0x7f2ba8075ec0 msgr2=0x7f2ba80762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.925+0000 7f2baea00700 1 -- 172.21.15.16:0/642756595 shutdown_connections 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.925+0000 7f2baea00700 1 -- 172.21.15.16:0/642756595 wait complete. 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.926+0000 7f2baea00700 1 Processor -- start 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.926+0000 7f2baea00700 1 -- start start 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2baea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ba8068dd0 0x7f2ba81a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2baea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8103910 0x7f2ba81a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2baea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba8105e00 0x7f2ba819d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2baea00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba819de70 con 0x7f2ba8103910 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2baea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba819dfe0 con 0x7f2ba8105e00 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2baea00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba819e1a0 con 0x7f2ba8068dd0 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2ba7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ba8068dd0 0x7f2ba81a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2ba7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ba8068dd0 0x7f2ba81a36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59438/0 (socket says 172.21.15.16:59438) 2024-04-05T23:32:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2ba7e00700 1 -- 172.21.15.16:0/3798297777 learned_addr learned my addr 172.21.15.16:0/3798297777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2ba7400700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8103910 0x7f2ba81a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.927+0000 7f2baca00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba8105e00 0x7f2ba819d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.928+0000 7f2ba7e00700 1 -- 172.21.15.16:0/3798297777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba8105e00 msgr2=0x7f2ba819d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.928+0000 7f2ba7e00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba8105e00 0x7f2ba819d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.928+0000 7f2ba7e00700 1 -- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8103910 msgr2=0x7f2ba81a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.928+0000 7f2ba7e00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8103910 0x7f2ba81a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.928+0000 7f2ba7e00700 1 -- 172.21.15.16:0/3798297777 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b940095f0 con 0x7f2ba8068dd0 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.928+0000 7f2ba7e00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ba8068dd0 0x7f2ba81a36c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800e9a0 tx=0x7f2b9800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.929+0000 7f2ba4c00700 1 -- 172.21.15.16:0/3798297777 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98009790 con 0x7f2ba8068dd0 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.929+0000 7f2ba4c00700 1 -- 172.21.15.16:0/3798297777 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b98010430 con 0x7f2ba8068dd0 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.929+0000 7f2ba4c00700 1 -- 172.21.15.16:0/3798297777 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98018680 con 0x7f2ba8068dd0 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.929+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba819e480 con 0x7f2ba8068dd0 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.929+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba81aa6b0 con 0x7f2ba8068dd0 2024-04-05T23:32:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.930+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2ba81177c0 con 0x7f2ba8068dd0 2024-04-05T23:32:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.932+0000 7f2ba4c00700 1 -- 172.21.15.16:0/3798297777 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f2b980105a0 con 0x7f2ba8068dd0 2024-04-05T23:32:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.934+0000 7f2ba4c00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2b8c0a1230 0x7f2b8c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.934+0000 7f2ba4c00700 1 -- 172.21.15.16:0/3798297777 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2b98014070 con 0x7f2ba8068dd0 2024-04-05T23:32:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.934+0000 7f2ba7400700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2b8c0a1230 0x7f2b8c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.934+0000 7f2ba4c00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] conn(0x7f2b8c0aaa10 0x7f2b8c0ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.934+0000 7f2ba4c00700 1 -- 172.21.15.16:0/3798297777 --> [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b8c0ad4e0 con 0x7f2b8c0aaa10 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.934+0000 7f2ba4c00700 1 -- 172.21.15.16:0/3798297777 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b980cd160 con 0x7f2ba8068dd0 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.934+0000 7f2baca00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] conn(0x7f2b8c0aaa10 0x7f2b8c0ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.935+0000 7f2ba7400700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2b8c0a1230 0x7f2b8c0a36f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2ba819ec20 tx=0x7f2b94005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.935+0000 7f2baca00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] conn(0x7f2b8c0aaa10 0x7f2b8c0ace30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.937+0000 7f2ba4c00700 1 -- 172.21.15.16:0/3798297777 <== osd.9 v2:172.21.15.16:6826/2518129600 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2b8c0ad4e0 con 0x7f2b8c0aaa10 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.955+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 --> [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2ba81032c0 con 0x7f2b8c0aaa10 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.955+0000 7f2ba4c00700 1 -- 172.21.15.16:0/3798297777 <== osd.9 v2:172.21.15.16:6826/2518129600 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2ba81032c0 con 0x7f2b8c0aaa10 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] conn(0x7f2b8c0aaa10 msgr2=0x7f2b8c0ace30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] conn(0x7f2b8c0aaa10 0x7f2b8c0ace30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2b8c0a1230 msgr2=0x7f2b8c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2b8c0a1230 0x7f2b8c0a36f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2ba819ec20 tx=0x7f2b94005fd0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ba8068dd0 msgr2=0x7f2ba81a36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 --2- 172.21.15.16:0/3798297777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ba8068dd0 0x7f2ba81a36c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800e9a0 tx=0x7f2b9800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baca00700 1 -- 172.21.15.16:0/3798297777 reap_dead start 2024-04-05T23:32:33.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 shutdown_connections 2024-04-05T23:32:33.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 >> 172.21.15.16:0/3798297777 conn(0x7f2ba8075ec0 msgr2=0x7f2ba81028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:33.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 shutdown_connections 2024-04-05T23:32:33.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:32.956+0000 7f2baea00700 1 -- 172.21.15.16:0/3798297777 wait complete. 2024-04-05T23:32:33.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.066+0000 7f49fc600700 1 -- 172.21.15.16:0/461575946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f40ffe10 msgr2=0x7f49f4100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.066+0000 7f49fc600700 1 --2- 172.21.15.16:0/461575946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f40ffe10 0x7f49f4100290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009990 tx=0x7f49e0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.066+0000 7f49fc600700 1 -- 172.21.15.16:0/461575946 shutdown_connections 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.066+0000 7f49fc600700 1 --2- 172.21.15.16:0/461575946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f41007d0 0x7f49f410cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.066+0000 7f49fc600700 1 --2- 172.21.15.16:0/461575946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f4105e40 0x7f49f4106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.066+0000 7f49fc600700 1 --2- 172.21.15.16:0/461575946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f40ffe10 0x7f49f4100290 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.066+0000 7f49fc600700 1 -- 172.21.15.16:0/461575946 >> 172.21.15.16:0/461575946 conn(0x7f49f4075ec0 msgr2=0x7f49f40762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.067+0000 7f49fc600700 1 -- 172.21.15.16:0/461575946 shutdown_connections 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.067+0000 7f49fc600700 1 -- 172.21.15.16:0/461575946 wait complete. 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.068+0000 7f49fc600700 1 Processor -- start 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.068+0000 7f49fc600700 1 -- start start 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.068+0000 7f49fc600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40ffe10 0x7f49f4197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.068+0000 7f49fc600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f41007d0 0x7f49f41978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.068+0000 7f49fc600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f4105e40 0x7f49f419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.068+0000 7f49fc600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f419c230 con 0x7f49f4105e40 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.068+0000 7f49fc600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f419c3a0 con 0x7f49f40ffe10 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.068+0000 7f49fc600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f419c560 con 0x7f49f41007d0 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49fa600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f4105e40 0x7f49f419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49f9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40ffe10 0x7f49f4197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49fa600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f4105e40 0x7f49f419bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39402/0 (socket says 172.21.15.16:39402) 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49fa600700 1 -- 172.21.15.16:0/3423860657 learned_addr learned my addr 172.21.15.16:0/3423860657 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49f9200700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f41007d0 0x7f49f41978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49fa600700 1 -- 172.21.15.16:0/3423860657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f41007d0 msgr2=0x7f49f41978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49fa600700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f41007d0 0x7f49f41978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49fa600700 1 -- 172.21.15.16:0/3423860657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40ffe10 msgr2=0x7f49f4197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49fa600700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40ffe10 0x7f49f4197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49fa600700 1 -- 172.21.15.16:0/3423860657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e8009520 con 0x7f49f4105e40 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.069+0000 7f49fa600700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f4105e40 0x7f49f419bc20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00ef30 tx=0x7f49ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.070+0000 7f49f2000700 1 -- 172.21.15.16:0/3423860657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec018070 con 0x7f49f4105e40 2024-04-05T23:32:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.070+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e00095f0 con 0x7f49f4105e40 2024-04-05T23:32:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.070+0000 7f49f2000700 1 -- 172.21.15.16:0/3423860657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49ec00f040 con 0x7f49f4105e40 2024-04-05T23:32:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.070+0000 7f49f2000700 1 -- 172.21.15.16:0/3423860657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec00cd00 con 0x7f49f4105e40 2024-04-05T23:32:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.070+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f41a3ed0 con 0x7f49f4105e40 2024-04-05T23:32:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.071+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f49f41014f0 con 0x7f49f4105e40 2024-04-05T23:32:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.073+0000 7f49f2000700 1 -- 172.21.15.16:0/3423860657 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f49ec003a00 con 0x7f49f4105e40 2024-04-05T23:32:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.074+0000 7f49f2000700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f49dc0a11b0 0x7f49dc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.075+0000 7f49f2000700 1 -- 172.21.15.16:0/3423860657 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f49ec0cce50 con 0x7f49f4105e40 2024-04-05T23:32:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.075+0000 7f49f9c00700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f49dc0a11b0 0x7f49dc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.075+0000 7f49f2000700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] conn(0x7f49dc0aa990 0x7f49dc0acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.075+0000 7f49f2000700 1 -- 172.21.15.16:0/3423860657 --> [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49dc0ad460 con 0x7f49dc0aa990 2024-04-05T23:32:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.075+0000 7f49f2000700 1 -- 172.21.15.16:0/3423860657 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49ec0cd1c0 con 0x7f49f4105e40 2024-04-05T23:32:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.075+0000 7f49f9200700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] conn(0x7f49dc0aa990 0x7f49dc0acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.075+0000 7f49f9c00700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f49dc0a11b0 0x7f49dc0a3670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009d70 tx=0x7f49e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.076+0000 7f49f9200700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] conn(0x7f49dc0aa990 0x7f49dc0acdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.077+0000 7f49f2000700 1 -- 172.21.15.16:0/3423860657 <== osd.7 v2:172.21.15.27:6816/3123372166 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f49dc0ad460 con 0x7f49dc0aa990 2024-04-05T23:32:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.094+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 --> [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f49f410ff80 con 0x7f49dc0aa990 2024-04-05T23:32:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49f2000700 1 -- 172.21.15.16:0/3423860657 <== osd.7 v2:172.21.15.27:6816/3123372166 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f49f410ff80 con 0x7f49dc0aa990 2024-04-05T23:32:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 >> [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] conn(0x7f49dc0aa990 msgr2=0x7f49dc0acdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49fc600700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] conn(0x7f49dc0aa990 0x7f49dc0acdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f49dc0a11b0 msgr2=0x7f49dc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49fc600700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f49dc0a11b0 0x7f49dc0a3670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009d70 tx=0x7f49e8009260 comp rx=0 tx=0).stop 2024-04-05T23:32:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f4105e40 msgr2=0x7f49f419bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49fc600700 1 --2- 172.21.15.16:0/3423860657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f4105e40 0x7f49f419bc20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00ef30 tx=0x7f49ec00c3a0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49fa600700 1 -- 172.21.15.16:0/3423860657 reap_dead start 2024-04-05T23:32:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 shutdown_connections 2024-04-05T23:32:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.095+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 >> 172.21.15.16:0/3423860657 conn(0x7f49f4075ec0 msgr2=0x7f49f4104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.096+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 shutdown_connections 2024-04-05T23:32:33.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.096+0000 7f49fc600700 1 -- 172.21.15.16:0/3423860657 wait complete. 2024-04-05T23:32:33.341 INFO:teuthology.orchestra.run.smithi016.stdout:128849018923 2024-04-05T23:32:33.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-05T23:32:33.629 INFO:teuthology.orchestra.run.smithi016.stdout:266287972375 2024-04-05T23:32:33.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.9 2024-04-05T23:32:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.136+0000 7f13c8600700 1 -- 172.21.15.16:0/541565846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13c0068890 msgr2=0x7f13c0068cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.136+0000 7f13c8600700 1 --2- 172.21.15.16:0/541565846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13c0068890 0x7f13c0068cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0098f0 tx=0x7f13ac009c00 comp rx=0 tx=0).stop 2024-04-05T23:32:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.136+0000 7f13c8600700 1 -- 172.21.15.16:0/541565846 shutdown_connections 2024-04-05T23:32:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.136+0000 7f13c8600700 1 --2- 172.21.15.16:0/541565846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c0106f60 0x7f13c0107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.136+0000 7f13c8600700 1 --2- 172.21.15.16:0/541565846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c0069230 0x7f13c010dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.136+0000 7f13c8600700 1 --2- 172.21.15.16:0/541565846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13c0068890 0x7f13c0068cf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.136+0000 7f13c8600700 1 -- 172.21.15.16:0/541565846 >> 172.21.15.16:0/541565846 conn(0x7f13c0075ec0 msgr2=0x7f13c00762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.136+0000 7f13c8600700 1 -- 172.21.15.16:0/541565846 shutdown_connections 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.136+0000 7f13c8600700 1 -- 172.21.15.16:0/541565846 wait complete. 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.137+0000 7f13c8600700 1 Processor -- start 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.137+0000 7f13c8600700 1 -- start start 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.137+0000 7f13c8600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c0068890 0x7f13c01995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.137+0000 7f13c8600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13c0069230 0x7f13c0199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.137+0000 7f13c8600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c0106f60 0x7f13c019dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.137+0000 7f13c8600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c019e4d0 con 0x7f13c0106f60 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.137+0000 7f13c8600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c019e640 con 0x7f13c0069230 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.137+0000 7f13c8600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c019e800 con 0x7f13c0068890 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c6600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c0106f60 0x7f13c019dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c6600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c0106f60 0x7f13c019dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39418/0 (socket says 172.21.15.16:39418) 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c0068890 0x7f13c01995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c6600700 1 -- 172.21.15.16:0/411163558 learned_addr learned my addr 172.21.15.16:0/411163558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c5200700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13c0069230 0x7f13c0199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c6600700 1 -- 172.21.15.16:0/411163558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c0068890 msgr2=0x7f13c01995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c6600700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c0068890 0x7f13c01995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c6600700 1 -- 172.21.15.16:0/411163558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13c0069230 msgr2=0x7f13c0199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c6600700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13c0069230 0x7f13c0199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c6600700 1 -- 172.21.15.16:0/411163558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac009520 con 0x7f13c0106f60 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.138+0000 7f13c6600700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c0106f60 0x7f13c019dec0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f13b8007ee0 tx=0x7f13b800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.139+0000 7f13be000700 1 -- 172.21.15.16:0/411163558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b800ddd0 con 0x7f13c0106f60 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.139+0000 7f13be000700 1 -- 172.21.15.16:0/411163558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13b800f040 con 0x7f13c0106f60 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.139+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c019eae0 con 0x7f13c0106f60 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.139+0000 7f13be000700 1 -- 172.21.15.16:0/411163558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8004800 con 0x7f13c0106f60 2024-04-05T23:32:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.139+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c01a6340 con 0x7f13c0106f60 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.140+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f13c0111130 con 0x7f13c0106f60 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.142+0000 7f13be000700 1 -- 172.21.15.16:0/411163558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f13b8003a00 con 0x7f13c0106f60 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.143+0000 7f13be000700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f13a80a11b0 0x7f13a80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.143+0000 7f13be000700 1 -- 172.21.15.16:0/411163558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f13b80cccc0 con 0x7f13c0106f60 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.143+0000 7f13c5c00700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f13a80a11b0 0x7f13a80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.143+0000 7f13be000700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] conn(0x7f13a80aa990 0x7f13a80acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.144+0000 7f13be000700 1 -- 172.21.15.16:0/411163558 --> [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13a80ad460 con 0x7f13a80aa990 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.144+0000 7f13be000700 1 -- 172.21.15.16:0/411163558 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13b80cd030 con 0x7f13c0106f60 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.144+0000 7f13c5200700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] conn(0x7f13a80aa990 0x7f13a80acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.144+0000 7f13c5c00700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f13a80a11b0 0x7f13a80a3670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f13b4005fd0 tx=0x7f13b4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.144+0000 7f13c5200700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] conn(0x7f13a80aa990 0x7f13a80acdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.145+0000 7f13be000700 1 -- 172.21.15.16:0/411163558 <== osd.0 v2:172.21.15.16:6802/2345860669 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f13a80ad460 con 0x7f13a80aa990 2024-04-05T23:32:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.162+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 --> [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f13c0102610 con 0x7f13a80aa990 2024-04-05T23:32:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.163+0000 7f13be000700 1 -- 172.21.15.16:0/411163558 <== osd.0 v2:172.21.15.16:6802/2345860669 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f13c0102610 con 0x7f13a80aa990 2024-04-05T23:32:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.163+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] conn(0x7f13a80aa990 msgr2=0x7f13a80acdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.163+0000 7f13c8600700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] conn(0x7f13a80aa990 0x7f13a80acdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.163+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f13a80a11b0 msgr2=0x7f13a80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.163+0000 7f13c8600700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f13a80a11b0 0x7f13a80a3670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f13b4005fd0 tx=0x7f13b4009310 comp rx=0 tx=0).stop 2024-04-05T23:32:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.163+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c0106f60 msgr2=0x7f13c019dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.163+0000 7f13c8600700 1 --2- 172.21.15.16:0/411163558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c0106f60 0x7f13c019dec0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f13b8007ee0 tx=0x7f13b800d3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.164+0000 7f13c6600700 1 -- 172.21.15.16:0/411163558 reap_dead start 2024-04-05T23:32:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.164+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 shutdown_connections 2024-04-05T23:32:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.164+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 >> 172.21.15.16:0/411163558 conn(0x7f13c0075ec0 msgr2=0x7f13c010c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.164+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 shutdown_connections 2024-04-05T23:32:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:33.164+0000 7f13c8600700 1 -- 172.21.15.16:0/411163558 wait complete. 2024-04-05T23:32:34.261 INFO:teuthology.orchestra.run.smithi016.stdout:206158430239 2024-04-05T23:32:34.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-05T23:32:34.439 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:34 smithi016 ceph-mon[38248]: pgmap v247: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-05T23:32:34.469 INFO:teuthology.orchestra.run.smithi016.stdout:51539607606 2024-04-05T23:32:34.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-05T23:32:34.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.469+0000 7fb26f800700 1 -- 172.21.15.16:0/3532585944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb268103910 msgr2=0x7fb268103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.469+0000 7fb26f800700 1 --2- 172.21.15.16:0/3532585944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb268103910 0x7fb268103dc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb26000b0a0 tx=0x7fb26000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.470+0000 7fb26f800700 1 -- 172.21.15.16:0/3532585944 shutdown_connections 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.470+0000 7fb26f800700 1 --2- 172.21.15.16:0/3532585944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb268103910 0x7fb268103dc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.470+0000 7fb26f800700 1 --2- 172.21.15.16:0/3532585944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb268105e00 0x7fb268068890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.470+0000 7fb26f800700 1 --2- 172.21.15.16:0/3532585944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb268068dd0 0x7fb268069250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.470+0000 7fb26f800700 1 -- 172.21.15.16:0/3532585944 >> 172.21.15.16:0/3532585944 conn(0x7fb268075ec0 msgr2=0x7fb2680762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.470+0000 7fb26f800700 1 -- 172.21.15.16:0/3532585944 shutdown_connections 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.470+0000 7fb26f800700 1 -- 172.21.15.16:0/3532585944 wait complete. 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.471+0000 7fb26f800700 1 Processor -- start 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.471+0000 7fb26f800700 1 -- start start 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb26f800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb268068dd0 0x7fb2681a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb26f800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb268103910 0x7fb2681a3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb26f800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb268105e00 0x7fb26819d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb26f800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26819df20 con 0x7fb268105e00 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb26f800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26819e090 con 0x7fb268068dd0 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb26f800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26819e250 con 0x7fb268103910 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb267e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb268103910 0x7fb2681a3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb26d800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb268105e00 0x7fb26819d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb267e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb268103910 0x7fb2681a3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59506/0 (socket says 172.21.15.16:59506) 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb267e00700 1 -- 172.21.15.16:0/94064140 learned_addr learned my addr 172.21.15.16:0/94064140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.472+0000 7fb26ce00700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb268068dd0 0x7fb2681a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb267e00700 1 -- 172.21.15.16:0/94064140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb268068dd0 msgr2=0x7fb2681a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb267e00700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb268068dd0 0x7fb2681a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb267e00700 1 -- 172.21.15.16:0/94064140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb268105e00 msgr2=0x7fb26819d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb267e00700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb268105e00 0x7fb26819d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb267e00700 1 -- 172.21.15.16:0/94064140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb260009d00 con 0x7fb268103910 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb26ce00700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb268068dd0 0x7fb2681a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb267e00700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb268103910 0x7fb2681a3cc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb25400e9a0 tx=0x7fb25400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb265600700 1 -- 172.21.15.16:0/94064140 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254009790 con 0x7fb268103910 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb265600700 1 -- 172.21.15.16:0/94064140 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb254010430 con 0x7fb268103910 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.473+0000 7fb265600700 1 -- 172.21.15.16:0/94064140 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254018680 con 0x7fb268103910 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.474+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26819e530 con 0x7fb268103910 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.474+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2681aa770 con 0x7fb268103910 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.475+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb268117720 con 0x7fb268103910 2024-04-05T23:32:34.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.478+0000 7fb265600700 1 -- 172.21.15.16:0/94064140 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fb2540105a0 con 0x7fb268103910 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.480+0000 7fb265600700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2500a1230 0x7fb2500a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.480+0000 7fb265600700 1 -- 172.21.15.16:0/94064140 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb254014070 con 0x7fb268103910 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.480+0000 7fb26ce00700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2500a1230 0x7fb2500a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.480+0000 7fb265600700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] conn(0x7fb2500aaa10 0x7fb2500ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.480+0000 7fb265600700 1 -- 172.21.15.16:0/94064140 --> [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2500ad4e0 con 0x7fb2500aaa10 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.480+0000 7fb265600700 1 -- 172.21.15.16:0/94064140 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2540d12c0 con 0x7fb268103910 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.481+0000 7fb26d800700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] conn(0x7fb2500aaa10 0x7fb2500ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.481+0000 7fb26ce00700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2500a1230 0x7fb2500a36f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009fd0 tx=0x7fb25c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.481+0000 7fb26d800700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] conn(0x7fb2500aaa10 0x7fb2500ace30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.482+0000 7fb265600700 1 -- 172.21.15.16:0/94064140 <== osd.11 v2:172.21.15.130:6824/4254452589 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb2500ad4e0 con 0x7fb2500aaa10 2024-04-05T23:32:34.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.503+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 --> [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb268103240 con 0x7fb2500aaa10 2024-04-05T23:32:34.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.503+0000 7fb265600700 1 -- 172.21.15.16:0/94064140 <== osd.11 v2:172.21.15.130:6824/4254452589 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb268103240 con 0x7fb2500aaa10 2024-04-05T23:32:34.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 >> [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] conn(0x7fb2500aaa10 msgr2=0x7fb2500ace30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26f800700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] conn(0x7fb2500aaa10 0x7fb2500ace30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2500a1230 msgr2=0x7fb2500a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26f800700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb2500a1230 0x7fb2500a36f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009fd0 tx=0x7fb25c009260 comp rx=0 tx=0).stop 2024-04-05T23:32:34.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb268103910 msgr2=0x7fb2681a3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26f800700 1 --2- 172.21.15.16:0/94064140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb268103910 0x7fb2681a3cc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb25400e9a0 tx=0x7fb25400ecb0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26d800700 1 -- 172.21.15.16:0/94064140 reap_dead start 2024-04-05T23:32:34.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 shutdown_connections 2024-04-05T23:32:34.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 >> 172.21.15.16:0/94064140 conn(0x7fb268075ec0 msgr2=0x7fb2681028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:34.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.504+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 shutdown_connections 2024-04-05T23:32:34.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.505+0000 7fb26f800700 1 -- 172.21.15.16:0/94064140 wait complete. 2024-04-05T23:32:34.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:34 smithi027 ceph-mon[78597]: pgmap v247: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-05T23:32:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.805+0000 7f0b26600700 1 -- 172.21.15.16:0/2302418081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b20103350 msgr2=0x7f0b20103730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.805+0000 7f0b26600700 1 --2- 172.21.15.16:0/2302418081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b20103350 0x7f0b20103730 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b10009990 tx=0x7f0b10009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.805+0000 7f0b26600700 1 -- 172.21.15.16:0/2302418081 shutdown_connections 2024-04-05T23:32:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.805+0000 7f0b26600700 1 --2- 172.21.15.16:0/2302418081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b20108430 0x7f0b20110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.805+0000 7f0b26600700 1 --2- 172.21.15.16:0/2302418081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b20103350 0x7f0b20103730 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.805+0000 7f0b26600700 1 --2- 172.21.15.16:0/2302418081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b20103d00 0x7f0b20107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.805+0000 7f0b26600700 1 -- 172.21.15.16:0/2302418081 >> 172.21.15.16:0/2302418081 conn(0x7f0b200febc0 msgr2=0x7f0b20100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.805+0000 7f0b26600700 1 -- 172.21.15.16:0/2302418081 shutdown_connections 2024-04-05T23:32:34.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.805+0000 7f0b26600700 1 -- 172.21.15.16:0/2302418081 wait complete. 2024-04-05T23:32:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.806+0000 7f0b26600700 1 Processor -- start 2024-04-05T23:32:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.806+0000 7f0b26600700 1 -- start start 2024-04-05T23:32:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.806+0000 7f0b26600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b20103350 0x7f0b201995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.806+0000 7f0b26600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b20103d00 0x7f0b20199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.806+0000 7f0b26600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b20108430 0x7f0b2019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.806+0000 7f0b26600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2019e4a0 con 0x7f0b20103350 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.806+0000 7f0b26600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2019e610 con 0x7f0b20103d00 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.806+0000 7f0b26600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2019e7d0 con 0x7f0b20108430 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b20103350 0x7f0b201995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b20103350 0x7f0b201995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39452/0 (socket says 172.21.15.16:39452) 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1f400700 1 -- 172.21.15.16:0/398561464 learned_addr learned my addr 172.21.15.16:0/398561464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1fe00700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b20108430 0x7f0b2019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1f400700 1 -- 172.21.15.16:0/398561464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b20108430 msgr2=0x7f0b2019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1f400700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b20108430 0x7f0b2019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1f400700 1 -- 172.21.15.16:0/398561464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b20103d00 msgr2=0x7f0b20199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1f400700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b20103d00 0x7f0b20199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1f400700 1 -- 172.21.15.16:0/398561464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b100095f0 con 0x7f0b20103350 2024-04-05T23:32:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.807+0000 7f0b1f400700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b20103350 0x7f0b201995a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0b10005910 tx=0x7f0b10009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.808+0000 7f0b17e00700 1 -- 172.21.15.16:0/398561464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1001d070 con 0x7f0b20103350 2024-04-05T23:32:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.808+0000 7f0b17e00700 1 -- 172.21.15.16:0/398561464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b10022470 con 0x7f0b20103350 2024-04-05T23:32:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.808+0000 7f0b17e00700 1 -- 172.21.15.16:0/398561464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b10017650 con 0x7f0b20103350 2024-04-05T23:32:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.808+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2019ea50 con 0x7f0b20103350 2024-04-05T23:32:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.808+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b201a6140 con 0x7f0b20103350 2024-04-05T23:32:34.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.809+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b20113350 con 0x7f0b20103350 2024-04-05T23:32:34.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.811+0000 7f0b17e00700 1 -- 172.21.15.16:0/398561464 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f0b100225e0 con 0x7f0b20103350 2024-04-05T23:32:34.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.812+0000 7f0b17e00700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b040a11b0 0x7f0b040a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.812+0000 7f0b17e00700 1 -- 172.21.15.16:0/398561464 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0b100ce560 con 0x7f0b20103350 2024-04-05T23:32:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.812+0000 7f0b1ea00700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b040a11b0 0x7f0b040a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.813+0000 7f0b17e00700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] conn(0x7f0b040aa990 0x7f0b040acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.813+0000 7f0b17e00700 1 -- 172.21.15.16:0/398561464 --> [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b040ad460 con 0x7f0b040aa990 2024-04-05T23:32:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.813+0000 7f0b17e00700 1 -- 172.21.15.16:0/398561464 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b100ce8d0 con 0x7f0b20103350 2024-04-05T23:32:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.813+0000 7f0b1fe00700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] conn(0x7f0b040aa990 0x7f0b040acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.813+0000 7f0b1ea00700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b040a11b0 0x7f0b040a3670 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009490 tx=0x7f0b0c009420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:34.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.813+0000 7f0b1fe00700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] conn(0x7f0b040aa990 0x7f0b040acdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:34.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.814+0000 7f0b17e00700 1 -- 172.21.15.16:0/398561464 <== osd.3 v2:172.21.15.16:6810/1156660397 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0b040ad460 con 0x7f0b040aa990 2024-04-05T23:32:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.833+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 --> [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0b200615e0 con 0x7f0b040aa990 2024-04-05T23:32:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b17e00700 1 -- 172.21.15.16:0/398561464 <== osd.3 v2:172.21.15.16:6810/1156660397 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0b200615e0 con 0x7f0b040aa990 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] conn(0x7f0b040aa990 msgr2=0x7f0b040acdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] conn(0x7f0b040aa990 0x7f0b040acdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b040a11b0 msgr2=0x7f0b040a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0b040a11b0 0x7f0b040a3670 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009490 tx=0x7f0b0c009420 comp rx=0 tx=0).stop 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b20103350 msgr2=0x7f0b201995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 --2- 172.21.15.16:0/398561464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b20103350 0x7f0b201995a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0b10005910 tx=0x7f0b10009570 comp rx=0 tx=0).stop 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b1fe00700 1 -- 172.21.15.16:0/398561464 reap_dead start 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 shutdown_connections 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 >> 172.21.15.16:0/398561464 conn(0x7f0b200febc0 msgr2=0x7f0b200fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 shutdown_connections 2024-04-05T23:32:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:34.834+0000 7f0b26600700 1 -- 172.21.15.16:0/398561464 wait complete. 2024-04-05T23:32:34.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:34 smithi130 ceph-mon[79417]: pgmap v247: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-05T23:32:35.536 INFO:teuthology.orchestra.run.smithi016.stdout:317827579920 2024-04-05T23:32:35.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.11 2024-04-05T23:32:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.296+0000 7f002e000700 1 -- 172.21.15.16:0/2253753568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00281018d0 msgr2=0x7f002810ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.296+0000 7f002e000700 1 --2- 172.21.15.16:0/2253753568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00281018d0 0x7f002810ddb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f002000b0a0 tx=0x7f002000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.296+0000 7f002e000700 1 -- 172.21.15.16:0/2253753568 shutdown_connections 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.296+0000 7f002e000700 1 --2- 172.21.15.16:0/2253753568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00281018d0 0x7f002810ddb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.296+0000 7f002e000700 1 --2- 172.21.15.16:0/2253753568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0028106f70 0x7f0028107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.296+0000 7f002e000700 1 --2- 172.21.15.16:0/2253753568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028100f30 0x7f0028101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.296+0000 7f002e000700 1 -- 172.21.15.16:0/2253753568 >> 172.21.15.16:0/2253753568 conn(0x7f00280780d0 msgr2=0x7f00280784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.297+0000 7f002e000700 1 -- 172.21.15.16:0/2253753568 shutdown_connections 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.297+0000 7f002e000700 1 -- 172.21.15.16:0/2253753568 wait complete. 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.297+0000 7f002e000700 1 Processor -- start 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f002e000700 1 -- start start 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f002e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0028100f30 0x7f0028197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f002e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00281018d0 0x7f00281979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f002e000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0028106f70 0x7f002819bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f002e000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002819c380 con 0x7f00281018d0 2024-04-05T23:32:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f002e000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002819c4f0 con 0x7f0028100f30 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f002e000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002819c6b0 con 0x7f0028106f70 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f0027400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0028100f30 0x7f0028197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f0027400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0028100f30 0x7f0028197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48012/0 (socket says 172.21.15.16:48012) 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f0027400700 1 -- 172.21.15.16:0/1304829268 learned_addr learned my addr 172.21.15.16:0/1304829268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f0026a00700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00281018d0 0x7f00281979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.298+0000 7f0027e00700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0028106f70 0x7f002819bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f0027400700 1 -- 172.21.15.16:0/1304829268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0028106f70 msgr2=0x7f002819bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f0027400700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0028106f70 0x7f002819bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f0027400700 1 -- 172.21.15.16:0/1304829268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00281018d0 msgr2=0x7f00281979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f0027400700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00281018d0 0x7f00281979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f0027400700 1 -- 172.21.15.16:0/1304829268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020009d00 con 0x7f0028100f30 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f0027400700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0028100f30 0x7f0028197480 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f001800e9a0 tx=0x7f001800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f001fe00700 1 -- 172.21.15.16:0/1304829268 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018009790 con 0x7f0028100f30 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002819c990 con 0x7f0028100f30 2024-04-05T23:32:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f001fe00700 1 -- 172.21.15.16:0/1304829268 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0018010430 con 0x7f0028100f30 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.299+0000 7f001fe00700 1 -- 172.21.15.16:0/1304829268 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018018680 con 0x7f0028100f30 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.300+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00281a41f0 con 0x7f0028100f30 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.300+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f00281110f0 con 0x7f0028100f30 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.303+0000 7f001fe00700 1 -- 172.21.15.16:0/1304829268 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f00180105a0 con 0x7f0028100f30 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.304+0000 7f001fe00700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f000c0a11b0 0x7f000c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.304+0000 7f001fe00700 1 -- 172.21.15.16:0/1304829268 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0018014070 con 0x7f0028100f30 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.304+0000 7f0026a00700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f000c0a11b0 0x7f000c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.304+0000 7f001fe00700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] conn(0x7f000c0aa990 0x7f000c0acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.304+0000 7f001fe00700 1 -- 172.21.15.16:0/1304829268 --> [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f000c0ad460 con 0x7f000c0aa990 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.304+0000 7f001fe00700 1 -- 172.21.15.16:0/1304829268 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00180cd120 con 0x7f0028100f30 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.305+0000 7f0027e00700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] conn(0x7f000c0aa990 0x7f000c0acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.305+0000 7f0026a00700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f000c0a11b0 0x7f000c0a3670 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f00281989e0 tx=0x7f0014009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.305+0000 7f0027e00700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] conn(0x7f000c0aa990 0x7f000c0acdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.306+0000 7f001fe00700 1 -- 172.21.15.16:0/1304829268 <== osd.5 v2:172.21.15.130:6808/1595822481 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f000c0ad460 con 0x7f000c0aa990 2024-04-05T23:32:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.322+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 --> [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0028102620 con 0x7f000c0aa990 2024-04-05T23:32:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f001fe00700 1 -- 172.21.15.16:0/1304829268 <== osd.5 v2:172.21.15.130:6808/1595822481 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0028102620 con 0x7f000c0aa990 2024-04-05T23:32:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 >> [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] conn(0x7f000c0aa990 msgr2=0x7f000c0acdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] conn(0x7f000c0aa990 0x7f000c0acdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f000c0a11b0 msgr2=0x7f000c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f000c0a11b0 0x7f000c0a3670 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f00281989e0 tx=0x7f0014009260 comp rx=0 tx=0).stop 2024-04-05T23:32:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0028100f30 msgr2=0x7f0028197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 --2- 172.21.15.16:0/1304829268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0028100f30 0x7f0028197480 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f001800e9a0 tx=0x7f001800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f0027e00700 1 -- 172.21.15.16:0/1304829268 reap_dead start 2024-04-05T23:32:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 shutdown_connections 2024-04-05T23:32:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 >> 172.21.15.16:0/1304829268 conn(0x7f00280780d0 msgr2=0x7f00280ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 shutdown_connections 2024-04-05T23:32:35.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.323+0000 7f002e000700 1 -- 172.21.15.16:0/1304829268 wait complete. 2024-04-05T23:32:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.638+0000 7fb73a600700 1 -- 172.21.15.16:0/3301524743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734106f70 msgr2=0x7fb734107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.638+0000 7fb73a600700 1 --2- 172.21.15.16:0/3301524743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734106f70 0x7fb734107350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb724009990 tx=0x7fb724009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.638+0000 7fb73a600700 1 -- 172.21.15.16:0/3301524743 shutdown_connections 2024-04-05T23:32:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.638+0000 7fb73a600700 1 --2- 172.21.15.16:0/3301524743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7341018f0 0x7fb73410dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.638+0000 7fb73a600700 1 --2- 172.21.15.16:0/3301524743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734106f70 0x7fb734107350 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.638+0000 7fb73a600700 1 --2- 172.21.15.16:0/3301524743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb734100f30 0x7fb7341013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.638+0000 7fb73a600700 1 -- 172.21.15.16:0/3301524743 >> 172.21.15.16:0/3301524743 conn(0x7fb734078040 msgr2=0x7fb734078450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.638+0000 7fb73a600700 1 -- 172.21.15.16:0/3301524743 shutdown_connections 2024-04-05T23:32:35.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.638+0000 7fb73a600700 1 -- 172.21.15.16:0/3301524743 wait complete. 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.640+0000 7fb73a600700 1 Processor -- start 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.640+0000 7fb73a600700 1 -- start start 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.640+0000 7fb73a600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb734100f30 0x7fb7341994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.640+0000 7fb73a600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7341018f0 0x7fb7341999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.640+0000 7fb73a600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734106f70 0x7fb73419dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.640+0000 7fb73a600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73419e3a0 con 0x7fb7341018f0 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.640+0000 7fb73a600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73419e510 con 0x7fb734106f70 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb73a600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73419e6d0 con 0x7fb734100f30 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb732a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7341018f0 0x7fb7341999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb732a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7341018f0 0x7fb7341999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39476/0 (socket says 172.21.15.16:39476) 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb732a00700 1 -- 172.21.15.16:0/1268979799 learned_addr learned my addr 172.21.15.16:0/1268979799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb733400700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb734100f30 0x7fb7341994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb733e00700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734106f70 0x7fb73419dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb733400700 1 -- 172.21.15.16:0/1268979799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734106f70 msgr2=0x7fb73419dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb733400700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734106f70 0x7fb73419dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb733400700 1 -- 172.21.15.16:0/1268979799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7341018f0 msgr2=0x7fb7341999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb733400700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7341018f0 0x7fb7341999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb733400700 1 -- 172.21.15.16:0/1268979799 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7240095f0 con 0x7fb734100f30 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.641+0000 7fb732a00700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7341018f0 0x7fb7341999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.642+0000 7fb733400700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb734100f30 0x7fb7341994a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb7240098e0 tx=0x7fb724005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.642+0000 7fb72be00700 1 -- 172.21.15.16:0/1268979799 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72401d070 con 0x7fb734100f30 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.642+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73419e950 con 0x7fb734100f30 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.642+0000 7fb72be00700 1 -- 172.21.15.16:0/1268979799 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb724022470 con 0x7fb734100f30 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.642+0000 7fb72be00700 1 -- 172.21.15.16:0/1268979799 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724017650 con 0x7fb734100f30 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.642+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7341a6210 con 0x7fb734100f30 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.643+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb7341110d0 con 0x7fb734100f30 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.646+0000 7fb72be00700 1 -- 172.21.15.16:0/1268979799 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fb7240225e0 con 0x7fb734100f30 2024-04-05T23:32:35.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.647+0000 7fb72be00700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7180a1230 0x7fb7180a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.647+0000 7fb72be00700 1 -- 172.21.15.16:0/1268979799 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb7240ce570 con 0x7fb734100f30 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.647+0000 7fb732a00700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7180a1230 0x7fb7180a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.648+0000 7fb72be00700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] conn(0x7fb7180aaa00 0x7fb7180ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.648+0000 7fb72be00700 1 -- 172.21.15.16:0/1268979799 --> [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7180ad4d0 con 0x7fb7180aaa00 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.648+0000 7fb72be00700 1 -- 172.21.15.16:0/1268979799 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7240ce920 con 0x7fb734100f30 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.648+0000 7fb733e00700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] conn(0x7fb7180aaa00 0x7fb7180ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.648+0000 7fb732a00700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7180a1230 0x7fb7180a36f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb73419aa00 tx=0x7fb720009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.648+0000 7fb733e00700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] conn(0x7fb7180aaa00 0x7fb7180ace20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.649+0000 7fb72be00700 1 -- 172.21.15.16:0/1268979799 <== osd.8 v2:172.21.15.130:6816/3206875138 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb7180ad4d0 con 0x7fb7180aaa00 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.684+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 --> [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb7341025b0 con 0x7fb7180aaa00 2024-04-05T23:32:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb72be00700 1 -- 172.21.15.16:0/1268979799 <== osd.8 v2:172.21.15.130:6816/3206875138 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb7341025b0 con 0x7fb7180aaa00 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 >> [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] conn(0x7fb7180aaa00 msgr2=0x7fb7180ace20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] conn(0x7fb7180aaa00 0x7fb7180ace20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7180a1230 msgr2=0x7fb7180a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7180a1230 0x7fb7180a36f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb73419aa00 tx=0x7fb720009260 comp rx=0 tx=0).stop 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb734100f30 msgr2=0x7fb7341994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 --2- 172.21.15.16:0/1268979799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb734100f30 0x7fb7341994a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb7240098e0 tx=0x7fb724005740 comp rx=0 tx=0).stop 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb733e00700 1 -- 172.21.15.16:0/1268979799 reap_dead start 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 shutdown_connections 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 >> 172.21.15.16:0/1268979799 conn(0x7fb734078040 msgr2=0x7fb73410c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 shutdown_connections 2024-04-05T23:32:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:35.685+0000 7fb73a600700 1 -- 172.21.15.16:0/1268979799 wait complete. 2024-04-05T23:32:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:36.336 INFO:teuthology.orchestra.run.smithi016.stdout:34359738427 2024-04-05T23:32:36.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-05T23:32:36.342 INFO:teuthology.orchestra.run.smithi016.stdout:98784247855 2024-04-05T23:32:36.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-05T23:32:36.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:36 smithi016 ceph-mon[38248]: pgmap v248: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-05T23:32:36.482 INFO:teuthology.orchestra.run.smithi016.stdout:236223201309 2024-04-05T23:32:36.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.8 2024-04-05T23:32:36.490 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:36.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:36 smithi027 ceph-mon[78597]: pgmap v248: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-05T23:32:36.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:36 smithi130 ceph-mon[79417]: pgmap v248: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-05T23:32:37.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-05T23:32:37.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-05T23:32:37.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-05T23:32:37.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-05T23:32:37.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-05T23:32:37.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-05T23:32:37.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-05T23:32:37.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:37 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-04-05T23:32:37.669 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:37.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-05T23:32:37.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-05T23:32:37.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-05T23:32:37.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-05T23:32:37.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-05T23:32:37.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-05T23:32:37.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-05T23:32:37.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:37 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-04-05T23:32:37.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-05T23:32:37.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-05T23:32:37.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-05T23:32:37.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-05T23:32:37.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-05T23:32:37.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-05T23:32:37.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-05T23:32:37.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:37 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-04-05T23:32:38.208 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:38.209 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:38.209 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:38.209 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:38.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.666+0000 7ffbe8200700 1 -- 172.21.15.16:0/2368884514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbe0106f70 msgr2=0x7ffbe0107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.666+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2368884514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbe0106f70 0x7ffbe0107350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4009990 tx=0x7ffbd4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.667+0000 7ffbe8200700 1 -- 172.21.15.16:0/2368884514 shutdown_connections 2024-04-05T23:32:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.667+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2368884514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe01018d0 0x7ffbe010ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.667+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2368884514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbe0106f70 0x7ffbe0107350 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.667+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2368884514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe0100f30 0x7ffbe0101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.667+0000 7ffbe8200700 1 -- 172.21.15.16:0/2368884514 >> 172.21.15.16:0/2368884514 conn(0x7ffbe00780d0 msgr2=0x7ffbe00784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.667+0000 7ffbe8200700 1 -- 172.21.15.16:0/2368884514 shutdown_connections 2024-04-05T23:32:38.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.667+0000 7ffbe8200700 1 -- 172.21.15.16:0/2368884514 wait complete. 2024-04-05T23:32:38.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.669+0000 7ffbe8200700 1 Processor -- start 2024-04-05T23:32:38.662 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:38.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.669+0000 7ffbe8200700 1 -- start start 2024-04-05T23:32:38.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.670+0000 7ffbe8200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe0100f30 0x7ffbe010af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:38.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.671+0000 7ffbe8200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe01018d0 0x7ffbe010b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:38.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.671+0000 7ffbe8200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbe0106f70 0x7ffbe01a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:38.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.671+0000 7ffbe8200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe01a3990 con 0x7ffbe01018d0 2024-04-05T23:32:38.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.671+0000 7ffbe8200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe01a3b00 con 0x7ffbe0100f30 2024-04-05T23:32:38.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.671+0000 7ffbe8200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe01a3cc0 con 0x7ffbe0106f70 2024-04-05T23:32:38.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.671+0000 7ffbe4e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe01018d0 0x7ffbe010b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:38.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.672+0000 7ffbe5800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe0100f30 0x7ffbe010af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:38.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.672+0000 7ffbe5800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe0100f30 0x7ffbe010af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48050/0 (socket says 172.21.15.16:48050) 2024-04-05T23:32:38.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.672+0000 7ffbe5800700 1 -- 172.21.15.16:0/2479393234 learned_addr learned my addr 172.21.15.16:0/2479393234 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:38.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.672+0000 7ffbe6200700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbe0106f70 0x7ffbe01a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:38.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.672+0000 7ffbe5800700 1 -- 172.21.15.16:0/2479393234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbe0106f70 msgr2=0x7ffbe01a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:38.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.672+0000 7ffbe5800700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbe0106f70 0x7ffbe01a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.672+0000 7ffbe5800700 1 -- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe01018d0 msgr2=0x7ffbe010b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:38.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.673+0000 7ffbe5800700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe01018d0 0x7ffbe010b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.673+0000 7ffbe5800700 1 -- 172.21.15.16:0/2479393234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc009520 con 0x7ffbe0100f30 2024-04-05T23:32:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.673+0000 7ffbe4e00700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe01018d0 0x7ffbe010b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.673+0000 7ffbe5800700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe0100f30 0x7ffbe010af70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4005770 tx=0x7ffbd40057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.674+0000 7ffbde000700 1 -- 172.21.15.16:0/2479393234 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd401d070 con 0x7ffbe0100f30 2024-04-05T23:32:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.674+0000 7ffbde000700 1 -- 172.21.15.16:0/2479393234 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffbd4021410 con 0x7ffbe0100f30 2024-04-05T23:32:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.674+0000 7ffbde000700 1 -- 172.21.15.16:0/2479393234 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4017650 con 0x7ffbe0100f30 2024-04-05T23:32:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.675+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd40095f0 con 0x7ffbe0100f30 2024-04-05T23:32:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.675+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe01aa430 con 0x7ffbe0100f30 2024-04-05T23:32:38.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.676+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe01042b0 con 0x7ffbe0100f30 2024-04-05T23:32:38.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.680+0000 7ffbde000700 1 -- 172.21.15.16:0/2479393234 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7ffbd40217f0 con 0x7ffbe0100f30 2024-04-05T23:32:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.682+0000 7ffbde000700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffbc80a12b0 0x7ffbc80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.683+0000 7ffbde000700 1 -- 172.21.15.16:0/2479393234 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ffbd40cdb40 con 0x7ffbe0100f30 2024-04-05T23:32:38.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.683+0000 7ffbe4e00700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffbc80a12b0 0x7ffbc80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:38.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.684+0000 7ffbe4e00700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffbc80a12b0 0x7ffbc80a3770 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009fd0 tx=0x7ffbcc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:38.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.686+0000 7ffbde000700 1 -- 172.21.15.16:0/2479393234 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ffbd40971b0 con 0x7ffbe0100f30 2024-04-05T23:32:38.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: pgmap v249: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-05T23:32:38.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-05T23:32:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-05T23:32:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-05T23:32:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-05T23:32:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-05T23:32:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-05T23:32:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-05T23:32:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-04-05T23:32:38.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:38 smithi027 ceph-mon[78597]: osdmap e80: 12 total, 12 up, 12 in 2024-04-05T23:32:38.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.790+0000 7fceb0e00700 1 -- 172.21.15.16:0/924355297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceac106f70 msgr2=0x7fceac107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:38.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.790+0000 7fceb0e00700 1 --2- 172.21.15.16:0/924355297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceac106f70 0x7fceac107350 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0098f0 tx=0x7fce9c009c00 comp rx=0 tx=0).stop 2024-04-05T23:32:38.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.790+0000 7fceb0e00700 1 -- 172.21.15.16:0/924355297 shutdown_connections 2024-04-05T23:32:38.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.790+0000 7fceb0e00700 1 --2- 172.21.15.16:0/924355297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceac106f70 0x7fceac107350 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.790+0000 7fceb0e00700 1 --2- 172.21.15.16:0/924355297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac1018d0 0x7fceac10ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.790+0000 7fceb0e00700 1 --2- 172.21.15.16:0/924355297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fceac100f30 0x7fceac101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.790+0000 7fceb0e00700 1 -- 172.21.15.16:0/924355297 >> 172.21.15.16:0/924355297 conn(0x7fceac0780d0 msgr2=0x7fceac0784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.791+0000 7fceb0e00700 1 -- 172.21.15.16:0/924355297 shutdown_connections 2024-04-05T23:32:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.791+0000 7fceb0e00700 1 -- 172.21.15.16:0/924355297 wait complete. 2024-04-05T23:32:38.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.792+0000 7fceb0e00700 1 Processor -- start 2024-04-05T23:32:38.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.792+0000 7fceb0e00700 1 -- start start 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fceb0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac100f30 0x7fceac0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fceb0e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceac1018d0 0x7fceac073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fceb0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fceac106f70 0x7fceac073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fceb0e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac075150 con 0x7fceac1018d0 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fceb0e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac1a9d90 con 0x7fceac100f30 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fceb0e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac1a9f00 con 0x7fceac106f70 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fcea9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac100f30 0x7fceac0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fcea9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac100f30 0x7fceac0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48062/0 (socket says 172.21.15.16:48062) 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fceaa800700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fceac106f70 0x7fceac073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.793+0000 7fcea9e00700 1 -- 172.21.15.16:0/1028083538 learned_addr learned my addr 172.21.15.16:0/1028083538 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.794+0000 7fcea9400700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceac1018d0 0x7fceac073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.794+0000 7fceaa800700 1 -- 172.21.15.16:0/1028083538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac100f30 msgr2=0x7fceac0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:38.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.794+0000 7fceaa800700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac100f30 0x7fceac0731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.794+0000 7fceaa800700 1 -- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceac1018d0 msgr2=0x7fceac073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.794+0000 7fceaa800700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceac1018d0 0x7fceac073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.794+0000 7fceaa800700 1 -- 172.21.15.16:0/1028083538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c009520 con 0x7fceac106f70 2024-04-05T23:32:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.794+0000 7fcea9400700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceac1018d0 0x7fceac073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:32:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.794+0000 7fceaa800700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fceac106f70 0x7fceac073c60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcea400efc0 tx=0x7fcea400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.794+0000 7fcea2a00700 1 -- 172.21.15.16:0/1028083538 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4018070 con 0x7fceac106f70 2024-04-05T23:32:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.795+0000 7fcea2a00700 1 -- 172.21.15.16:0/1028083538 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcea400f040 con 0x7fceac106f70 2024-04-05T23:32:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.795+0000 7fcea2a00700 1 -- 172.21.15.16:0/1028083538 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea400cbf0 con 0x7fceac106f70 2024-04-05T23:32:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.795+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceac1aa180 con 0x7fceac106f70 2024-04-05T23:32:38.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.795+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceac1aa630 con 0x7fceac106f70 2024-04-05T23:32:38.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.796+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceac109490 con 0x7fceac106f70 2024-04-05T23:32:38.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.801+0000 7fcea2a00700 1 -- 172.21.15.16:0/1028083538 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fcea400cd50 con 0x7fceac106f70 2024-04-05T23:32:38.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.803+0000 7fcea2a00700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce900a1230 0x7fce900a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:38.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.803+0000 7fcea2a00700 1 -- 172.21.15.16:0/1028083538 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcea40cceb0 con 0x7fceac106f70 2024-04-05T23:32:38.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.803+0000 7fcea9e00700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce900a1230 0x7fce900a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:38.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.803+0000 7fcea2a00700 1 -- 172.21.15.16:0/1028083538 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcea40cd2f0 con 0x7fceac106f70 2024-04-05T23:32:38.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.804+0000 7fcea9e00700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce900a1230 0x7fce900a36f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c010 tx=0x7fce9c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:38.810 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: pgmap v249: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-05T23:32:38.810 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-05T23:32:38.810 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-05T23:32:38.810 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-05T23:32:38.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-05T23:32:38.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-05T23:32:38.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-05T23:32:38.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-05T23:32:38.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-04-05T23:32:38.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:38 smithi016 ceph-mon[38248]: osdmap e80: 12 total, 12 up, 12 in 2024-04-05T23:32:38.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: pgmap v249: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-05T23:32:38.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-05T23:32:38.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-05T23:32:38.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-05T23:32:38.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-05T23:32:38.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-05T23:32:38.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-05T23:32:38.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-05T23:32:38.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-04-05T23:32:38.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:38 smithi130 ceph-mon[79417]: osdmap e80: 12 total, 12 up, 12 in 2024-04-05T23:32:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.926+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7ffbe0102620 con 0x7ffbe0100f30 2024-04-05T23:32:38.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.927+0000 7ffbde000700 1 -- 172.21.15.16:0/2479393234 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7ffbd409a7d0 con 0x7ffbe0100f30 2024-04-05T23:32:38.920 INFO:teuthology.orchestra.run.smithi016.stdout:287762808853 2024-04-05T23:32:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffbc80a12b0 msgr2=0x7ffbc80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffbc80a12b0 0x7ffbc80a3770 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009fd0 tx=0x7ffbcc009260 comp rx=0 tx=0).stop 2024-04-05T23:32:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe0100f30 msgr2=0x7ffbe010af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe0100f30 0x7ffbe010af70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4005770 tx=0x7ffbd40057a0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 shutdown_connections 2024-04-05T23:32:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffbc80a12b0 0x7ffbc80a3770 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbe0106f70 0x7ffbe01a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe0100f30 0x7ffbe010af70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 --2- 172.21.15.16:0/2479393234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe01018d0 0x7ffbe010b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 >> 172.21.15.16:0/2479393234 conn(0x7ffbe00780d0 msgr2=0x7ffbe010c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 shutdown_connections 2024-04-05T23:32:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:38.929+0000 7ffbe8200700 1 -- 172.21.15.16:0/2479393234 wait complete. 2024-04-05T23:32:39.036 INFO:teuthology.orchestra.run.smithi016.stdout:180388626468 2024-04-05T23:32:39.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.042+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fceac04ebb0 con 0x7fceac106f70 2024-04-05T23:32:39.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.043+0000 7fcea2a00700 1 -- 172.21.15.16:0/1028083538 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fcea4096520 con 0x7fceac106f70 2024-04-05T23:32:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.045+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce900a1230 msgr2=0x7fce900a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.045+0000 7fceb0e00700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce900a1230 0x7fce900a36f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c010 tx=0x7fce9c019040 comp rx=0 tx=0).stop 2024-04-05T23:32:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.045+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fceac106f70 msgr2=0x7fceac073c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.045+0000 7fceb0e00700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fceac106f70 0x7fceac073c60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcea400efc0 tx=0x7fcea400c3a0 comp rx=0 tx=0).stop 2024-04-05T23:32:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.047+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 shutdown_connections 2024-04-05T23:32:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.047+0000 7fceb0e00700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fce900a1230 0x7fce900a36f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.047+0000 7fceb0e00700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fceac106f70 0x7fceac073c60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.047+0000 7fceb0e00700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac100f30 0x7fceac0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.047+0000 7fceb0e00700 1 --2- 172.21.15.16:0/1028083538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceac1018d0 0x7fceac073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.047+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 >> 172.21.15.16:0/1028083538 conn(0x7fceac0780d0 msgr2=0x7fceac0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.047+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 shutdown_connections 2024-04-05T23:32:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:39.047+0000 7fceb0e00700 1 -- 172.21.15.16:0/1028083538 wait complete. 2024-04-05T23:32:39.050 INFO:teuthology.orchestra.run.smithi016.stdout:150323855398 2024-04-05T23:32:39.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-05T23:32:39.443 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808851 got 287762808853 for osd.10 2024-04-05T23:32:39.444 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:39.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:39 smithi016 ceph-mon[38248]: osdmap e81: 12 total, 12 up, 12 in 2024-04-05T23:32:39.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:39 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2479393234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-05T23:32:39.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:39 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1028083538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-05T23:32:39.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:39 smithi027 ceph-mon[78597]: osdmap e81: 12 total, 12 up, 12 in 2024-04-05T23:32:39.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:39 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2479393234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-05T23:32:39.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:39 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1028083538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-05T23:32:39.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:39 smithi130 ceph-mon[79417]: osdmap e81: 12 total, 12 up, 12 in 2024-04-05T23:32:39.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:39 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2479393234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-05T23:32:39.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:39 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1028083538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-05T23:32:40.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:40 smithi016 ceph-mon[38248]: pgmap v252: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9 op/s 2024-04-05T23:32:40.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:40 smithi027 ceph-mon[78597]: pgmap v252: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9 op/s 2024-04-05T23:32:40.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:40 smithi130 ceph-mon[79417]: pgmap v252: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9 op/s 2024-04-05T23:32:40.958 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626466 got 180388626468 for osd.6 2024-04-05T23:32:40.958 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.425+0000 7f5c3dc00700 1 -- 172.21.15.16:0/2613645051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c38106f70 msgr2=0x7f5c38107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.425+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/2613645051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c38106f70 0x7f5c38107350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009990 tx=0x7f5c28009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.425+0000 7f5c3dc00700 1 -- 172.21.15.16:0/2613645051 shutdown_connections 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.425+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/2613645051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c381018d0 0x7f5c3810ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.425+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/2613645051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c38106f70 0x7f5c38107350 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.425+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/2613645051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c38100f30 0x7f5c38101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.425+0000 7f5c3dc00700 1 -- 172.21.15.16:0/2613645051 >> 172.21.15.16:0/2613645051 conn(0x7f5c380780d0 msgr2=0x7f5c380784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.425+0000 7f5c3dc00700 1 -- 172.21.15.16:0/2613645051 shutdown_connections 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.425+0000 7f5c3dc00700 1 -- 172.21.15.16:0/2613645051 wait complete. 2024-04-05T23:32:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.426+0000 7f5c3dc00700 1 Processor -- start 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.426+0000 7f5c3dc00700 1 -- start start 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c3dc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c38100f30 0x7f5c38199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c3dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c381018d0 0x7f5c38199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c3dc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c38106f70 0x7f5c3819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c3dc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3819e450 con 0x7f5c38106f70 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c3dc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3819e5c0 con 0x7f5c381018d0 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c3dc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3819e780 con 0x7f5c38100f30 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c36000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c381018d0 0x7f5c38199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c36000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c381018d0 0x7f5c38199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48078/0 (socket says 172.21.15.16:48078) 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c36000700 1 -- 172.21.15.16:0/3430590279 learned_addr learned my addr 172.21.15.16:0/3430590279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c37400700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c38106f70 0x7f5c3819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c36a00700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c38100f30 0x7f5c38199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c37400700 1 -- 172.21.15.16:0/3430590279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c38100f30 msgr2=0x7f5c38199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c37400700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c38100f30 0x7f5c38199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c37400700 1 -- 172.21.15.16:0/3430590279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c381018d0 msgr2=0x7f5c38199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c37400700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c381018d0 0x7f5c38199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c37400700 1 -- 172.21.15.16:0/3430590279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c280095f0 con 0x7f5c38106f70 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.427+0000 7f5c36000700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c381018d0 0x7f5c38199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.428+0000 7f5c37400700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c38106f70 0x7f5c3819de40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000efc0 tx=0x7f5c3000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.428+0000 7f5c2f400700 1 -- 172.21.15.16:0/3430590279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30018070 con 0x7f5c38106f70 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.428+0000 7f5c2f400700 1 -- 172.21.15.16:0/3430590279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c3000f040 con 0x7f5c38106f70 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.428+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3819ea00 con 0x7f5c38106f70 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.428+0000 7f5c2f400700 1 -- 172.21.15.16:0/3430590279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3000cc80 con 0x7f5c38106f70 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.428+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c381a62c0 con 0x7f5c38106f70 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.429+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c381042b0 con 0x7f5c38106f70 2024-04-05T23:32:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.431+0000 7f5c2f400700 1 -- 172.21.15.16:0/3430590279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f5c30003a00 con 0x7f5c38106f70 2024-04-05T23:32:41.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.433+0000 7f5c2f400700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5c1c0a11b0 0x7f5c1c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:41.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.433+0000 7f5c2f400700 1 -- 172.21.15.16:0/3430590279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5c300cc510 con 0x7f5c38106f70 2024-04-05T23:32:41.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.433+0000 7f5c36a00700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5c1c0a11b0 0x7f5c1c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:41.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.434+0000 7f5c36a00700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5c1c0a11b0 0x7f5c1c0a3670 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5c28006010 tx=0x7f5c2800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:41.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.438+0000 7f5c2f400700 1 -- 172.21.15.16:0/3430590279 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5c30095be0 con 0x7f5c38106f70 2024-04-05T23:32:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.659+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5c38102620 con 0x7f5c38106f70 2024-04-05T23:32:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.659+0000 7f5c2f400700 1 -- 172.21.15.16:0/3430590279 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5c30099200 con 0x7f5c38106f70 2024-04-05T23:32:41.652 INFO:teuthology.orchestra.run.smithi016.stdout:73014444085 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5c1c0a11b0 msgr2=0x7f5c1c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5c1c0a11b0 0x7f5c1c0a3670 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5c28006010 tx=0x7f5c2800b260 comp rx=0 tx=0).stop 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c38106f70 msgr2=0x7f5c3819de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c38106f70 0x7f5c3819de40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000efc0 tx=0x7f5c3000c3a0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 shutdown_connections 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5c1c0a11b0 0x7f5c1c0a3670 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c38106f70 0x7f5c3819de40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c38100f30 0x7f5c38199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 --2- 172.21.15.16:0/3430590279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c381018d0 0x7f5c38199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 >> 172.21.15.16:0/3430590279 conn(0x7f5c380780d0 msgr2=0x7f5c3810c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:41.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 shutdown_connections 2024-04-05T23:32:41.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:41.662+0000 7f5c3dc00700 1 -- 172.21.15.16:0/3430590279 wait complete. 2024-04-05T23:32:42.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.606+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3327224587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38068890 msgr2=0x7fbb38068cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.606+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3327224587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38068890 0x7fbb38068cf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbb240098f0 tx=0x7fbb24009c00 comp rx=0 tx=0).stop 2024-04-05T23:32:42.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.606+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3327224587 shutdown_connections 2024-04-05T23:32:42.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.606+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3327224587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb38106f60 0x7fbb38107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.606+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3327224587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb38069230 0x7fbb3810dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.606+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3327224587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38068890 0x7fbb38068cf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.606+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3327224587 >> 172.21.15.16:0/3327224587 conn(0x7fbb38075ec0 msgr2=0x7fbb380762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:42.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.606+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3327224587 shutdown_connections 2024-04-05T23:32:42.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.606+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3327224587 wait complete. 2024-04-05T23:32:42.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.607+0000 7fbb3ea00700 1 Processor -- start 2024-04-05T23:32:42.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ea00700 1 -- start start 2024-04-05T23:32:42.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb38068890 0x7fbb381995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb38069230 0x7fbb38199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38106f60 0x7fbb3819dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ea00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3819e4f0 con 0x7fbb38068890 2024-04-05T23:32:42.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3819e660 con 0x7fbb38106f60 2024-04-05T23:32:42.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ea00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3819e820 con 0x7fbb38069230 2024-04-05T23:32:42.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38106f60 0x7fbb3819dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb37e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb38068890 0x7fbb381995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38106f60 0x7fbb3819dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33708/0 (socket says 172.21.15.16:33708) 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb3ca00700 1 -- 172.21.15.16:0/3835693799 learned_addr learned my addr 172.21.15.16:0/3835693799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.608+0000 7fbb37400700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb38069230 0x7fbb38199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb3ca00700 1 -- 172.21.15.16:0/3835693799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb38069230 msgr2=0x7fbb38199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb3ca00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb38069230 0x7fbb38199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb3ca00700 1 -- 172.21.15.16:0/3835693799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb38068890 msgr2=0x7fbb381995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb3ca00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb38068890 0x7fbb381995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb3ca00700 1 -- 172.21.15.16:0/3835693799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb280095f0 con 0x7fbb38106f60 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb3ca00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38106f60 0x7fbb3819dee0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000ef30 tx=0x7fbb3000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb34c00700 1 -- 172.21.15.16:0/3835693799 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30018070 con 0x7fbb38106f60 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb34c00700 1 -- 172.21.15.16:0/3835693799 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb3000f040 con 0x7fbb38106f60 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb24009520 con 0x7fbb38106f60 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.609+0000 7fbb34c00700 1 -- 172.21.15.16:0/3835693799 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3000cd00 con 0x7fbb38106f60 2024-04-05T23:32:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.610+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb381a6190 con 0x7fbb38106f60 2024-04-05T23:32:42.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.611+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb38102610 con 0x7fbb38106f60 2024-04-05T23:32:42.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.613+0000 7fbb34c00700 1 -- 172.21.15.16:0/3835693799 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fbb30003a00 con 0x7fbb38106f60 2024-04-05T23:32:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.614+0000 7fbb34c00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb1c0a11b0 0x7fbb1c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.614+0000 7fbb34c00700 1 -- 172.21.15.16:0/3835693799 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbb300cc720 con 0x7fbb38106f60 2024-04-05T23:32:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.614+0000 7fbb37e00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb1c0a11b0 0x7fbb1c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.615+0000 7fbb37e00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb1c0a11b0 0x7fbb1c0a3670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbb28005fd0 tx=0x7fbb28009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:42.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.616+0000 7fbb34c00700 1 -- 172.21.15.16:0/3835693799 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbb30095df0 con 0x7fbb38106f60 2024-04-05T23:32:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:42 smithi027 ceph-mon[78597]: pgmap v253: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:42.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:42 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3430590279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-05T23:32:42.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:42 smithi016 ceph-mon[38248]: pgmap v253: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:42.721 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:42 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3430590279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-05T23:32:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:42 smithi130 ceph-mon[79417]: pgmap v253: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:42.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:42 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3430590279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-05T23:32:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.870+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fbb3819a900 con 0x7fbb38106f60 2024-04-05T23:32:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.870+0000 7fbb34c00700 1 -- 172.21.15.16:0/3835693799 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fbb30099410 con 0x7fbb38106f60 2024-04-05T23:32:42.864 INFO:teuthology.orchestra.run.smithi016.stdout:266287972377 2024-04-05T23:32:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb1c0a11b0 msgr2=0x7fbb1c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb1c0a11b0 0x7fbb1c0a3670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbb28005fd0 tx=0x7fbb28009310 comp rx=0 tx=0).stop 2024-04-05T23:32:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38106f60 msgr2=0x7fbb3819dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38106f60 0x7fbb3819dee0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000ef30 tx=0x7fbb3000c3a0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 shutdown_connections 2024-04-05T23:32:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbb1c0a11b0 0x7fbb1c0a3670 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb38068890 0x7fbb381995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb38106f60 0x7fbb3819dee0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 --2- 172.21.15.16:0/3835693799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb38069230 0x7fbb38199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 >> 172.21.15.16:0/3835693799 conn(0x7fbb38075ec0 msgr2=0x7fbb3810c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:42.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 shutdown_connections 2024-04-05T23:32:42.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.873+0000 7fbb3ea00700 1 -- 172.21.15.16:0/3835693799 wait complete. 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.496+0000 7f63bdc00700 1 -- 172.21.15.16:0/1646061858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b8100f30 msgr2=0x7f63b8101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.496+0000 7f63bdc00700 1 --2- 172.21.15.16:0/1646061858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b8100f30 0x7f63b8101390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009990 tx=0x7f63a4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.496+0000 7f63bdc00700 1 -- 172.21.15.16:0/1646061858 shutdown_connections 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.496+0000 7f63bdc00700 1 --2- 172.21.15.16:0/1646061858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63b8106f70 0x7f63b8107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.496+0000 7f63bdc00700 1 --2- 172.21.15.16:0/1646061858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b81018d0 0x7f63b810ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.496+0000 7f63bdc00700 1 --2- 172.21.15.16:0/1646061858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b8100f30 0x7f63b8101390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.496+0000 7f63bdc00700 1 -- 172.21.15.16:0/1646061858 >> 172.21.15.16:0/1646061858 conn(0x7f63b80780d0 msgr2=0x7f63b80784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.496+0000 7f63bdc00700 1 -- 172.21.15.16:0/1646061858 shutdown_connections 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.496+0000 7f63bdc00700 1 -- 172.21.15.16:0/1646061858 wait complete. 2024-04-05T23:32:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.497+0000 7f63bdc00700 1 Processor -- start 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.497+0000 7f63bdc00700 1 -- start start 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63bdc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b8100f30 0x7f63b8199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63bdc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63b81018d0 0x7f63b8199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63bdc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b8106f70 0x7f63b819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63bdc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b819e410 con 0x7f63b81018d0 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63bdc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b819e580 con 0x7f63b8100f30 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63bdc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b819e740 con 0x7f63b8106f70 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63b81018d0 0x7f63b8199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63b81018d0 0x7f63b8199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40586/0 (socket says 172.21.15.16:40586) 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b6000700 1 -- 172.21.15.16:0/23604738 learned_addr learned my addr 172.21.15.16:0/23604738 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b7400700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b8106f70 0x7f63b819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b6a00700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b8100f30 0x7f63b8199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b6000700 1 -- 172.21.15.16:0/23604738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b8106f70 msgr2=0x7f63b819de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b6000700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b8106f70 0x7f63b819de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b6000700 1 -- 172.21.15.16:0/23604738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b8100f30 msgr2=0x7f63b8199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b6000700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b8100f30 0x7f63b8199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.498+0000 7f63b6000700 1 -- 172.21.15.16:0/23604738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40095f0 con 0x7f63b81018d0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.499+0000 7f63b6000700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63b81018d0 0x7f63b8199aa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f63a4005770 tx=0x7f63a40057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.499+0000 7f63af400700 1 -- 172.21.15.16:0/23604738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a401d070 con 0x7f63b81018d0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.499+0000 7f63af400700 1 -- 172.21.15.16:0/23604738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63a4021410 con 0x7f63b81018d0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.499+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b819e9c0 con 0x7f63b81018d0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.499+0000 7f63af400700 1 -- 172.21.15.16:0/23604738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4017650 con 0x7f63b81018d0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.499+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b81a6280 con 0x7f63b81018d0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.501+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b8102620 con 0x7f63b81018d0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.504+0000 7f63af400700 1 -- 172.21.15.16:0/23604738 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f63a400fb00 con 0x7f63b81018d0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.505+0000 7f63af400700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f639c0a1230 0x7f639c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.505+0000 7f63af400700 1 -- 172.21.15.16:0/23604738 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f63a40cdc00 con 0x7f63b81018d0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.506+0000 7f63b6a00700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f639c0a1230 0x7f639c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.507+0000 7f63b6a00700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f639c0a1230 0x7f639c0a36f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009be0 tx=0x7f63a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.513+0000 7f63af400700 1 -- 172.21.15.16:0/23604738 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f63a40972d0 con 0x7f63b81018d0 2024-04-05T23:32:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.758+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f63b804ebb0 con 0x7f63b81018d0 2024-04-05T23:32:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.758+0000 7f63af400700 1 -- 172.21.15.16:0/23604738 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f63a409a8f0 con 0x7f63b81018d0 2024-04-05T23:32:42.912 INFO:teuthology.orchestra.run.smithi016.stdout:236223201310 2024-04-05T23:32:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.761+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f639c0a1230 msgr2=0x7f639c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.761+0000 7f63bdc00700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f639c0a1230 0x7f639c0a36f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009be0 tx=0x7f63a8009260 comp rx=0 tx=0).stop 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.761+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63b81018d0 msgr2=0x7f63b8199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.761+0000 7f63bdc00700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63b81018d0 0x7f63b8199aa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f63a4005770 tx=0x7f63a40057a0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.762+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 shutdown_connections 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.762+0000 7f63bdc00700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f639c0a1230 0x7f639c0a36f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.762+0000 7f63bdc00700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b8106f70 0x7f63b819de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.762+0000 7f63bdc00700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b8100f30 0x7f63b8199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.762+0000 7f63bdc00700 1 --2- 172.21.15.16:0/23604738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63b81018d0 0x7f63b8199aa0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.762+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 >> 172.21.15.16:0/23604738 conn(0x7f63b80780d0 msgr2=0x7f63b80ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.762+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 shutdown_connections 2024-04-05T23:32:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.762+0000 7f63bdc00700 1 -- 172.21.15.16:0/23604738 wait complete. 2024-04-05T23:32:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.932+0000 7fd176e00700 1 -- 172.21.15.16:0/1558437204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 msgr2=0x7fd17010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.932+0000 7fd176e00700 1 --2- 172.21.15.16:0/1558437204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 0x7fd17010dd30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd168009f90 tx=0x7fd16800b330 comp rx=0 tx=0).stop 2024-04-05T23:32:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.933+0000 7fd176e00700 1 -- 172.21.15.16:0/1558437204 shutdown_connections 2024-04-05T23:32:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.933+0000 7fd176e00700 1 --2- 172.21.15.16:0/1558437204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd170106f80 0x7fd170107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.933+0000 7fd176e00700 1 --2- 172.21.15.16:0/1558437204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 0x7fd17010dd30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.933+0000 7fd176e00700 1 --2- 172.21.15.16:0/1558437204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd170100f60 0x7fd1701013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.933+0000 7fd176e00700 1 -- 172.21.15.16:0/1558437204 >> 172.21.15.16:0/1558437204 conn(0x7fd170078040 msgr2=0x7fd170078450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.933+0000 7fd176e00700 1 -- 172.21.15.16:0/1558437204 shutdown_connections 2024-04-05T23:32:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.933+0000 7fd176e00700 1 -- 172.21.15.16:0/1558437204 wait complete. 2024-04-05T23:32:42.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.934+0000 7fd176e00700 1 Processor -- start 2024-04-05T23:32:42.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd176e00700 1 -- start start 2024-04-05T23:32:42.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd176e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd170100f60 0x7fd170199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd176e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 0x7fd170199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd176e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd170106f80 0x7fd17019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd176e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17019e0c0 con 0x7fd170100f60 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd176e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17019e230 con 0x7fd170101920 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd176e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17019e3f0 con 0x7fd170106f80 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd16f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 0x7fd170199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd16f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 0x7fd170199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33724/0 (socket says 172.21.15.16:33724) 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.935+0000 7fd16f400700 1 -- 172.21.15.16:0/1870716210 learned_addr learned my addr 172.21.15.16:0/1870716210 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.936+0000 7fd174e00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd170106f80 0x7fd17019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.936+0000 7fd16fe00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd170100f60 0x7fd170199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.936+0000 7fd16f400700 1 -- 172.21.15.16:0/1870716210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd170106f80 msgr2=0x7fd17019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.936+0000 7fd16f400700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd170106f80 0x7fd17019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.936+0000 7fd16f400700 1 -- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd170100f60 msgr2=0x7fd170199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:42.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.936+0000 7fd16f400700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd170100f60 0x7fd170199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.936+0000 7fd16f400700 1 -- 172.21.15.16:0/1870716210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c0095f0 con 0x7fd170101920 2024-04-05T23:32:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.936+0000 7fd16fe00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd170100f60 0x7fd170199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:32:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.936+0000 7fd16f400700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 0x7fd170199750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00ec10 tx=0x7fd15c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.937+0000 7fd16cc00700 1 -- 172.21.15.16:0/1870716210 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c00cd40 con 0x7fd170101920 2024-04-05T23:32:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.937+0000 7fd16cc00700 1 -- 172.21.15.16:0/1870716210 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd15c0044e0 con 0x7fd170101920 2024-04-05T23:32:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.937+0000 7fd16cc00700 1 -- 172.21.15.16:0/1870716210 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c010690 con 0x7fd170101920 2024-04-05T23:32:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.937+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd168009c40 con 0x7fd170101920 2024-04-05T23:32:42.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.937+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1701a5d60 con 0x7fd170101920 2024-04-05T23:32:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.938+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1701025c0 con 0x7fd170101920 2024-04-05T23:32:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.941+0000 7fd16cc00700 1 -- 172.21.15.16:0/1870716210 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fd15c004020 con 0x7fd170101920 2024-04-05T23:32:42.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.943+0000 7fd16cc00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd1540a1230 0x7fd1540a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.943+0000 7fd16cc00700 1 -- 172.21.15.16:0/1870716210 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd15c014070 con 0x7fd170101920 2024-04-05T23:32:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.943+0000 7fd16fe00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd1540a1230 0x7fd1540a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.944+0000 7fd16fe00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd1540a1230 0x7fd1540a36f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd160005d90 tx=0x7fd160005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:42.944+0000 7fd16cc00700 1 -- 172.21.15.16:0/1870716210 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd15c096960 con 0x7fd170101920 2024-04-05T23:32:43.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.193+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd170074290 con 0x7fd170101920 2024-04-05T23:32:43.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.194+0000 7fd16cc00700 1 -- 172.21.15.16:0/1870716210 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd15c099f80 con 0x7fd170101920 2024-04-05T23:32:43.187 INFO:teuthology.orchestra.run.smithi016.stdout:51539607608 2024-04-05T23:32:43.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.196+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd1540a1230 msgr2=0x7fd1540a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd1540a1230 0x7fd1540a36f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd160005d90 tx=0x7fd160005d00 comp rx=0 tx=0).stop 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 msgr2=0x7fd170199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 0x7fd170199750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00ec10 tx=0x7fd15c00ef20 comp rx=0 tx=0).stop 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 shutdown_connections 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd1540a1230 0x7fd1540a36f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd170100f60 0x7fd170199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd170106f80 0x7fd17019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 --2- 172.21.15.16:0/1870716210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd170101920 0x7fd170199750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 >> 172.21.15.16:0/1870716210 conn(0x7fd170078040 msgr2=0x7fd1700ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 shutdown_connections 2024-04-05T23:32:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.197+0000 7fd176e00700 1 -- 172.21.15.16:0/1870716210 wait complete. 2024-04-05T23:32:43.624 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/23604738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-05T23:32:43.624 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3835693799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-05T23:32:43.624 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1870716210' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-05T23:32:43.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:43 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/23604738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-05T23:32:43.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:43 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3835693799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-05T23:32:43.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:43 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1870716210' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-05T23:32:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.814+0000 7f3886a00700 1 -- 172.21.15.16:0/1111119999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880074f40 msgr2=0x7f3880073110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.814+0000 7f3886a00700 1 --2- 172.21.15.16:0/1111119999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880074f40 0x7f3880073110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3870009990 tx=0x7f3870009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.814+0000 7f3886a00700 1 -- 172.21.15.16:0/1111119999 shutdown_connections 2024-04-05T23:32:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.814+0000 7f3886a00700 1 --2- 172.21.15.16:0/1111119999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880074f40 0x7f3880073110 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.814+0000 7f3886a00700 1 --2- 172.21.15.16:0/1111119999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880111b70 0x7f3880113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.814+0000 7f3886a00700 1 --2- 172.21.15.16:0/1111119999 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3880073650 0x7f3880073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.814+0000 7f3886a00700 1 -- 172.21.15.16:0/1111119999 >> 172.21.15.16:0/1111119999 conn(0x7f38800780d0 msgr2=0x7f38800784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.815+0000 7f3886a00700 1 -- 172.21.15.16:0/1111119999 shutdown_connections 2024-04-05T23:32:43.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.815+0000 7f3886a00700 1 -- 172.21.15.16:0/1111119999 wait complete. 2024-04-05T23:32:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f3886a00700 1 Processor -- start 2024-04-05T23:32:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f3886a00700 1 -- start start 2024-04-05T23:32:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f3886a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880073650 0x7f38801a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f3886a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3880074f40 0x7f38801a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:43.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f3886a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880111b70 0x7f388019d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f3886a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388019ddc0 con 0x7f3880073650 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f3886a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388019df30 con 0x7f3880111b70 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f3886a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388019e0f0 con 0x7f3880074f40 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f387fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880073650 0x7f38801a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f3884a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880111b70 0x7f388019d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f387f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3880074f40 0x7f38801a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f387fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880073650 0x7f38801a3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40636/0 (socket says 172.21.15.16:40636) 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.816+0000 7f387fe00700 1 -- 172.21.15.16:0/709806393 learned_addr learned my addr 172.21.15.16:0/709806393 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387f400700 1 -- 172.21.15.16:0/709806393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880111b70 msgr2=0x7f388019d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387f400700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880111b70 0x7f388019d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387f400700 1 -- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880073650 msgr2=0x7f38801a3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387f400700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880073650 0x7f38801a3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:43.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387f400700 1 -- 172.21.15.16:0/709806393 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38700095f0 con 0x7f3880074f40 2024-04-05T23:32:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f3884a00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880111b70 0x7f388019d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387fe00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880073650 0x7f38801a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387f400700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3880074f40 0x7f38801a3b60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f386c00e9f0 tx=0x7f386c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387cc00700 1 -- 172.21.15.16:0/709806393 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c00c9c0 con 0x7f3880074f40 2024-04-05T23:32:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387cc00700 1 -- 172.21.15.16:0/709806393 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f386c010430 con 0x7f3880074f40 2024-04-05T23:32:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388019e3d0 con 0x7f3880074f40 2024-04-05T23:32:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.817+0000 7f387cc00700 1 -- 172.21.15.16:0/709806393 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c018680 con 0x7f3880074f40 2024-04-05T23:32:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.818+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38801aa610 con 0x7f3880074f40 2024-04-05T23:32:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.819+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38800747f0 con 0x7f3880074f40 2024-04-05T23:32:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.821+0000 7f387cc00700 1 -- 172.21.15.16:0/709806393 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f386c010860 con 0x7f3880074f40 2024-04-05T23:32:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.822+0000 7f387cc00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f38640a11b0 0x7f38640a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.822+0000 7f387cc00700 1 -- 172.21.15.16:0/709806393 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f386c014070 con 0x7f3880074f40 2024-04-05T23:32:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.822+0000 7f387fe00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f38640a11b0 0x7f38640a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.823+0000 7f387fe00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f38640a11b0 0x7f38640a3670 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f387000b2e0 tx=0x7f3870005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:43.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:43.824+0000 7f387cc00700 1 -- 172.21.15.16:0/709806393 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f386c096190 con 0x7f3880074f40 2024-04-05T23:32:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:43 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/23604738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-05T23:32:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:43 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3835693799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-05T23:32:43.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:43 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1870716210' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-05T23:32:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.051+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f388004ebb0 con 0x7f3880074f40 2024-04-05T23:32:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.052+0000 7f387cc00700 1 -- 172.21.15.16:0/709806393 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f386c0997b0 con 0x7f3880074f40 2024-04-05T23:32:44.045 INFO:teuthology.orchestra.run.smithi016.stdout:317827579922 2024-04-05T23:32:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f38640a11b0 msgr2=0x7f38640a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f38640a11b0 0x7f38640a3670 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f387000b2e0 tx=0x7f3870005bc0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3880074f40 msgr2=0x7f38801a3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3880074f40 0x7f38801a3b60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f386c00e9f0 tx=0x7f386c00ed00 comp rx=0 tx=0).stop 2024-04-05T23:32:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 shutdown_connections 2024-04-05T23:32:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f38640a11b0 0x7f38640a3670 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3880073650 0x7f38801a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880111b70 0x7f388019d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 --2- 172.21.15.16:0/709806393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3880074f40 0x7f38801a3b60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 >> 172.21.15.16:0/709806393 conn(0x7f38800780d0 msgr2=0x7f3880110930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 shutdown_connections 2024-04-05T23:32:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.055+0000 7f3886a00700 1 -- 172.21.15.16:0/709806393 wait complete. 2024-04-05T23:32:44.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.399+0000 7f732ec00700 1 -- 172.21.15.16:0/2791235283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f732810d9d0 msgr2=0x7f732810fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:44.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.399+0000 7f732ec00700 1 --2- 172.21.15.16:0/2791235283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f732810d9d0 0x7f732810fdc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f732000b0a0 tx=0x7f732000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.399+0000 7f732ec00700 1 -- 172.21.15.16:0/2791235283 shutdown_connections 2024-04-05T23:32:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.399+0000 7f732ec00700 1 --2- 172.21.15.16:0/2791235283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f732810d9d0 0x7f732810fdc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.399+0000 7f732ec00700 1 --2- 172.21.15.16:0/2791235283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7328101df0 0x7f73280ff250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.399+0000 7f732ec00700 1 --2- 172.21.15.16:0/2791235283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73280ff790 0x7f73280ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.399+0000 7f732ec00700 1 -- 172.21.15.16:0/2791235283 >> 172.21.15.16:0/2791235283 conn(0x7f73280761d0 msgr2=0x7f73280fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.399+0000 7f732ec00700 1 -- 172.21.15.16:0/2791235283 shutdown_connections 2024-04-05T23:32:44.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.399+0000 7f732ec00700 1 -- 172.21.15.16:0/2791235283 wait complete. 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.400+0000 7f732ec00700 1 Processor -- start 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.400+0000 7f732ec00700 1 -- start start 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732ec00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73280ff790 0x7f7328199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732ec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328101df0 0x7f7328199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732ec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f732810d9d0 0x7f732819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732ec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732819e450 con 0x7f7328101df0 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732ec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732819e5c0 con 0x7f732810d9d0 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732ec00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732819e780 con 0x7f73280ff790 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328101df0 0x7f7328199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328101df0 0x7f7328199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40654/0 (socket says 172.21.15.16:40654) 2024-04-05T23:32:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732ce00700 1 -- 172.21.15.16:0/1621103714 learned_addr learned my addr 172.21.15.16:0/1621103714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732e200700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f732810d9d0 0x7f732819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.401+0000 7f732d800700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73280ff790 0x7f7328199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f732e200700 1 -- 172.21.15.16:0/1621103714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73280ff790 msgr2=0x7f7328199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f732e200700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73280ff790 0x7f7328199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f732e200700 1 -- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328101df0 msgr2=0x7f7328199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f732e200700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328101df0 0x7f7328199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f732e200700 1 -- 172.21.15.16:0/1621103714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7314009520 con 0x7f732810d9d0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f732ce00700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328101df0 0x7f7328199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f732e200700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f732810d9d0 0x7f732819de40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f732000b0a0 tx=0x7f73200078c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f7326000700 1 -- 172.21.15.16:0/1621103714 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732000e040 con 0x7f732810d9d0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f7326000700 1 -- 172.21.15.16:0/1621103714 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f732001d470 con 0x7f732810d9d0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f7326000700 1 -- 172.21.15.16:0/1621103714 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732001b6a0 con 0x7f732810d9d0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c009520 con 0x7f732810d9d0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.402+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7320009d00 con 0x7f732810d9d0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.403+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732819ecc0 con 0x7f732810d9d0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.405+0000 7f7326000700 1 -- 172.21.15.16:0/1621103714 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f732001d5e0 con 0x7f732810d9d0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.406+0000 7f7326000700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73100a11f0 0x7f73100a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.407+0000 7f7326000700 1 -- 172.21.15.16:0/1621103714 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f73200cda10 con 0x7f732810d9d0 2024-04-05T23:32:44.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.407+0000 7f732d800700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73100a11f0 0x7f73100a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.407+0000 7f732d800700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73100a11f0 0x7f73100a36b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7328068860 tx=0x7f731c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.411+0000 7f7326000700 1 -- 172.21.15.16:0/1621103714 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7320097080 con 0x7f732810d9d0 2024-04-05T23:32:44.516 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:44 smithi016 ceph-mon[38248]: pgmap v254: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:44.517 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:44 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/709806393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-05T23:32:44.520 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201309 got 236223201310 for osd.8 2024-04-05T23:32:44.520 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.647+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7328100260 con 0x7f732810d9d0 2024-04-05T23:32:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.648+0000 7f7326000700 1 -- 172.21.15.16:0/1621103714 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f732009a6a0 con 0x7f732810d9d0 2024-04-05T23:32:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:98784247856 2024-04-05T23:32:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73100a11f0 msgr2=0x7f73100a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73100a11f0 0x7f73100a36b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7328068860 tx=0x7f731c009260 comp rx=0 tx=0).stop 2024-04-05T23:32:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f732810d9d0 msgr2=0x7f732819de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f732810d9d0 0x7f732819de40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f732000b0a0 tx=0x7f73200078c0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 shutdown_connections 2024-04-05T23:32:44.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73100a11f0 0x7f73100a36b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f732810d9d0 0x7f732819de40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73280ff790 0x7f7328199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 --2- 172.21.15.16:0/1621103714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328101df0 0x7f7328199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:44.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 >> 172.21.15.16:0/1621103714 conn(0x7f73280761d0 msgr2=0x7f732810ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:44.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.650+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 shutdown_connections 2024-04-05T23:32:44.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:44.651+0000 7f732ec00700 1 -- 172.21.15.16:0/1621103714 wait complete. 2024-04-05T23:32:44.675 INFO:tasks.cephadm.ceph_manager.ceph:need seq 73014444082 got 73014444085 for osd.2 2024-04-05T23:32:44.675 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:44.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:44 smithi027 ceph-mon[78597]: pgmap v254: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:44.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:44 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/709806393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-05T23:32:44.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:44 smithi130 ceph-mon[79417]: pgmap v254: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:44.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:44 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/709806393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-05T23:32:44.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972375 got 266287972377 for osd.9 2024-04-05T23:32:44.913 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.155+0000 7fb4ece00700 1 -- 172.21.15.16:0/528589950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 msgr2=0x7fb4e810ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.155+0000 7fb4ece00700 1 --2- 172.21.15.16:0/528589950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 0x7fb4e810ddb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb4e000b0a0 tx=0x7fb4e000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.155+0000 7fb4ece00700 1 -- 172.21.15.16:0/528589950 shutdown_connections 2024-04-05T23:32:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.155+0000 7fb4ece00700 1 --2- 172.21.15.16:0/528589950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 0x7fb4e810ddb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.155+0000 7fb4ece00700 1 --2- 172.21.15.16:0/528589950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e8106f70 0x7fb4e8107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.155+0000 7fb4ece00700 1 --2- 172.21.15.16:0/528589950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4e8100f30 0x7fb4e8101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.155+0000 7fb4ece00700 1 -- 172.21.15.16:0/528589950 >> 172.21.15.16:0/528589950 conn(0x7fb4e80780d0 msgr2=0x7fb4e80784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.155+0000 7fb4ece00700 1 -- 172.21.15.16:0/528589950 shutdown_connections 2024-04-05T23:32:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.155+0000 7fb4ece00700 1 -- 172.21.15.16:0/528589950 wait complete. 2024-04-05T23:32:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.156+0000 7fb4ece00700 1 Processor -- start 2024-04-05T23:32:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.156+0000 7fb4ece00700 1 -- start start 2024-04-05T23:32:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4ece00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4e8100f30 0x7fb4e8107dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4ece00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 0x7fb4e8108300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4ece00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e8106f70 0x7fb4e8108840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4ece00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e81a9ca0 con 0x7fb4e81018d0 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4ece00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e81a9de0 con 0x7fb4e8106f70 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4ece00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e81a9f70 con 0x7fb4e8100f30 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e5400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 0x7fb4e8108300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e5400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 0x7fb4e8108300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40682/0 (socket says 172.21.15.16:40682) 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e5400700 1 -- 172.21.15.16:0/4177230589 learned_addr learned my addr 172.21.15.16:0/4177230589 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e6800700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e8106f70 0x7fb4e8108840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e5e00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4e8100f30 0x7fb4e8107dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e5400700 1 -- 172.21.15.16:0/4177230589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4e8100f30 msgr2=0x7fb4e8107dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e5400700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4e8100f30 0x7fb4e8107dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e5400700 1 -- 172.21.15.16:0/4177230589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e8106f70 msgr2=0x7fb4e8108840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e5400700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e8106f70 0x7fb4e8108840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.157+0000 7fb4e5400700 1 -- 172.21.15.16:0/4177230589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e0009d00 con 0x7fb4e81018d0 2024-04-05T23:32:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.158+0000 7fb4e6800700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e8106f70 0x7fb4e8108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.158+0000 7fb4e5400700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 0x7fb4e8108300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400e9d0 tx=0x7fb4d400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.158+0000 7fb4dea00700 1 -- 172.21.15.16:0/4177230589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4009790 con 0x7fb4e81018d0 2024-04-05T23:32:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.158+0000 7fb4dea00700 1 -- 172.21.15.16:0/4177230589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4d400ee70 con 0x7fb4e81018d0 2024-04-05T23:32:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.158+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e81aa250 con 0x7fb4e81018d0 2024-04-05T23:32:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.158+0000 7fb4dea00700 1 -- 172.21.15.16:0/4177230589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4010ea0 con 0x7fb4e81018d0 2024-04-05T23:32:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.158+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e81aa7a0 con 0x7fb4e81018d0 2024-04-05T23:32:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.159+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e81042b0 con 0x7fb4e81018d0 2024-04-05T23:32:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.161+0000 7fb4dea00700 1 -- 172.21.15.16:0/4177230589 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fb4d400cbd0 con 0x7fb4e81018d0 2024-04-05T23:32:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.163+0000 7fb4dea00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4cc0a1260 0x7fb4cc0a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.163+0000 7fb4dea00700 1 -- 172.21.15.16:0/4177230589 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb4d4014070 con 0x7fb4e81018d0 2024-04-05T23:32:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.163+0000 7fb4e5e00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4cc0a1260 0x7fb4cc0a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.164+0000 7fb4e5e00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4cc0a1260 0x7fb4cc0a3720 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009be0 tx=0x7fb4d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.168+0000 7fb4dea00700 1 -- 172.21.15.16:0/4177230589 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb4d4052ab0 con 0x7fb4e81018d0 2024-04-05T23:32:45.214 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579920 got 317827579922 for osd.11 2024-04-05T23:32:45.215 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:45.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.409+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb4e8102620 con 0x7fb4e81018d0 2024-04-05T23:32:45.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.410+0000 7fb4dea00700 1 -- 172.21.15.16:0/4177230589 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb4e8102620 con 0x7fb4e81018d0 2024-04-05T23:32:45.403 INFO:teuthology.orchestra.run.smithi016.stdout:206158430241 2024-04-05T23:32:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4cc0a1260 msgr2=0x7fb4cc0a3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4cc0a1260 0x7fb4cc0a3720 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009be0 tx=0x7fb4d8009260 comp rx=0 tx=0).stop 2024-04-05T23:32:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 msgr2=0x7fb4e8108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 0x7fb4e8108300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400e9d0 tx=0x7fb4d400ece0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 shutdown_connections 2024-04-05T23:32:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb4cc0a1260 0x7fb4cc0a3720 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4e8106f70 0x7fb4e8108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4e8100f30 0x7fb4e8107dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 --2- 172.21.15.16:0/4177230589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4e81018d0 0x7fb4e8108300 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.412+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 >> 172.21.15.16:0/4177230589 conn(0x7fb4e80780d0 msgr2=0x7fb4e810c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.413+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 shutdown_connections 2024-04-05T23:32:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:45.413+0000 7fb4ece00700 1 -- 172.21.15.16:0/4177230589 wait complete. 2024-04-05T23:32:45.566 INFO:tasks.cephadm.ceph_manager.ceph:need seq 98784247855 got 98784247856 for osd.3 2024-04-05T23:32:45.567 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:45.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:45 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1621103714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-05T23:32:45.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:45 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1621103714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-05T23:32:45.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:45 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1621103714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-05T23:32:46.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 51539607606 got 51539607608 for osd.1 2024-04-05T23:32:46.353 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.073+0000 7feca7400700 1 -- 172.21.15.16:0/2250794800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca01018d0 msgr2=0x7feca010ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.073+0000 7feca7400700 1 --2- 172.21.15.16:0/2250794800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca01018d0 0x7feca010ddb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fec9800b0a0 tx=0x7fec9800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.073+0000 7feca7400700 1 -- 172.21.15.16:0/2250794800 shutdown_connections 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.073+0000 7feca7400700 1 --2- 172.21.15.16:0/2250794800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca01018d0 0x7feca010ddb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.073+0000 7feca7400700 1 --2- 172.21.15.16:0/2250794800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca0106f70 0x7feca0107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.073+0000 7feca7400700 1 --2- 172.21.15.16:0/2250794800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feca0100f30 0x7feca0101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.073+0000 7feca7400700 1 -- 172.21.15.16:0/2250794800 >> 172.21.15.16:0/2250794800 conn(0x7feca00780d0 msgr2=0x7feca00784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.074+0000 7feca7400700 1 -- 172.21.15.16:0/2250794800 shutdown_connections 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.074+0000 7feca7400700 1 -- 172.21.15.16:0/2250794800 wait complete. 2024-04-05T23:32:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca7400700 1 Processor -- start 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca7400700 1 -- start start 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feca0100f30 0x7feca00731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca01018d0 0x7feca0073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca0106f70 0x7feca0073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca7400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca01a5950 con 0x7feca0106f70 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca7400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca01a5ac0 con 0x7feca01018d0 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca7400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca01a5c80 con 0x7feca0100f30 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7fec9fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca01018d0 0x7feca0073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca4a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feca0100f30 0x7feca00731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7feca5400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca0106f70 0x7feca0073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7fec9fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca01018d0 0x7feca0073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33800/0 (socket says 172.21.15.16:33800) 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.075+0000 7fec9fe00700 1 -- 172.21.15.16:0/3011689512 learned_addr learned my addr 172.21.15.16:0/3011689512 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7feca5400700 1 -- 172.21.15.16:0/3011689512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feca0100f30 msgr2=0x7feca00731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7feca5400700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feca0100f30 0x7feca00731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7feca5400700 1 -- 172.21.15.16:0/3011689512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca01018d0 msgr2=0x7feca0073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7feca5400700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca01018d0 0x7feca0073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7feca5400700 1 -- 172.21.15.16:0/3011689512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec98009d00 con 0x7feca0106f70 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7fec9fe00700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca01018d0 0x7feca0073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7feca5400700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca0106f70 0x7feca0073c60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fec9800b0a0 tx=0x7fec980094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7fec9d600700 1 -- 172.21.15.16:0/3011689512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9800e040 con 0x7feca0106f70 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7fec9d600700 1 -- 172.21.15.16:0/3011689512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec9801d480 con 0x7feca0106f70 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7fec9d600700 1 -- 172.21.15.16:0/3011689512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9801b670 con 0x7feca0106f70 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.076+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca01a5f00 con 0x7feca0106f70 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.077+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca01a6320 con 0x7feca0106f70 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.078+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca0109490 con 0x7feca0106f70 2024-04-05T23:32:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.079+0000 7fec9d600700 1 -- 172.21.15.16:0/3011689512 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fec9801d5f0 con 0x7feca0106f70 2024-04-05T23:32:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.081+0000 7fec9d600700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fec880a11b0 0x7fec880a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.082+0000 7fec9d600700 1 -- 172.21.15.16:0/3011689512 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fec980ce1e0 con 0x7feca0106f70 2024-04-05T23:32:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.082+0000 7feca4a00700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fec880a11b0 0x7fec880a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.082+0000 7feca4a00700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fec880a11b0 0x7fec880a3670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec94007b60 tx=0x7fec940073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.086+0000 7fec9d600700 1 -- 172.21.15.16:0/3011689512 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fec98097850 con 0x7feca0106f70 2024-04-05T23:32:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.306+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7feca00746c0 con 0x7feca0106f70 2024-04-05T23:32:46.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.306+0000 7fec9d600700 1 -- 172.21.15.16:0/3011689512 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fec9809ae70 con 0x7feca0106f70 2024-04-05T23:32:46.379 INFO:teuthology.orchestra.run.smithi016.stdout:128849018926 2024-04-05T23:32:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fec880a11b0 msgr2=0x7fec880a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fec880a11b0 0x7fec880a3670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec94007b60 tx=0x7fec940073d0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca0106f70 msgr2=0x7feca0073c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca0106f70 0x7feca0073c60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fec9800b0a0 tx=0x7fec980094c0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 shutdown_connections 2024-04-05T23:32:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fec880a11b0 0x7fec880a3670 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feca0106f70 0x7feca0073c60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feca0100f30 0x7feca00731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 --2- 172.21.15.16:0/3011689512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca01018d0 0x7feca0073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:46.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 >> 172.21.15.16:0/3011689512 conn(0x7feca00780d0 msgr2=0x7feca00ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:46.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 shutdown_connections 2024-04-05T23:32:46.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:46.309+0000 7feca7400700 1 -- 172.21.15.16:0/3011689512 wait complete. 2024-04-05T23:32:46.574 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430239 got 206158430241 for osd.7 2024-04-05T23:32:46.574 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:46.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:46 smithi016 ceph-mon[38248]: pgmap v255: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:46.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:46 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4177230589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-05T23:32:46.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:46 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3011689512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-05T23:32:46.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:46 smithi027 ceph-mon[78597]: pgmap v255: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:46 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4177230589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-05T23:32:46.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:46 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3011689512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-05T23:32:46.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:46 smithi130 ceph-mon[79417]: pgmap v255: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:46.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:46 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4177230589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-05T23:32:46.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:46 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3011689512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-05T23:32:47.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.156+0000 7f4f50000700 1 -- 172.21.15.16:0/2370757290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 msgr2=0x7f4f48107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.156+0000 7f4f50000700 1 --2- 172.21.15.16:0/2370757290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 0x7f4f48107350 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009990 tx=0x7f4f3c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.156+0000 7f4f50000700 1 -- 172.21.15.16:0/2370757290 shutdown_connections 2024-04-05T23:32:47.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.156+0000 7f4f50000700 1 --2- 172.21.15.16:0/2370757290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 0x7f4f48107350 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.156+0000 7f4f50000700 1 --2- 172.21.15.16:0/2370757290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f481018d0 0x7f4f4810ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.156+0000 7f4f50000700 1 --2- 172.21.15.16:0/2370757290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f48100f30 0x7f4f48101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.157+0000 7f4f50000700 1 -- 172.21.15.16:0/2370757290 >> 172.21.15.16:0/2370757290 conn(0x7f4f480780d0 msgr2=0x7f4f480784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:47.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.157+0000 7f4f50000700 1 -- 172.21.15.16:0/2370757290 shutdown_connections 2024-04-05T23:32:47.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.157+0000 7f4f50000700 1 -- 172.21.15.16:0/2370757290 wait complete. 2024-04-05T23:32:47.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.158+0000 7f4f50000700 1 Processor -- start 2024-04-05T23:32:47.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.158+0000 7f4f50000700 1 -- start start 2024-04-05T23:32:47.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.158+0000 7f4f50000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f48100f30 0x7f4f48197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:47.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.158+0000 7f4f50000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f481018d0 0x7f4f48197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.158+0000 7f4f50000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 0x7f4f4819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.158+0000 7f4f50000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4819c250 con 0x7f4f48106f70 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.158+0000 7f4f50000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4819c3c0 con 0x7f4f48100f30 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.158+0000 7f4f50000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4819c580 con 0x7f4f481018d0 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 0x7f4f4819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f48100f30 0x7f4f48197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 0x7f4f4819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40720/0 (socket says 172.21.15.16:40720) 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f48100f30 0x7f4f48197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33810/0 (socket says 172.21.15.16:33810) 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4e000700 1 -- 172.21.15.16:0/3628942124 learned_addr learned my addr 172.21.15.16:0/3628942124 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4cc00700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f481018d0 0x7f4f48197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4d600700 1 -- 172.21.15.16:0/3628942124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f481018d0 msgr2=0x7f4f48197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4d600700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f481018d0 0x7f4f48197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4d600700 1 -- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 msgr2=0x7f4f4819bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4d600700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 0x7f4f4819bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4d600700 1 -- 172.21.15.16:0/3628942124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c0095f0 con 0x7f4f48100f30 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4e000700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 0x7f4f4819bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.159+0000 7f4f4d600700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f48100f30 0x7f4f48197350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0098e0 tx=0x7f4f3c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.160+0000 7f4f46000700 1 -- 172.21.15.16:0/3628942124 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c01d070 con 0x7f4f48100f30 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.160+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4819c800 con 0x7f4f48100f30 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.160+0000 7f4f46000700 1 -- 172.21.15.16:0/3628942124 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f3c022470 con 0x7f4f48100f30 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.160+0000 7f4f46000700 1 -- 172.21.15.16:0/3628942124 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c017650 con 0x7f4f48100f30 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.160+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f481a40c0 con 0x7f4f48100f30 2024-04-05T23:32:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.161+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f48109490 con 0x7f4f48100f30 2024-04-05T23:32:47.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.163+0000 7f4f46000700 1 -- 172.21.15.16:0/3628942124 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f4f3c0225e0 con 0x7f4f48100f30 2024-04-05T23:32:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.164+0000 7f4f46000700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f4f300a11b0 0x7f4f300a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.164+0000 7f4f46000700 1 -- 172.21.15.16:0/3628942124 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4f3c0cde40 con 0x7f4f48100f30 2024-04-05T23:32:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.164+0000 7f4f4cc00700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f4f300a11b0 0x7f4f300a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.165+0000 7f4f4cc00700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f4f300a11b0 0x7f4f300a3670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4f481988b0 tx=0x7f4f34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.167+0000 7f4f46000700 1 -- 172.21.15.16:0/3628942124 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4f3c0974b0 con 0x7f4f48100f30 2024-04-05T23:32:47.325 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018923 got 128849018926 for osd.4 2024-04-05T23:32:47.326 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:47.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.387+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4f4804ebb0 con 0x7f4f48100f30 2024-04-05T23:32:47.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.387+0000 7f4f46000700 1 -- 172.21.15.16:0/3628942124 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4f3c09aad0 con 0x7f4f48100f30 2024-04-05T23:32:47.380 INFO:teuthology.orchestra.run.smithi016.stdout:34359738429 2024-04-05T23:32:47.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f4f300a11b0 msgr2=0x7f4f300a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f4f300a11b0 0x7f4f300a3670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4f481988b0 tx=0x7f4f34009260 comp rx=0 tx=0).stop 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f48100f30 msgr2=0x7f4f48197350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f48100f30 0x7f4f48197350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0098e0 tx=0x7f4f3c005740 comp rx=0 tx=0).stop 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 shutdown_connections 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f4f300a11b0 0x7f4f300a3670 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f48106f70 0x7f4f4819bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f48100f30 0x7f4f48197350 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 --2- 172.21.15.16:0/3628942124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f481018d0 0x7f4f48197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 >> 172.21.15.16:0/3628942124 conn(0x7f4f480780d0 msgr2=0x7f4f480ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 shutdown_connections 2024-04-05T23:32:47.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.390+0000 7f4f50000700 1 -- 172.21.15.16:0/3628942124 wait complete. 2024-04-05T23:32:47.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:47 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3628942124' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-05T23:32:47.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.696+0000 7fc221000700 1 -- 172.21.15.16:0/126193297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c100f30 msgr2=0x7fc21c101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.696+0000 7fc221000700 1 --2- 172.21.15.16:0/126193297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c100f30 0x7fc21c101390 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc208009990 tx=0x7fc208009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.697+0000 7fc221000700 1 -- 172.21.15.16:0/126193297 shutdown_connections 2024-04-05T23:32:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.697+0000 7fc221000700 1 --2- 172.21.15.16:0/126193297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc21c1018d0 0x7fc21c10ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.697+0000 7fc221000700 1 --2- 172.21.15.16:0/126193297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc21c106f70 0x7fc21c107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.697+0000 7fc221000700 1 --2- 172.21.15.16:0/126193297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c100f30 0x7fc21c101390 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.697+0000 7fc221000700 1 -- 172.21.15.16:0/126193297 >> 172.21.15.16:0/126193297 conn(0x7fc21c0780d0 msgr2=0x7fc21c0784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.697+0000 7fc221000700 1 -- 172.21.15.16:0/126193297 shutdown_connections 2024-04-05T23:32:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.697+0000 7fc221000700 1 -- 172.21.15.16:0/126193297 wait complete. 2024-04-05T23:32:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.698+0000 7fc221000700 1 Processor -- start 2024-04-05T23:32:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.698+0000 7fc221000700 1 -- start start 2024-04-05T23:32:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.698+0000 7fc221000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc21c100f30 0x7fc21c1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.698+0000 7fc221000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c1018d0 0x7fc21c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.698+0000 7fc221000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc21c106f70 0x7fc21c19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.698+0000 7fc221000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c19e480 con 0x7fc21c1018d0 2024-04-05T23:32:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.698+0000 7fc221000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c19e5f0 con 0x7fc21c106f70 2024-04-05T23:32:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.698+0000 7fc221000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c19e7b0 con 0x7fc21c100f30 2024-04-05T23:32:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.698+0000 7fc219e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc21c100f30 0x7fc21c1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c1018d0 0x7fc21c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc21c100f30 0x7fc21c1995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:48170/0 (socket says 172.21.15.16:48170) 2024-04-05T23:32:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219e00700 1 -- 172.21.15.16:0/33649647 learned_addr learned my addr 172.21.15.16:0/33649647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c1018d0 0x7fc21c199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40736/0 (socket says 172.21.15.16:40736) 2024-04-05T23:32:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc21a800700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc21c106f70 0x7fc21c19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219400700 1 -- 172.21.15.16:0/33649647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc21c100f30 msgr2=0x7fc21c1995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219400700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc21c100f30 0x7fc21c1995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219400700 1 -- 172.21.15.16:0/33649647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc21c106f70 msgr2=0x7fc21c19de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219400700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc21c106f70 0x7fc21c19de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219400700 1 -- 172.21.15.16:0/33649647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2080095f0 con 0x7fc21c1018d0 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219e00700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc21c100f30 0x7fc21c1995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc219400700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c1018d0 0x7fc21c199b10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc2080098e0 tx=0x7fc208005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.699+0000 7fc212a00700 1 -- 172.21.15.16:0/33649647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20801d070 con 0x7fc21c1018d0 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.700+0000 7fc212a00700 1 -- 172.21.15.16:0/33649647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc208022470 con 0x7fc21c1018d0 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.700+0000 7fc212a00700 1 -- 172.21.15.16:0/33649647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208017650 con 0x7fc21c1018d0 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.700+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21c19ea30 con 0x7fc21c1018d0 2024-04-05T23:32:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.700+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21c1a6120 con 0x7fc21c1018d0 2024-04-05T23:32:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.701+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21c102620 con 0x7fc21c1018d0 2024-04-05T23:32:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.703+0000 7fc212a00700 1 -- 172.21.15.16:0/33649647 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fc2080225e0 con 0x7fc21c1018d0 2024-04-05T23:32:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.704+0000 7fc212a00700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2000a1230 0x7fc2000a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.704+0000 7fc212a00700 1 -- 172.21.15.16:0/33649647 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc2080cdec0 con 0x7fc21c1018d0 2024-04-05T23:32:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.704+0000 7fc219e00700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2000a1230 0x7fc2000a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.705+0000 7fc219e00700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2000a1230 0x7fc2000a36f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc20c011550 tx=0x7fc20c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:47.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.708+0000 7fc212a00700 1 -- 172.21.15.16:0/33649647 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc2080976b0 con 0x7fc21c1018d0 2024-04-05T23:32:47.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:47 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3628942124' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-05T23:32:47.817 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738427 got 34359738429 for osd.0 2024-04-05T23:32:47.817 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:47.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:47 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3628942124' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-05T23:32:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.929+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc21c04ebb0 con 0x7fc21c1018d0 2024-04-05T23:32:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.930+0000 7fc212a00700 1 -- 172.21.15.16:0/33649647 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc20809acd0 con 0x7fc21c1018d0 2024-04-05T23:32:47.923 INFO:teuthology.orchestra.run.smithi016.stdout:150323855401 2024-04-05T23:32:47.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2000a1230 msgr2=0x7fc2000a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2000a1230 0x7fc2000a36f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc20c011550 tx=0x7fc20c009260 comp rx=0 tx=0).stop 2024-04-05T23:32:47.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c1018d0 msgr2=0x7fc21c199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:47.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c1018d0 0x7fc21c199b10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc2080098e0 tx=0x7fc208005740 comp rx=0 tx=0).stop 2024-04-05T23:32:47.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 shutdown_connections 2024-04-05T23:32:47.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc2000a1230 0x7fc2000a36f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc21c106f70 0x7fc21c19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc21c100f30 0x7fc21c1995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 --2- 172.21.15.16:0/33649647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc21c1018d0 0x7fc21c199b10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.932+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 >> 172.21.15.16:0/33649647 conn(0x7fc21c0780d0 msgr2=0x7fc21c0ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.933+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 shutdown_connections 2024-04-05T23:32:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:47.933+0000 7fc221000700 1 -- 172.21.15.16:0/33649647 wait complete. 2024-04-05T23:32:48.313 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855398 got 150323855401 for osd.5 2024-04-05T23:32:48.314 DEBUG:teuthology.parallel:result is None 2024-04-05T23:32:48.314 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-05T23:32:48.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-05T23:32:48.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:48 smithi016 ceph-mon[38248]: pgmap v256: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 65 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:48.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:48 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/33649647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-05T23:32:48.683 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:48.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:48 smithi027 ceph-mon[78597]: pgmap v256: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 65 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:48.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:48 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/33649647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-05T23:32:48.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:48 smithi130 ceph-mon[79417]: pgmap v256: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 65 B/s, 1 keys/s, 1 objects/s recovering 2024-04-05T23:32:48.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:48 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/33649647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-05T23:32:50.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.036+0000 7f3e73400700 1 -- 172.21.15.16:0/2254832047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c10d8e0 msgr2=0x7f3e6c10fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:50.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.036+0000 7f3e73400700 1 --2- 172.21.15.16:0/2254832047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c10d8e0 0x7f3e6c10fcd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400b0a0 tx=0x7f3e6400b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.036+0000 7f3e73400700 1 -- 172.21.15.16:0/2254832047 shutdown_connections 2024-04-05T23:32:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.036+0000 7f3e73400700 1 --2- 172.21.15.16:0/2254832047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c10d8e0 0x7f3e6c10fcd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.036+0000 7f3e73400700 1 --2- 172.21.15.16:0/2254832047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c073110 0x7f3e6c0734f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.036+0000 7f3e73400700 1 --2- 172.21.15.16:0/2254832047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3e6c073a30 0x7f3e6c10d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.036+0000 7f3e73400700 1 -- 172.21.15.16:0/2254832047 >> 172.21.15.16:0/2254832047 conn(0x7f3e6c0780d0 msgr2=0x7f3e6c0784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:50.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.036+0000 7f3e73400700 1 -- 172.21.15.16:0/2254832047 shutdown_connections 2024-04-05T23:32:50.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.036+0000 7f3e73400700 1 -- 172.21.15.16:0/2254832047 wait complete. 2024-04-05T23:32:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.037+0000 7f3e73400700 1 Processor -- start 2024-04-05T23:32:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.037+0000 7f3e73400700 1 -- start start 2024-04-05T23:32:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e73400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c073110 0x7f3e6c1996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e73400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c073a30 0x7f3e6c199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e73400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3e6c10d8e0 0x7f3e6c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:50.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e73400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c19e5a0 con 0x7f3e6c073a30 2024-04-05T23:32:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e73400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c19e710 con 0x7f3e6c073110 2024-04-05T23:32:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e73400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c19e8d0 con 0x7f3e6c10d8e0 2024-04-05T23:32:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e6be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c073a30 0x7f3e6c199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e70a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c073110 0x7f3e6c1996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:50.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e6be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c073a30 0x7f3e6c199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40748/0 (socket says 172.21.15.16:40748) 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e6be00700 1 -- 172.21.15.16:0/2270883997 learned_addr learned my addr 172.21.15.16:0/2270883997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e71400700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3e6c10d8e0 0x7f3e6c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e6be00700 1 -- 172.21.15.16:0/2270883997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3e6c10d8e0 msgr2=0x7f3e6c19df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e6be00700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3e6c10d8e0 0x7f3e6c19df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e6be00700 1 -- 172.21.15.16:0/2270883997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c073110 msgr2=0x7f3e6c1996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e6be00700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c073110 0x7f3e6c1996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e6be00700 1 -- 172.21.15.16:0/2270883997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e64009d00 con 0x7f3e6c073a30 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e70a00700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c073110 0x7f3e6c1996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.038+0000 7f3e71400700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3e6c10d8e0 0x7f3e6c19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.039+0000 7f3e6be00700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c073a30 0x7f3e6c199be0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800e9a0 tx=0x7f3e5800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.039+0000 7f3e69600700 1 -- 172.21.15.16:0/2270883997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58009790 con 0x7f3e6c073a30 2024-04-05T23:32:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.039+0000 7f3e69600700 1 -- 172.21.15.16:0/2270883997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e5800cbd0 con 0x7f3e6c073a30 2024-04-05T23:32:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.039+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6c19ebb0 con 0x7f3e6c073a30 2024-04-05T23:32:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.039+0000 7f3e69600700 1 -- 172.21.15.16:0/2270883997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58018680 con 0x7f3e6c073a30 2024-04-05T23:32:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.039+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6c1a6410 con 0x7f3e6c073a30 2024-04-05T23:32:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.040+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6c108af0 con 0x7f3e6c073a30 2024-04-05T23:32:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.042+0000 7f3e69600700 1 -- 172.21.15.16:0/2270883997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f3e5800cd40 con 0x7f3e6c073a30 2024-04-05T23:32:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.043+0000 7f3e69600700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3e540a11b0 0x7f3e540a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.043+0000 7f3e69600700 1 -- 172.21.15.16:0/2270883997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3e58014070 con 0x7f3e6c073a30 2024-04-05T23:32:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.043+0000 7f3e70a00700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3e540a11b0 0x7f3e540a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.044+0000 7f3e70a00700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3e540a11b0 0x7f3e540a3670 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3e60011490 tx=0x7f3e60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.048+0000 7f3e69600700 1 -- 172.21.15.16:0/2270883997 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3e58052ab0 con 0x7f3e6c073a30 2024-04-05T23:32:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.252+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e6c19a9b0 con 0x7f3e540a11b0 2024-04-05T23:32:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.263+0000 7f3e69600700 1 -- 172.21.15.16:0/2270883997 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340298 (secure 0 0 0) 0x7f3e6c19a9b0 con 0x7f3e540a11b0 2024-04-05T23:32:50.263 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3e540a11b0 msgr2=0x7f3e540a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3e540a11b0 0x7f3e540a3670 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3e60011490 tx=0x7f3e60009260 comp rx=0 tx=0).stop 2024-04-05T23:32:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c073a30 msgr2=0x7f3e6c199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c073a30 0x7f3e6c199be0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800e9a0 tx=0x7f3e5800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 shutdown_connections 2024-04-05T23:32:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3e540a11b0 0x7f3e540a3670 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3e6c10d8e0 0x7f3e6c19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c073110 0x7f3e6c1996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 --2- 172.21.15.16:0/2270883997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e6c073a30 0x7f3e6c199be0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 >> 172.21.15.16:0/2270883997 conn(0x7f3e6c0780d0 msgr2=0x7f3e6c0ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 shutdown_connections 2024-04-05T23:32:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:50.269+0000 7f3e73400700 1 -- 172.21.15.16:0/2270883997 wait complete. 2024-04-05T23:32:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-05T23:32:50.577 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:50 smithi016 ceph-mon[38248]: pgmap v257: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 59 B/s, 0 keys/s, 1 objects/s recovering 2024-04-05T23:32:50.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:50 smithi027 ceph-mon[78597]: pgmap v257: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 59 B/s, 0 keys/s, 1 objects/s recovering 2024-04-05T23:32:50.770 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-04-05T23:32:49.302901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656662,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":540,"num_read_kb":392,"num_write":415,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":341,"ondisk_log_size":341,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365496,"kb_used_data":12564,"kb_used_omap":18,"kb_used_meta":352749,"kb_avail":1124576328,"statfs":{"total":1151940427776,"available":1151566159872,"internally_reserved":0,"allocated":12865536,"data_stored":7117871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":361215301},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":99,"num_read_kb":49,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.931061"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371867+0000","last_change":"2024-04-05T23:32:04.536879+0000","last_active":"2024-04-05T23:32:37.371867+0000","last_peered":"2024-04-05T23:32:37.371867+0000","last_clean":"2024-04-05T23:32:37.371867+0000","last_became_active":"2024-04-05T23:32:04.536183+0000","last_became_peered":"2024-04-05T23:32:04.536183+0000","last_unstale":"2024-04-05T23:32:37.371867+0000","last_undegraded":"2024-04-05T23:32:37.371867+0000","last_fullsized":"2024-04-05T23:32:37.371867+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:13:22.589464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380649+0000","last_change":"2024-04-05T23:32:04.540791+0000","last_active":"2024-04-05T23:32:37.380649+0000","last_peered":"2024-04-05T23:32:37.380649+0000","last_clean":"2024-04-05T23:32:37.380649+0000","last_became_active":"2024-04-05T23:32:04.540455+0000","last_became_peered":"2024-04-05T23:32:04.540455+0000","last_unstale":"2024-04-05T23:32:37.380649+0000","last_undegraded":"2024-04-05T23:32:37.380649+0000","last_fullsized":"2024-04-05T23:32:37.380649+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:06:19.090672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380615+0000","last_change":"2024-04-05T23:32:04.543428+0000","last_active":"2024-04-05T23:32:37.380615+0000","last_peered":"2024-04-05T23:32:37.380615+0000","last_clean":"2024-04-05T23:32:37.380615+0000","last_became_active":"2024-04-05T23:32:04.543071+0000","last_became_peered":"2024-04-05T23:32:04.543071+0000","last_unstale":"2024-04-05T23:32:37.380615+0000","last_undegraded":"2024-04-05T23:32:37.380615+0000","last_fullsized":"2024-04-05T23:32:37.380615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:35:44.911749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378253+0000","last_change":"2024-04-05T23:32:04.531444+0000","last_active":"2024-04-05T23:32:37.378253+0000","last_peered":"2024-04-05T23:32:37.378253+0000","last_clean":"2024-04-05T23:32:37.378253+0000","last_became_active":"2024-04-05T23:32:04.529556+0000","last_became_peered":"2024-04-05T23:32:04.529556+0000","last_unstale":"2024-04-05T23:32:37.378253+0000","last_undegraded":"2024-04-05T23:32:37.378253+0000","last_fullsized":"2024-04-05T23:32:37.378253+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:48.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380525+0000","last_change":"2024-04-05T23:32:04.529565+0000","last_active":"2024-04-05T23:32:37.380525+0000","last_peered":"2024-04-05T23:32:37.380525+0000","last_clean":"2024-04-05T23:32:37.380525+0000","last_became_active":"2024-04-05T23:32:04.529226+0000","last_became_peered":"2024-04-05T23:32:04.529226+0000","last_unstale":"2024-04-05T23:32:37.380525+0000","last_undegraded":"2024-04-05T23:32:37.380525+0000","last_fullsized":"2024-04-05T23:32:37.380525+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:44:25.208996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375831+0000","last_change":"2024-04-05T23:32:04.546260+0000","last_active":"2024-04-05T23:32:37.375831+0000","last_peered":"2024-04-05T23:32:37.375831+0000","last_clean":"2024-04-05T23:32:37.375831+0000","last_became_active":"2024-04-05T23:32:04.545851+0000","last_became_peered":"2024-04-05T23:32:04.545851+0000","last_unstale":"2024-04-05T23:32:37.375831+0000","last_undegraded":"2024-04-05T23:32:37.375831+0000","last_fullsized":"2024-04-05T23:32:37.375831+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:59:16.735635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378362+0000","last_change":"2024-04-05T23:32:04.536348+0000","last_active":"2024-04-05T23:32:37.378362+0000","last_peered":"2024-04-05T23:32:37.378362+0000","last_clean":"2024-04-05T23:32:37.378362+0000","last_became_active":"2024-04-05T23:32:04.536015+0000","last_became_peered":"2024-04-05T23:32:04.536015+0000","last_unstale":"2024-04-05T23:32:37.378362+0000","last_undegraded":"2024-04-05T23:32:37.378362+0000","last_fullsized":"2024-04-05T23:32:37.378362+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:02:24.897957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371406+0000","last_change":"2024-04-05T23:32:04.532528+0000","last_active":"2024-04-05T23:32:37.371406+0000","last_peered":"2024-04-05T23:32:37.371406+0000","last_clean":"2024-04-05T23:32:37.371406+0000","last_became_active":"2024-04-05T23:32:04.532104+0000","last_became_peered":"2024-04-05T23:32:04.532104+0000","last_unstale":"2024-04-05T23:32:37.371406+0000","last_undegraded":"2024-04-05T23:32:37.371406+0000","last_fullsized":"2024-04-05T23:32:37.371406+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:06.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.381590+0000","last_change":"2024-04-05T23:32:04.531223+0000","last_active":"2024-04-05T23:32:37.381590+0000","last_peered":"2024-04-05T23:32:37.381590+0000","last_clean":"2024-04-05T23:32:37.381590+0000","last_became_active":"2024-04-05T23:32:04.530863+0000","last_became_peered":"2024-04-05T23:32:04.530863+0000","last_unstale":"2024-04-05T23:32:37.381590+0000","last_undegraded":"2024-04-05T23:32:37.381590+0000","last_fullsized":"2024-04-05T23:32:37.381590+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:03:35.912984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.376132+0000","last_change":"2024-04-05T23:32:04.541481+0000","last_active":"2024-04-05T23:32:37.376132+0000","last_peered":"2024-04-05T23:32:37.376132+0000","last_clean":"2024-04-05T23:32:37.376132+0000","last_became_active":"2024-04-05T23:32:04.541133+0000","last_became_peered":"2024-04-05T23:32:04.541133+0000","last_unstale":"2024-04-05T23:32:37.376132+0000","last_undegraded":"2024-04-05T23:32:37.376132+0000","last_fullsized":"2024-04-05T23:32:37.376132+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:01:22.825587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378586+0000","last_change":"2024-04-05T23:32:04.542390+0000","last_active":"2024-04-05T23:32:37.378586+0000","last_peered":"2024-04-05T23:32:37.378586+0000","last_clean":"2024-04-05T23:32:37.378586+0000","last_became_active":"2024-04-05T23:32:04.542026+0000","last_became_peered":"2024-04-05T23:32:04.542026+0000","last_unstale":"2024-04-05T23:32:37.378586+0000","last_undegraded":"2024-04-05T23:32:37.378586+0000","last_fullsized":"2024-04-05T23:32:37.378586+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:15:42.243807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378195+0000","last_change":"2024-04-05T23:32:04.540096+0000","last_active":"2024-04-05T23:32:37.378195+0000","last_peered":"2024-04-05T23:32:37.378195+0000","last_clean":"2024-04-05T23:32:37.378195+0000","last_became_active":"2024-04-05T23:32:04.539764+0000","last_became_peered":"2024-04-05T23:32:04.539764+0000","last_unstale":"2024-04-05T23:32:37.378195+0000","last_undegraded":"2024-04-05T23:32:37.378195+0000","last_fullsized":"2024-04-05T23:32:37.378195+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:57:36.442393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378551+0000","last_change":"2024-04-05T23:32:04.540839+0000","last_active":"2024-04-05T23:32:37.378551+0000","last_peered":"2024-04-05T23:32:37.378551+0000","last_clean":"2024-04-05T23:32:37.378551+0000","last_became_active":"2024-04-05T23:32:04.540511+0000","last_became_peered":"2024-04-05T23:32:04.540511+0000","last_unstale":"2024-04-05T23:32:37.378551+0000","last_undegraded":"2024-04-05T23:32:37.378551+0000","last_fullsized":"2024-04-05T23:32:37.378551+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:58:24.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380309+0000","last_change":"2024-04-05T23:32:04.539222+0000","last_active":"2024-04-05T23:32:37.380309+0000","last_peered":"2024-04-05T23:32:37.380309+0000","last_clean":"2024-04-05T23:32:37.380309+0000","last_became_active":"2024-04-05T23:32:04.538906+0000","last_became_peered":"2024-04-05T23:32:04.538906+0000","last_unstale":"2024-04-05T23:32:37.380309+0000","last_undegraded":"2024-04-05T23:32:37.380309+0000","last_fullsized":"2024-04-05T23:32:37.380309+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:05:24.429153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380277+0000","last_change":"2024-04-05T23:32:04.544155+0000","last_active":"2024-04-05T23:32:37.380277+0000","last_peered":"2024-04-05T23:32:37.380277+0000","last_clean":"2024-04-05T23:32:37.380277+0000","last_became_active":"2024-04-05T23:32:04.543852+0000","last_became_peered":"2024-04-05T23:32:04.543852+0000","last_unstale":"2024-04-05T23:32:37.380277+0000","last_undegraded":"2024-04-05T23:32:37.380277+0000","last_fullsized":"2024-04-05T23:32:37.380277+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:18:34.353542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380824+0000","last_change":"2024-04-05T23:32:04.533772+0000","last_active":"2024-04-05T23:32:37.380824+0000","last_peered":"2024-04-05T23:32:37.380824+0000","last_clean":"2024-04-05T23:32:37.380824+0000","last_became_active":"2024-04-05T23:32:04.533416+0000","last_became_peered":"2024-04-05T23:32:04.533416+0000","last_unstale":"2024-04-05T23:32:37.380824+0000","last_undegraded":"2024-04-05T23:32:37.380824+0000","last_fullsized":"2024-04-05T23:32:37.380824+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:36:55.072349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371943+0000","last_change":"2024-04-05T23:32:04.551902+0000","last_active":"2024-04-05T23:32:37.371943+0000","last_peered":"2024-04-05T23:32:37.371943+0000","last_clean":"2024-04-05T23:32:37.371943+0000","last_became_active":"2024-04-05T23:32:04.551595+0000","last_became_peered":"2024-04-05T23:32:04.551595+0000","last_unstale":"2024-04-05T23:32:37.371943+0000","last_undegraded":"2024-04-05T23:32:37.371943+0000","last_fullsized":"2024-04-05T23:32:37.371943+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:14:46.648789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375868+0000","last_change":"2024-04-05T23:32:04.542967+0000","last_active":"2024-04-05T23:32:37.375868+0000","last_peered":"2024-04-05T23:32:37.375868+0000","last_clean":"2024-04-05T23:32:37.375868+0000","last_became_active":"2024-04-05T23:32:04.541767+0000","last_became_peered":"2024-04-05T23:32:04.541767+0000","last_unstale":"2024-04-05T23:32:37.375868+0000","last_undegraded":"2024-04-05T23:32:37.375868+0000","last_fullsized":"2024-04-05T23:32:37.375868+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:27:46.150249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375102+0000","last_change":"2024-04-05T23:32:04.549512+0000","last_active":"2024-04-05T23:32:37.375102+0000","last_peered":"2024-04-05T23:32:37.375102+0000","last_clean":"2024-04-05T23:32:37.375102+0000","last_became_active":"2024-04-05T23:32:04.549184+0000","last_became_peered":"2024-04-05T23:32:04.549184+0000","last_unstale":"2024-04-05T23:32:37.375102+0000","last_undegraded":"2024-04-05T23:32:37.375102+0000","last_fullsized":"2024-04-05T23:32:37.375102+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:48:24.186408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:42.490087+0000","last_change":"2024-04-05T23:32:04.005188+0000","last_active":"2024-04-05T23:32:42.490087+0000","last_peered":"2024-04-05T23:32:42.490087+0000","last_clean":"2024-04-05T23:32:42.490087+0000","last_became_active":"2024-04-05T23:32:04.004818+0000","last_became_peered":"2024-04-05T23:32:04.004818+0000","last_unstale":"2024-04-05T23:32:42.490087+0000","last_undegraded":"2024-04-05T23:32:42.490087+0000","last_fullsized":"2024-04-05T23:32:42.490087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:49:41.146761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.372092+0000","last_change":"2024-04-05T23:32:04.544881+0000","last_active":"2024-04-05T23:32:37.372092+0000","last_peered":"2024-04-05T23:32:37.372092+0000","last_clean":"2024-04-05T23:32:37.372092+0000","last_became_active":"2024-04-05T23:32:04.542149+0000","last_became_peered":"2024-04-05T23:32:04.542149+0000","last_unstale":"2024-04-05T23:32:37.372092+0000","last_undegraded":"2024-04-05T23:32:37.372092+0000","last_fullsized":"2024-04-05T23:32:37.372092+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:28:51.713053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.372234+0000","last_change":"2024-04-05T23:32:04.011450+0000","last_active":"2024-04-05T23:32:37.372234+0000","last_peered":"2024-04-05T23:32:37.372234+0000","last_clean":"2024-04-05T23:32:37.372234+0000","last_became_active":"2024-04-05T23:32:04.009607+0000","last_became_peered":"2024-04-05T23:32:04.009607+0000","last_unstale":"2024-04-05T23:32:37.372234+0000","last_undegraded":"2024-04-05T23:32:37.372234+0000","last_fullsized":"2024-04-05T23:32:37.372234+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:16:45.483108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.381468+0000","last_change":"2024-04-05T23:32:04.550370+0000","last_active":"2024-04-05T23:32:37.381468+0000","last_peered":"2024-04-05T23:32:37.381468+0000","last_clean":"2024-04-05T23:32:37.381468+0000","last_became_active":"2024-04-05T23:32:04.549980+0000","last_became_peered":"2024-04-05T23:32:04.549980+0000","last_unstale":"2024-04-05T23:32:37.381468+0000","last_undegraded":"2024-04-05T23:32:37.381468+0000","last_fullsized":"2024-04-05T23:32:37.381468+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:44:30.704366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380842+0000","last_change":"2024-04-05T23:32:04.004169+0000","last_active":"2024-04-05T23:32:37.380842+0000","last_peered":"2024-04-05T23:32:37.380842+0000","last_clean":"2024-04-05T23:32:37.380842+0000","last_became_active":"2024-04-05T23:32:04.003646+0000","last_became_peered":"2024-04-05T23:32:04.003646+0000","last_unstale":"2024-04-05T23:32:37.380842+0000","last_undegraded":"2024-04-05T23:32:37.380842+0000","last_fullsized":"2024-04-05T23:32:37.380842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:57:02.781815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.381679+0000","last_change":"2024-04-05T23:32:04.542066+0000","last_active":"2024-04-05T23:32:37.381679+0000","last_peered":"2024-04-05T23:32:37.381679+0000","last_clean":"2024-04-05T23:32:37.381679+0000","last_became_active":"2024-04-05T23:32:04.540735+0000","last_became_peered":"2024-04-05T23:32:04.540735+0000","last_unstale":"2024-04-05T23:32:37.381679+0000","last_undegraded":"2024-04-05T23:32:37.381679+0000","last_fullsized":"2024-04-05T23:32:37.381679+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:02:03.595888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.397068+0000","last_change":"2024-04-05T23:32:38.396641+0000","last_active":"2024-04-05T23:32:38.397068+0000","last_peered":"2024-04-05T23:32:38.397068+0000","last_clean":"2024-04-05T23:32:38.397068+0000","last_became_active":"2024-04-05T23:32:38.379708+0000","last_became_peered":"2024-04-05T23:32:38.379708+0000","last_unstale":"2024-04-05T23:32:38.397068+0000","last_undegraded":"2024-04-05T23:32:38.397068+0000","last_fullsized":"2024-04-05T23:32:38.397068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:00.370725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378242+0000","last_change":"2024-04-05T23:32:04.545780+0000","last_active":"2024-04-05T23:32:37.378242+0000","last_peered":"2024-04-05T23:32:37.378242+0000","last_clean":"2024-04-05T23:32:37.378242+0000","last_became_active":"2024-04-05T23:32:04.545279+0000","last_became_peered":"2024-04-05T23:32:04.545279+0000","last_unstale":"2024-04-05T23:32:37.378242+0000","last_undegraded":"2024-04-05T23:32:37.378242+0000","last_fullsized":"2024-04-05T23:32:37.378242+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:10.577280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:43.462109+0000","last_change":"2024-04-05T23:32:04.004189+0000","last_active":"2024-04-05T23:32:43.462109+0000","last_peered":"2024-04-05T23:32:43.462109+0000","last_clean":"2024-04-05T23:32:43.462109+0000","last_became_active":"2024-04-05T23:32:04.003841+0000","last_became_peered":"2024-04-05T23:32:04.003841+0000","last_unstale":"2024-04-05T23:32:43.462109+0000","last_undegraded":"2024-04-05T23:32:43.462109+0000","last_fullsized":"2024-04-05T23:32:43.462109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:33:26.898657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380547+0000","last_change":"2024-04-05T23:32:04.529538+0000","last_active":"2024-04-05T23:32:37.380547+0000","last_peered":"2024-04-05T23:32:37.380547+0000","last_clean":"2024-04-05T23:32:37.380547+0000","last_became_active":"2024-04-05T23:32:04.529231+0000","last_became_peered":"2024-04-05T23:32:04.529231+0000","last_unstale":"2024-04-05T23:32:37.380547+0000","last_undegraded":"2024-04-05T23:32:37.380547+0000","last_fullsized":"2024-04-05T23:32:37.380547+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:49:15.608966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375765+0000","last_change":"2024-04-05T23:32:04.002763+0000","last_active":"2024-04-05T23:32:37.375765+0000","last_peered":"2024-04-05T23:32:37.375765+0000","last_clean":"2024-04-05T23:32:37.375765+0000","last_became_active":"2024-04-05T23:32:04.002410+0000","last_became_peered":"2024-04-05T23:32:04.002410+0000","last_unstale":"2024-04-05T23:32:37.375765+0000","last_undegraded":"2024-04-05T23:32:37.375765+0000","last_fullsized":"2024-04-05T23:32:37.375765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:15.193813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.462329+0000","last_change":"2024-04-05T23:32:38.461447+0000","last_active":"2024-04-05T23:32:38.462329+0000","last_peered":"2024-04-05T23:32:38.462329+0000","last_clean":"2024-04-05T23:32:38.462329+0000","last_became_active":"2024-04-05T23:32:38.460968+0000","last_became_peered":"2024-04-05T23:32:38.460968+0000","last_unstale":"2024-04-05T23:32:38.462329+0000","last_undegraded":"2024-04-05T23:32:38.462329+0000","last_fullsized":"2024-04-05T23:32:38.462329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:42:21.093881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378913+0000","last_change":"2024-04-05T23:32:04.001937+0000","last_active":"2024-04-05T23:32:37.378913+0000","last_peered":"2024-04-05T23:32:37.378913+0000","last_clean":"2024-04-05T23:32:37.378913+0000","last_became_active":"2024-04-05T23:32:04.001508+0000","last_became_peered":"2024-04-05T23:32:04.001508+0000","last_unstale":"2024-04-05T23:32:37.378913+0000","last_undegraded":"2024-04-05T23:32:37.378913+0000","last_fullsized":"2024-04-05T23:32:37.378913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:09:20.682084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371478+0000","last_change":"2024-04-05T23:32:04.531576+0000","last_active":"2024-04-05T23:32:37.371478+0000","last_peered":"2024-04-05T23:32:37.371478+0000","last_clean":"2024-04-05T23:32:37.371478+0000","last_became_active":"2024-04-05T23:32:04.531234+0000","last_became_peered":"2024-04-05T23:32:04.531234+0000","last_unstale":"2024-04-05T23:32:37.371478+0000","last_undegraded":"2024-04-05T23:32:37.371478+0000","last_fullsized":"2024-04-05T23:32:37.371478+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:21:57.769861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377030+0000","last_change":"2024-04-05T23:32:04.001570+0000","last_active":"2024-04-05T23:32:37.377030+0000","last_peered":"2024-04-05T23:32:37.377030+0000","last_clean":"2024-04-05T23:32:37.377030+0000","last_became_active":"2024-04-05T23:32:04.001224+0000","last_became_peered":"2024-04-05T23:32:04.001224+0000","last_unstale":"2024-04-05T23:32:37.377030+0000","last_undegraded":"2024-04-05T23:32:37.377030+0000","last_fullsized":"2024-04-05T23:32:37.377030+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:44:41.070205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375868+0000","last_change":"2024-04-05T23:32:04.542029+0000","last_active":"2024-04-05T23:32:37.375868+0000","last_peered":"2024-04-05T23:32:37.375868+0000","last_clean":"2024-04-05T23:32:37.375868+0000","last_became_active":"2024-04-05T23:32:04.541684+0000","last_became_peered":"2024-04-05T23:32:04.541684+0000","last_unstale":"2024-04-05T23:32:37.375868+0000","last_undegraded":"2024-04-05T23:32:37.375868+0000","last_fullsized":"2024-04-05T23:32:37.375868+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:07:06.456917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":92,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:46.466090+0000","last_change":"2024-04-05T23:32:04.004986+0000","last_active":"2024-04-05T23:32:46.466090+0000","last_peered":"2024-04-05T23:32:46.466090+0000","last_clean":"2024-04-05T23:32:46.466090+0000","last_became_active":"2024-04-05T23:32:04.004606+0000","last_became_peered":"2024-04-05T23:32:04.004606+0000","last_unstale":"2024-04-05T23:32:46.466090+0000","last_undegraded":"2024-04-05T23:32:46.466090+0000","last_fullsized":"2024-04-05T23:32:46.466090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:23:41.060500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.381509+0000","last_change":"2024-04-05T23:32:04.541334+0000","last_active":"2024-04-05T23:32:37.381509+0000","last_peered":"2024-04-05T23:32:37.381509+0000","last_clean":"2024-04-05T23:32:37.381509+0000","last_became_active":"2024-04-05T23:32:04.540967+0000","last_became_peered":"2024-04-05T23:32:04.540967+0000","last_unstale":"2024-04-05T23:32:37.381509+0000","last_undegraded":"2024-04-05T23:32:37.381509+0000","last_fullsized":"2024-04-05T23:32:37.381509+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:16:33.045172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.466417+0000","last_change":"2024-04-05T23:32:38.466417+0000","last_active":"2024-04-05T23:32:38.466417+0000","last_peered":"2024-04-05T23:32:38.466417+0000","last_clean":"2024-04-05T23:32:38.466417+0000","last_became_active":"2024-04-05T23:32:38.462293+0000","last_became_peered":"2024-04-05T23:32:38.462293+0000","last_unstale":"2024-04-05T23:32:38.466417+0000","last_undegraded":"2024-04-05T23:32:38.466417+0000","last_fullsized":"2024-04-05T23:32:38.466417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:31.118636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.385818+0000","last_change":"2024-04-05T23:32:38.385818+0000","last_active":"2024-04-05T23:32:38.385818+0000","last_peered":"2024-04-05T23:32:38.385818+0000","last_clean":"2024-04-05T23:32:38.385818+0000","last_became_active":"2024-04-05T23:32:38.385423+0000","last_became_peered":"2024-04-05T23:32:38.385423+0000","last_unstale":"2024-04-05T23:32:38.385818+0000","last_undegraded":"2024-04-05T23:32:38.385818+0000","last_fullsized":"2024-04-05T23:32:38.385818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:07:11.374783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378141+0000","last_change":"2024-04-05T23:32:04.000697+0000","last_active":"2024-04-05T23:32:37.378141+0000","last_peered":"2024-04-05T23:32:37.378141+0000","last_clean":"2024-04-05T23:32:37.378141+0000","last_became_active":"2024-04-05T23:32:04.000377+0000","last_became_peered":"2024-04-05T23:32:04.000377+0000","last_unstale":"2024-04-05T23:32:37.378141+0000","last_undegraded":"2024-04-05T23:32:37.378141+0000","last_fullsized":"2024-04-05T23:32:37.378141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:55:11.227637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.383367+0000","last_change":"2024-04-05T23:32:38.383367+0000","last_active":"2024-04-05T23:32:38.383367+0000","last_peered":"2024-04-05T23:32:38.383367+0000","last_clean":"2024-04-05T23:32:38.383367+0000","last_became_active":"2024-04-05T23:32:38.383002+0000","last_became_peered":"2024-04-05T23:32:38.383002+0000","last_unstale":"2024-04-05T23:32:38.383367+0000","last_undegraded":"2024-04-05T23:32:38.383367+0000","last_fullsized":"2024-04-05T23:32:38.383367+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:30:59.947387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'30","reported_seq":128,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:46.478741+0000","last_change":"2024-04-05T23:32:04.007322+0000","last_active":"2024-04-05T23:32:46.478741+0000","last_peered":"2024-04-05T23:32:46.478741+0000","last_clean":"2024-04-05T23:32:46.478741+0000","last_became_active":"2024-04-05T23:32:04.006971+0000","last_became_peered":"2024-04-05T23:32:04.006971+0000","last_unstale":"2024-04-05T23:32:46.478741+0000","last_undegraded":"2024-04-05T23:32:46.478741+0000","last_fullsized":"2024-04-05T23:32:46.478741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:04:38.413482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59830,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":100,"num_write":32,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380649+0000","last_change":"2024-04-05T23:32:04.535342+0000","last_active":"2024-04-05T23:32:37.380649+0000","last_peered":"2024-04-05T23:32:37.380649+0000","last_clean":"2024-04-05T23:32:37.380649+0000","last_became_active":"2024-04-05T23:32:04.535053+0000","last_became_peered":"2024-04-05T23:32:04.535053+0000","last_unstale":"2024-04-05T23:32:37.380649+0000","last_undegraded":"2024-04-05T23:32:37.380649+0000","last_fullsized":"2024-04-05T23:32:37.380649+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:19:20.683836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.391798+0000","last_change":"2024-04-05T23:32:38.391418+0000","last_active":"2024-04-05T23:32:38.391798+0000","last_peered":"2024-04-05T23:32:38.391798+0000","last_clean":"2024-04-05T23:32:38.391798+0000","last_became_active":"2024-04-05T23:32:38.379643+0000","last_became_peered":"2024-04-05T23:32:38.379643+0000","last_unstale":"2024-04-05T23:32:38.391798+0000","last_undegraded":"2024-04-05T23:32:38.391798+0000","last_fullsized":"2024-04-05T23:32:38.391798+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:37:12.343153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378225+0000","last_change":"2024-04-05T23:32:04.536455+0000","last_active":"2024-04-05T23:32:37.378225+0000","last_peered":"2024-04-05T23:32:37.378225+0000","last_clean":"2024-04-05T23:32:37.378225+0000","last_became_active":"2024-04-05T23:32:04.533994+0000","last_became_peered":"2024-04-05T23:32:04.533994+0000","last_unstale":"2024-04-05T23:32:37.378225+0000","last_undegraded":"2024-04-05T23:32:37.378225+0000","last_fullsized":"2024-04-05T23:32:37.378225+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:38:18.952823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377668+0000","last_change":"2024-04-05T23:32:04.000708+0000","last_active":"2024-04-05T23:32:37.377668+0000","last_peered":"2024-04-05T23:32:37.377668+0000","last_clean":"2024-04-05T23:32:37.377668+0000","last_became_active":"2024-04-05T23:32:04.000354+0000","last_became_peered":"2024-04-05T23:32:04.000354+0000","last_unstale":"2024-04-05T23:32:37.377668+0000","last_undegraded":"2024-04-05T23:32:37.377668+0000","last_fullsized":"2024-04-05T23:32:37.377668+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:05:34.291607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378606+0000","last_change":"2024-04-05T23:32:04.532146+0000","last_active":"2024-04-05T23:32:37.378606+0000","last_peered":"2024-04-05T23:32:37.378606+0000","last_clean":"2024-04-05T23:32:37.378606+0000","last_became_active":"2024-04-05T23:32:04.531735+0000","last_became_peered":"2024-04-05T23:32:04.531735+0000","last_unstale":"2024-04-05T23:32:37.378606+0000","last_undegraded":"2024-04-05T23:32:37.378606+0000","last_fullsized":"2024-04-05T23:32:37.378606+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:52:44.898871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:46.473449+0000","last_change":"2024-04-05T23:32:04.000144+0000","last_active":"2024-04-05T23:32:46.473449+0000","last_peered":"2024-04-05T23:32:46.473449+0000","last_clean":"2024-04-05T23:32:46.473449+0000","last_became_active":"2024-04-05T23:32:03.999712+0000","last_became_peered":"2024-04-05T23:32:03.999712+0000","last_unstale":"2024-04-05T23:32:46.473449+0000","last_undegraded":"2024-04-05T23:32:46.473449+0000","last_fullsized":"2024-04-05T23:32:46.473449+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:43.775018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378466+0000","last_change":"2024-04-05T23:32:04.537623+0000","last_active":"2024-04-05T23:32:37.378466+0000","last_peered":"2024-04-05T23:32:37.378466+0000","last_clean":"2024-04-05T23:32:37.378466+0000","last_became_active":"2024-04-05T23:32:04.537399+0000","last_became_peered":"2024-04-05T23:32:04.537399+0000","last_unstale":"2024-04-05T23:32:37.378466+0000","last_undegraded":"2024-04-05T23:32:37.378466+0000","last_fullsized":"2024-04-05T23:32:37.378466+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:01:01.643133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380396+0000","last_change":"2024-04-05T23:32:04.006806+0000","last_active":"2024-04-05T23:32:37.380396+0000","last_peered":"2024-04-05T23:32:37.380396+0000","last_clean":"2024-04-05T23:32:37.380396+0000","last_became_active":"2024-04-05T23:32:04.006397+0000","last_became_peered":"2024-04-05T23:32:04.006397+0000","last_unstale":"2024-04-05T23:32:37.380396+0000","last_undegraded":"2024-04-05T23:32:37.380396+0000","last_fullsized":"2024-04-05T23:32:37.380396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:42:01.758863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377159+0000","last_change":"2024-04-05T23:32:04.545952+0000","last_active":"2024-04-05T23:32:37.377159+0000","last_peered":"2024-04-05T23:32:37.377159+0000","last_clean":"2024-04-05T23:32:37.377159+0000","last_became_active":"2024-04-05T23:32:04.545528+0000","last_became_peered":"2024-04-05T23:32:04.545528+0000","last_unstale":"2024-04-05T23:32:37.377159+0000","last_undegraded":"2024-04-05T23:32:37.377159+0000","last_fullsized":"2024-04-05T23:32:37.377159+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:33:06.074173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380812+0000","last_change":"2024-04-05T23:32:03.999841+0000","last_active":"2024-04-05T23:32:37.380812+0000","last_peered":"2024-04-05T23:32:37.380812+0000","last_clean":"2024-04-05T23:32:37.380812+0000","last_became_active":"2024-04-05T23:32:03.999418+0000","last_became_peered":"2024-04-05T23:32:03.999418+0000","last_unstale":"2024-04-05T23:32:37.380812+0000","last_undegraded":"2024-04-05T23:32:37.380812+0000","last_fullsized":"2024-04-05T23:32:37.380812+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:00:41.452167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371767+0000","last_change":"2024-04-05T23:32:04.530012+0000","last_active":"2024-04-05T23:32:37.371767+0000","last_peered":"2024-04-05T23:32:37.371767+0000","last_clean":"2024-04-05T23:32:37.371767+0000","last_became_active":"2024-04-05T23:32:04.529628+0000","last_became_peered":"2024-04-05T23:32:04.529628+0000","last_unstale":"2024-04-05T23:32:37.371767+0000","last_undegraded":"2024-04-05T23:32:37.371767+0000","last_fullsized":"2024-04-05T23:32:37.371767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:40:22.036170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375890+0000","last_change":"2024-04-05T23:32:04.006157+0000","last_active":"2024-04-05T23:32:37.375890+0000","last_peered":"2024-04-05T23:32:37.375890+0000","last_clean":"2024-04-05T23:32:37.375890+0000","last_became_active":"2024-04-05T23:32:04.005853+0000","last_became_peered":"2024-04-05T23:32:04.005853+0000","last_unstale":"2024-04-05T23:32:37.375890+0000","last_undegraded":"2024-04-05T23:32:37.375890+0000","last_fullsized":"2024-04-05T23:32:37.375890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:48:39.849306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375866+0000","last_change":"2024-04-05T23:32:04.547367+0000","last_active":"2024-04-05T23:32:37.375866+0000","last_peered":"2024-04-05T23:32:37.375866+0000","last_clean":"2024-04-05T23:32:37.375866+0000","last_became_active":"2024-04-05T23:32:04.547040+0000","last_became_peered":"2024-04-05T23:32:04.547040+0000","last_unstale":"2024-04-05T23:32:37.375866+0000","last_undegraded":"2024-04-05T23:32:37.375866+0000","last_fullsized":"2024-04-05T23:32:37.375866+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:41:41.998984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:44.493445+0000","last_change":"2024-04-05T23:32:04.009851+0000","last_active":"2024-04-05T23:32:44.493445+0000","last_peered":"2024-04-05T23:32:44.493445+0000","last_clean":"2024-04-05T23:32:44.493445+0000","last_became_active":"2024-04-05T23:32:04.009460+0000","last_became_peered":"2024-04-05T23:32:04.009460+0000","last_unstale":"2024-04-05T23:32:44.493445+0000","last_undegraded":"2024-04-05T23:32:44.493445+0000","last_fullsized":"2024-04-05T23:32:44.493445+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:39:41.000439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.376044+0000","last_change":"2024-04-05T23:32:04.549621+0000","last_active":"2024-04-05T23:32:37.376044+0000","last_peered":"2024-04-05T23:32:37.376044+0000","last_clean":"2024-04-05T23:32:37.376044+0000","last_became_active":"2024-04-05T23:32:04.549325+0000","last_became_peered":"2024-04-05T23:32:04.549325+0000","last_unstale":"2024-04-05T23:32:37.376044+0000","last_undegraded":"2024-04-05T23:32:37.376044+0000","last_fullsized":"2024-04-05T23:32:37.376044+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:11:07.101463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377151+0000","last_change":"2024-04-05T23:32:04.546768+0000","last_active":"2024-04-05T23:32:37.377151+0000","last_peered":"2024-04-05T23:32:37.377151+0000","last_clean":"2024-04-05T23:32:37.377151+0000","last_became_active":"2024-04-05T23:32:04.546471+0000","last_became_peered":"2024-04-05T23:32:04.546471+0000","last_unstale":"2024-04-05T23:32:37.377151+0000","last_undegraded":"2024-04-05T23:32:37.377151+0000","last_fullsized":"2024-04-05T23:32:37.377151+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:48:22.728976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:44.469521+0000","last_change":"2024-04-05T23:32:04.007093+0000","last_active":"2024-04-05T23:32:44.469521+0000","last_peered":"2024-04-05T23:32:44.469521+0000","last_clean":"2024-04-05T23:32:44.469521+0000","last_became_active":"2024-04-05T23:32:04.006836+0000","last_became_peered":"2024-04-05T23:32:04.006836+0000","last_unstale":"2024-04-05T23:32:44.469521+0000","last_undegraded":"2024-04-05T23:32:44.469521+0000","last_fullsized":"2024-04-05T23:32:44.469521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:20:42.489368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371662+0000","last_change":"2024-04-05T23:32:04.538204+0000","last_active":"2024-04-05T23:32:37.371662+0000","last_peered":"2024-04-05T23:32:37.371662+0000","last_clean":"2024-04-05T23:32:37.371662+0000","last_became_active":"2024-04-05T23:32:04.535254+0000","last_became_peered":"2024-04-05T23:32:04.535254+0000","last_unstale":"2024-04-05T23:32:37.371662+0000","last_undegraded":"2024-04-05T23:32:37.371662+0000","last_fullsized":"2024-04-05T23:32:37.371662+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:06:01.059339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377772+0000","last_change":"2024-04-05T23:32:03.997704+0000","last_active":"2024-04-05T23:32:37.377772+0000","last_peered":"2024-04-05T23:32:37.377772+0000","last_clean":"2024-04-05T23:32:37.377772+0000","last_became_active":"2024-04-05T23:32:03.997275+0000","last_became_peered":"2024-04-05T23:32:03.997275+0000","last_unstale":"2024-04-05T23:32:37.377772+0000","last_undegraded":"2024-04-05T23:32:37.377772+0000","last_fullsized":"2024-04-05T23:32:37.377772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:58.355104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":336,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378679+0000","last_change":"2024-04-05T23:31:10.395041+0000","last_active":"2024-04-05T23:32:37.378679+0000","last_peered":"2024-04-05T23:32:37.378679+0000","last_clean":"2024-04-05T23:32:37.378679+0000","last_became_active":"2024-04-05T23:31:10.394479+0000","last_became_peered":"2024-04-05T23:31:10.394479+0000","last_unstale":"2024-04-05T23:32:37.378679+0000","last_undegraded":"2024-04-05T23:32:37.378679+0000","last_fullsized":"2024-04-05T23:32:37.378679+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_clean_scrub_stamp":"2024-04-05T23:28:33.945697+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:54:10.683526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.385626+0000","last_change":"2024-04-05T23:32:38.385626+0000","last_active":"2024-04-05T23:32:38.385626+0000","last_peered":"2024-04-05T23:32:38.385626+0000","last_clean":"2024-04-05T23:32:38.385626+0000","last_became_active":"2024-04-05T23:32:38.384654+0000","last_became_peered":"2024-04-05T23:32:38.384654+0000","last_unstale":"2024-04-05T23:32:38.385626+0000","last_undegraded":"2024-04-05T23:32:38.385626+0000","last_fullsized":"2024-04-05T23:32:38.385626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:55:42.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378881+0000","last_change":"2024-04-05T23:32:04.005978+0000","last_active":"2024-04-05T23:32:37.378881+0000","last_peered":"2024-04-05T23:32:37.378881+0000","last_clean":"2024-04-05T23:32:37.378881+0000","last_became_active":"2024-04-05T23:32:04.005638+0000","last_became_peered":"2024-04-05T23:32:04.005638+0000","last_unstale":"2024-04-05T23:32:37.378881+0000","last_undegraded":"2024-04-05T23:32:37.378881+0000","last_fullsized":"2024-04-05T23:32:37.378881+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:39:35.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380727+0000","last_change":"2024-04-05T23:32:04.006727+0000","last_active":"2024-04-05T23:32:37.380727+0000","last_peered":"2024-04-05T23:32:37.380727+0000","last_clean":"2024-04-05T23:32:37.380727+0000","last_became_active":"2024-04-05T23:32:04.006414+0000","last_became_peered":"2024-04-05T23:32:04.006414+0000","last_unstale":"2024-04-05T23:32:37.380727+0000","last_undegraded":"2024-04-05T23:32:37.380727+0000","last_fullsized":"2024-04-05T23:32:37.380727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:41:28.661118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378376+0000","last_change":"2024-04-05T23:32:04.536893+0000","last_active":"2024-04-05T23:32:37.378376+0000","last_peered":"2024-04-05T23:32:37.378376+0000","last_clean":"2024-04-05T23:32:37.378376+0000","last_became_active":"2024-04-05T23:32:04.536629+0000","last_became_peered":"2024-04-05T23:32:04.536629+0000","last_unstale":"2024-04-05T23:32:37.378376+0000","last_undegraded":"2024-04-05T23:32:37.378376+0000","last_fullsized":"2024-04-05T23:32:37.378376+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:27:58.279768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'10","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375826+0000","last_change":"2024-04-05T23:32:04.010202+0000","last_active":"2024-04-05T23:32:37.375826+0000","last_peered":"2024-04-05T23:32:37.375826+0000","last_clean":"2024-04-05T23:32:37.375826+0000","last_became_active":"2024-04-05T23:32:04.009889+0000","last_became_peered":"2024-04-05T23:32:04.009889+0000","last_unstale":"2024-04-05T23:32:37.375826+0000","last_undegraded":"2024-04-05T23:32:37.375826+0000","last_fullsized":"2024-04-05T23:32:37.375826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:12:13.121449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371893+0000","last_change":"2024-04-05T23:32:04.533576+0000","last_active":"2024-04-05T23:32:37.371893+0000","last_peered":"2024-04-05T23:32:37.371893+0000","last_clean":"2024-04-05T23:32:37.371893+0000","last_became_active":"2024-04-05T23:32:04.533066+0000","last_became_peered":"2024-04-05T23:32:04.533066+0000","last_unstale":"2024-04-05T23:32:37.371893+0000","last_undegraded":"2024-04-05T23:32:37.371893+0000","last_fullsized":"2024-04-05T23:32:37.371893+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:01:44.019210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377470+0000","last_change":"2024-04-05T23:32:04.005096+0000","last_active":"2024-04-05T23:32:37.377470+0000","last_peered":"2024-04-05T23:32:37.377470+0000","last_clean":"2024-04-05T23:32:37.377470+0000","last_became_active":"2024-04-05T23:32:04.004702+0000","last_became_peered":"2024-04-05T23:32:04.004702+0000","last_unstale":"2024-04-05T23:32:37.377470+0000","last_undegraded":"2024-04-05T23:32:37.377470+0000","last_fullsized":"2024-04-05T23:32:37.377470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:33:09.934039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380697+0000","last_change":"2024-04-05T23:32:04.535623+0000","last_active":"2024-04-05T23:32:37.380697+0000","last_peered":"2024-04-05T23:32:37.380697+0000","last_clean":"2024-04-05T23:32:37.380697+0000","last_became_active":"2024-04-05T23:32:04.535223+0000","last_became_peered":"2024-04-05T23:32:04.535223+0000","last_unstale":"2024-04-05T23:32:37.380697+0000","last_undegraded":"2024-04-05T23:32:37.380697+0000","last_fullsized":"2024-04-05T23:32:37.380697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:24:43.543119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380978+0000","last_change":"2024-04-05T23:32:04.004097+0000","last_active":"2024-04-05T23:32:37.380978+0000","last_peered":"2024-04-05T23:32:37.380978+0000","last_clean":"2024-04-05T23:32:37.380978+0000","last_became_active":"2024-04-05T23:32:04.003713+0000","last_became_peered":"2024-04-05T23:32:04.003713+0000","last_unstale":"2024-04-05T23:32:37.380978+0000","last_undegraded":"2024-04-05T23:32:37.380978+0000","last_fullsized":"2024-04-05T23:32:37.380978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:47:43.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380375+0000","last_change":"2024-04-05T23:32:04.538843+0000","last_active":"2024-04-05T23:32:37.380375+0000","last_peered":"2024-04-05T23:32:37.380375+0000","last_clean":"2024-04-05T23:32:37.380375+0000","last_became_active":"2024-04-05T23:32:04.538472+0000","last_became_peered":"2024-04-05T23:32:04.538472+0000","last_unstale":"2024-04-05T23:32:37.380375+0000","last_undegraded":"2024-04-05T23:32:37.380375+0000","last_fullsized":"2024-04-05T23:32:37.380375+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:40:40.863815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371533+0000","last_change":"2024-04-05T23:32:04.013064+0000","last_active":"2024-04-05T23:32:37.371533+0000","last_peered":"2024-04-05T23:32:37.371533+0000","last_clean":"2024-04-05T23:32:37.371533+0000","last_became_active":"2024-04-05T23:32:04.012751+0000","last_became_peered":"2024-04-05T23:32:04.012751+0000","last_unstale":"2024-04-05T23:32:37.371533+0000","last_undegraded":"2024-04-05T23:32:37.371533+0000","last_fullsized":"2024-04-05T23:32:37.371533+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:06:10.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371497+0000","last_change":"2024-04-05T23:32:04.529014+0000","last_active":"2024-04-05T23:32:37.371497+0000","last_peered":"2024-04-05T23:32:37.371497+0000","last_clean":"2024-04-05T23:32:37.371497+0000","last_became_active":"2024-04-05T23:32:04.528622+0000","last_became_peered":"2024-04-05T23:32:04.528622+0000","last_unstale":"2024-04-05T23:32:37.371497+0000","last_undegraded":"2024-04-05T23:32:37.371497+0000","last_fullsized":"2024-04-05T23:32:37.371497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:46:26.390934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375037+0000","last_change":"2024-04-05T23:32:04.002651+0000","last_active":"2024-04-05T23:32:37.375037+0000","last_peered":"2024-04-05T23:32:37.375037+0000","last_clean":"2024-04-05T23:32:37.375037+0000","last_became_active":"2024-04-05T23:32:04.002346+0000","last_became_peered":"2024-04-05T23:32:04.002346+0000","last_unstale":"2024-04-05T23:32:37.375037+0000","last_undegraded":"2024-04-05T23:32:37.375037+0000","last_fullsized":"2024-04-05T23:32:37.375037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:26:48.176254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371637+0000","last_change":"2024-04-05T23:32:04.538785+0000","last_active":"2024-04-05T23:32:37.371637+0000","last_peered":"2024-04-05T23:32:37.371637+0000","last_clean":"2024-04-05T23:32:37.371637+0000","last_became_active":"2024-04-05T23:32:04.538463+0000","last_became_peered":"2024-04-05T23:32:04.538463+0000","last_unstale":"2024-04-05T23:32:37.371637+0000","last_undegraded":"2024-04-05T23:32:37.371637+0000","last_fullsized":"2024-04-05T23:32:37.371637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:44:38.941857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380620+0000","last_change":"2024-04-05T23:32:04.003760+0000","last_active":"2024-04-05T23:32:37.380620+0000","last_peered":"2024-04-05T23:32:37.380620+0000","last_clean":"2024-04-05T23:32:37.380620+0000","last_became_active":"2024-04-05T23:32:04.003389+0000","last_became_peered":"2024-04-05T23:32:04.003389+0000","last_unstale":"2024-04-05T23:32:37.380620+0000","last_undegraded":"2024-04-05T23:32:37.380620+0000","last_fullsized":"2024-04-05T23:32:37.380620+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:42:00.353651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.379178+0000","last_change":"2024-04-05T23:32:04.540051+0000","last_active":"2024-04-05T23:32:37.379178+0000","last_peered":"2024-04-05T23:32:37.379178+0000","last_clean":"2024-04-05T23:32:37.379178+0000","last_became_active":"2024-04-05T23:32:04.539736+0000","last_became_peered":"2024-04-05T23:32:04.539736+0000","last_unstale":"2024-04-05T23:32:37.379178+0000","last_undegraded":"2024-04-05T23:32:37.379178+0000","last_fullsized":"2024-04-05T23:32:37.379178+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:54:14.086904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:46.497151+0000","last_change":"2024-04-05T23:32:04.006492+0000","last_active":"2024-04-05T23:32:46.497151+0000","last_peered":"2024-04-05T23:32:46.497151+0000","last_clean":"2024-04-05T23:32:46.497151+0000","last_became_active":"2024-04-05T23:32:04.006154+0000","last_became_peered":"2024-04-05T23:32:04.006154+0000","last_unstale":"2024-04-05T23:32:46.497151+0000","last_undegraded":"2024-04-05T23:32:46.497151+0000","last_fullsized":"2024-04-05T23:32:46.497151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:53:23.431400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378846+0000","last_change":"2024-04-05T23:32:04.540757+0000","last_active":"2024-04-05T23:32:37.378846+0000","last_peered":"2024-04-05T23:32:37.378846+0000","last_clean":"2024-04-05T23:32:37.378846+0000","last_became_active":"2024-04-05T23:32:04.540359+0000","last_became_peered":"2024-04-05T23:32:04.540359+0000","last_unstale":"2024-04-05T23:32:37.378846+0000","last_undegraded":"2024-04-05T23:32:37.378846+0000","last_fullsized":"2024-04-05T23:32:37.378846+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:43:52.533489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378021+0000","last_change":"2024-04-05T23:32:04.002736+0000","last_active":"2024-04-05T23:32:37.378021+0000","last_peered":"2024-04-05T23:32:37.378021+0000","last_clean":"2024-04-05T23:32:37.378021+0000","last_became_active":"2024-04-05T23:32:04.002277+0000","last_became_peered":"2024-04-05T23:32:04.002277+0000","last_unstale":"2024-04-05T23:32:37.378021+0000","last_undegraded":"2024-04-05T23:32:37.378021+0000","last_fullsized":"2024-04-05T23:32:37.378021+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:56:39.001732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378607+0000","last_change":"2024-04-05T23:32:04.537076+0000","last_active":"2024-04-05T23:32:37.378607+0000","last_peered":"2024-04-05T23:32:37.378607+0000","last_clean":"2024-04-05T23:32:37.378607+0000","last_became_active":"2024-04-05T23:32:04.536707+0000","last_became_peered":"2024-04-05T23:32:04.536707+0000","last_unstale":"2024-04-05T23:32:37.378607+0000","last_undegraded":"2024-04-05T23:32:37.378607+0000","last_fullsized":"2024-04-05T23:32:37.378607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:51:09.395408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.379112+0000","last_change":"2024-04-05T23:32:04.006122+0000","last_active":"2024-04-05T23:32:37.379112+0000","last_peered":"2024-04-05T23:32:37.379112+0000","last_clean":"2024-04-05T23:32:37.379112+0000","last_became_active":"2024-04-05T23:32:04.005683+0000","last_became_peered":"2024-04-05T23:32:04.005683+0000","last_unstale":"2024-04-05T23:32:37.379112+0000","last_undegraded":"2024-04-05T23:32:37.379112+0000","last_fullsized":"2024-04-05T23:32:37.379112+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:21:40.922614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378878+0000","last_change":"2024-04-05T23:32:04.533321+0000","last_active":"2024-04-05T23:32:37.378878+0000","last_peered":"2024-04-05T23:32:37.378878+0000","last_clean":"2024-04-05T23:32:37.378878+0000","last_became_active":"2024-04-05T23:32:04.532949+0000","last_became_peered":"2024-04-05T23:32:04.532949+0000","last_unstale":"2024-04-05T23:32:37.378878+0000","last_undegraded":"2024-04-05T23:32:37.378878+0000","last_fullsized":"2024-04-05T23:32:37.378878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:31.500132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371547+0000","last_change":"2024-04-05T23:32:04.536508+0000","last_active":"2024-04-05T23:32:37.371547+0000","last_peered":"2024-04-05T23:32:37.371547+0000","last_clean":"2024-04-05T23:32:37.371547+0000","last_became_active":"2024-04-05T23:32:04.536142+0000","last_became_peered":"2024-04-05T23:32:04.536142+0000","last_unstale":"2024-04-05T23:32:37.371547+0000","last_undegraded":"2024-04-05T23:32:37.371547+0000","last_fullsized":"2024-04-05T23:32:37.371547+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:18:28.497807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375905+0000","last_change":"2024-04-05T23:32:04.545511+0000","last_active":"2024-04-05T23:32:37.375905+0000","last_peered":"2024-04-05T23:32:37.375905+0000","last_clean":"2024-04-05T23:32:37.375905+0000","last_became_active":"2024-04-05T23:32:04.545191+0000","last_became_peered":"2024-04-05T23:32:04.545191+0000","last_unstale":"2024-04-05T23:32:37.375905+0000","last_undegraded":"2024-04-05T23:32:37.375905+0000","last_fullsized":"2024-04-05T23:32:37.375905+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:17:06.431429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375067+0000","last_change":"2024-04-05T23:32:04.549673+0000","last_active":"2024-04-05T23:32:37.375067+0000","last_peered":"2024-04-05T23:32:37.375067+0000","last_clean":"2024-04-05T23:32:37.375067+0000","last_became_active":"2024-04-05T23:32:04.549391+0000","last_became_peered":"2024-04-05T23:32:04.549391+0000","last_unstale":"2024-04-05T23:32:37.375067+0000","last_undegraded":"2024-04-05T23:32:37.375067+0000","last_fullsized":"2024-04-05T23:32:37.375067+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:30:22.689107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380491+0000","last_change":"2024-04-05T23:32:04.535735+0000","last_active":"2024-04-05T23:32:37.380491+0000","last_peered":"2024-04-05T23:32:37.380491+0000","last_clean":"2024-04-05T23:32:37.380491+0000","last_became_active":"2024-04-05T23:32:04.534461+0000","last_became_peered":"2024-04-05T23:32:04.534461+0000","last_unstale":"2024-04-05T23:32:37.380491+0000","last_undegraded":"2024-04-05T23:32:37.380491+0000","last_fullsized":"2024-04-05T23:32:37.380491+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:37:41.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378465+0000","last_change":"2024-04-05T23:32:04.532340+0000","last_active":"2024-04-05T23:32:37.378465+0000","last_peered":"2024-04-05T23:32:37.378465+0000","last_clean":"2024-04-05T23:32:37.378465+0000","last_became_active":"2024-04-05T23:32:04.530053+0000","last_became_peered":"2024-04-05T23:32:04.530053+0000","last_unstale":"2024-04-05T23:32:37.378465+0000","last_undegraded":"2024-04-05T23:32:37.378465+0000","last_fullsized":"2024-04-05T23:32:37.378465+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:16:11.168366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371640+0000","last_change":"2024-04-05T23:32:04.529972+0000","last_active":"2024-04-05T23:32:37.371640+0000","last_peered":"2024-04-05T23:32:37.371640+0000","last_clean":"2024-04-05T23:32:37.371640+0000","last_became_active":"2024-04-05T23:32:04.529551+0000","last_became_peered":"2024-04-05T23:32:04.529551+0000","last_unstale":"2024-04-05T23:32:37.371640+0000","last_undegraded":"2024-04-05T23:32:37.371640+0000","last_fullsized":"2024-04-05T23:32:37.371640+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:13.764971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375667+0000","last_change":"2024-04-05T23:32:04.529166+0000","last_active":"2024-04-05T23:32:37.375667+0000","last_peered":"2024-04-05T23:32:37.375667+0000","last_clean":"2024-04-05T23:32:37.375667+0000","last_became_active":"2024-04-05T23:32:04.528785+0000","last_became_peered":"2024-04-05T23:32:04.528785+0000","last_unstale":"2024-04-05T23:32:37.375667+0000","last_undegraded":"2024-04-05T23:32:37.375667+0000","last_fullsized":"2024-04-05T23:32:37.375667+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:29:43.964265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371809+0000","last_change":"2024-04-05T23:32:04.540631+0000","last_active":"2024-04-05T23:32:37.371809+0000","last_peered":"2024-04-05T23:32:37.371809+0000","last_clean":"2024-04-05T23:32:37.371809+0000","last_became_active":"2024-04-05T23:32:04.540327+0000","last_became_peered":"2024-04-05T23:32:04.540327+0000","last_unstale":"2024-04-05T23:32:37.371809+0000","last_undegraded":"2024-04-05T23:32:37.371809+0000","last_fullsized":"2024-04-05T23:32:37.371809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:22:48.825324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377032+0000","last_change":"2024-04-05T23:32:04.544845+0000","last_active":"2024-04-05T23:32:37.377032+0000","last_peered":"2024-04-05T23:32:37.377032+0000","last_clean":"2024-04-05T23:32:37.377032+0000","last_became_active":"2024-04-05T23:32:04.544472+0000","last_became_peered":"2024-04-05T23:32:04.544472+0000","last_unstale":"2024-04-05T23:32:37.377032+0000","last_undegraded":"2024-04-05T23:32:37.377032+0000","last_fullsized":"2024-04-05T23:32:37.377032+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:34:09.529644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.383681+0000","last_change":"2024-04-05T23:32:38.382577+0000","last_active":"2024-04-05T23:32:38.383681+0000","last_peered":"2024-04-05T23:32:38.383681+0000","last_clean":"2024-04-05T23:32:38.383681+0000","last_became_active":"2024-04-05T23:32:38.380286+0000","last_became_peered":"2024-04-05T23:32:38.380286+0000","last_unstale":"2024-04-05T23:32:38.383681+0000","last_undegraded":"2024-04-05T23:32:38.383681+0000","last_fullsized":"2024-04-05T23:32:38.383681+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:42:52.270935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380994+0000","last_change":"2024-04-05T23:32:04.534631+0000","last_active":"2024-04-05T23:32:37.380994+0000","last_peered":"2024-04-05T23:32:37.380994+0000","last_clean":"2024-04-05T23:32:37.380994+0000","last_became_active":"2024-04-05T23:32:04.534330+0000","last_became_peered":"2024-04-05T23:32:04.534330+0000","last_unstale":"2024-04-05T23:32:37.380994+0000","last_undegraded":"2024-04-05T23:32:37.380994+0000","last_fullsized":"2024-04-05T23:32:37.380994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:20:47.592174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378464+0000","last_change":"2024-04-05T23:32:04.531459+0000","last_active":"2024-04-05T23:32:37.378464+0000","last_peered":"2024-04-05T23:32:37.378464+0000","last_clean":"2024-04-05T23:32:37.378464+0000","last_became_active":"2024-04-05T23:32:04.530828+0000","last_became_peered":"2024-04-05T23:32:04.530828+0000","last_unstale":"2024-04-05T23:32:37.378464+0000","last_undegraded":"2024-04-05T23:32:37.378464+0000","last_fullsized":"2024-04-05T23:32:37.378464+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:51:34.090030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371594+0000","last_change":"2024-04-05T23:32:04.527711+0000","last_active":"2024-04-05T23:32:37.371594+0000","last_peered":"2024-04-05T23:32:37.371594+0000","last_clean":"2024-04-05T23:32:37.371594+0000","last_became_active":"2024-04-05T23:32:04.527374+0000","last_became_peered":"2024-04-05T23:32:04.527374+0000","last_unstale":"2024-04-05T23:32:37.371594+0000","last_undegraded":"2024-04-05T23:32:37.371594+0000","last_fullsized":"2024-04-05T23:32:37.371594+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:15:10.764340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66060,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":404,"num_read_kb":273,"num_write":232,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":198180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579923,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":897024,"data_stored":430251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":3.968}]},{"osd":1,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":4,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.748},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.54800000000000004}]},{"osd":5,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.63400000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.61199999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.621},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":0.67200000000000004}]}]},{"osd":10,"up_from":67,"seq":287762808855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1589248,"data_stored":1081166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.503}]},{"osd":2,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.83299999999999996}]},{"osd":4,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.441},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.38800000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.77500000000000002}]},{"osd":6,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.74399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.61499999999999999}]},{"osd":9,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.68300000000000005}]},{"osd":11,"last update":"Fri Apr 5 23:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.73799999999999999}]}]},{"osd":9,"up_from":62,"seq":266287972378,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":430285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.35699999999999998}]},{"osd":1,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.64600000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.0089999999999999}]},{"osd":5,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.1299999999999999}]},{"osd":6,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.0549999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.0920000000000001}]},{"osd":10,"last update":"Fri Apr 5 23:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.83399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]}]},{"osd":8,"up_from":55,"seq":236223201311,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":430081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71699999999999997,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.36899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77800000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.495}]},{"osd":1,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.45800000000000002,"15min":0.45700000000000002},"min":{"1min":0.30199999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69699999999999995,"5min":0.748,"15min":0.748},"last":0.46999999999999997}]},{"osd":3,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.38900000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76200000000000001,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.432}]},{"osd":6,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68799999999999994,"15min":0.69199999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.69999999999999996,"15min":0.70699999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90100000000000002,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.755}]},{"osd":7,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.72399999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.68999999999999995}]},{"osd":9,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71899999999999997}]},{"osd":10,"last update":"Fri Apr 5 23:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.74399999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]}]},{"osd":7,"up_from":48,"seq":206158430242,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32592,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712560,"statfs":{"total":95995035648,"available":95961661440,"internally_reserved":0,"allocated":1507328,"data_stored":1022646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.76400000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.433,"15min":0.42299999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42399999999999999,"15min":0.41899999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.496}]},{"osd":2,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55700000000000005,"15min":0.55600000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48899999999999999}]},{"osd":3,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.621,"15min":0.60999999999999999},"min":{"1min":0.44700000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.60699999999999998}]},{"osd":4,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.32900000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.38100000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.625,"15min":0.623},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.438},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.65400000000000003,"15min":0.65000000000000002},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.51500000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.69399999999999995,"15min":0.68400000000000005},"min":{"1min":0.498,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68899999999999995,"15min":0.68100000000000005},"min":{"1min":0.502,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.47999999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.67600000000000005,"15min":0.68600000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72699999999999998,"15min":0.72799999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.89500000000000002,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.54300000000000004}]},{"osd":9,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.68700000000000006}]},{"osd":10,"last update":"Fri Apr 5 23:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.63},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.73599999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]},{"osd":6,"up_from":42,"seq":180388626470,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":429681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47799999999999998,"15min":0.47499999999999998},"min":{"1min":0.34000000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73499999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.505,"15min":0.5},"min":{"1min":0.36399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.79600000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.623}]},{"osd":1,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.45900000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90400000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.68400000000000005}]},{"osd":2,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.627,"15min":0.627},"min":{"1min":0.39500000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92400000000000004,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.41799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85599999999999998,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.80600000000000005}]},{"osd":3,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78500000000000003,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.745},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67400000000000004,"15min":0.66900000000000004},"min":{"1min":0.47299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.92800000000000005,"5min":1.246,"15min":1.246},"last":0.64700000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.50600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.90200000000000002,"5min":1.095,"15min":1.095},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.754,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.49099999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.016,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.73899999999999999},"min":{"1min":0.54900000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0509999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":4.8869999999999996}]},{"osd":7,"last update":"Fri Apr 5 23:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.77200000000000002,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.94599999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0089999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":4.8559999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78400000000000003,"15min":0.79100000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.026,"5min":1.127,"15min":1.127},"last":4.8719999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.80000000000000004,"15min":0.79200000000000004},"min":{"1min":0.64000000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.78900000000000003}]},{"osd":9,"last update":"Fri Apr 5 23:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.89100000000000001}]},{"osd":10,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":1.012},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.93899999999999995}]}]},{"osd":1,"up_from":12,"seq":51539607609,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":954368,"data_stored":490424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.51600000000000001,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80900000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63500000000000001,"15min":0.68300000000000005},"min":{"1min":0.41399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.0069999999999999,"5min":4.4199999999999999,"15min":4.4199999999999999},"last":0.79300000000000004}]},{"osd":2,"last update":"Fri Apr 5 23:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55200000000000005,"15min":0.52000000000000002},"min":{"1min":0.34699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.57299999999999995,"15min":0.52100000000000002},"min":{"1min":0.53400000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68100000000000005}]},{"osd":3,"last update":"Fri Apr 5 23:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.56299999999999994,"15min":0.53400000000000003},"min":{"1min":0.44700000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77500000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.42799999999999999,"15min":0.40500000000000003},"min":{"1min":0.33700000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.495},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.46800000000000003,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.877,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.61399999999999999,"15min":0.59299999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.68300000000000005,"15min":0.66200000000000003},"min":{"1min":0.52100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.86099999999999999}]},{"osd":6,"last update":"Fri Apr 5 23:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.68700000000000006,"15min":0.67900000000000005},"min":{"1min":0.5,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65100000000000002,"15min":0.64800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80600000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.621,"15min":0.60599999999999998},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.505},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.38600000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.85499999999999998,"5min":1.002,"15min":1.002},"last":0.41399999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.624,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.72499999999999998,"15min":0.70899999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.083,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.75600000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:31:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.63400000000000001}]},{"osd":10,"last update":"Fri Apr 5 23:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36112,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709040,"statfs":{"total":95995035648,"available":95958056960,"internally_reserved":0,"allocated":917504,"data_stored":431293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 23:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.036,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53700000000000003,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82299999999999995,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.63200000000000001,"15min":0.58799999999999997},"min":{"1min":0.47999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0049999999999999,"5min":7.2060000000000004,"15min":7.2060000000000004},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:30:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.54200000000000004,"15min":0.52800000000000002},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.85699999999999998,"15min":0.93700000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82499999999999996,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67800000000000005,"15min":0.65600000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.052,"5min":1.395,"15min":1.395},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68899999999999995,"15min":0.66800000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.70999999999999996},"min":{"1min":0.55900000000000005,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.70099999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.88100000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.153,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.77800000000000002,"15min":0.77500000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.71099999999999997}]},{"osd":7,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.70699999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.92300000000000004,"15min":0.93999999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.98199999999999998,"15min":1.01},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1379999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.83299999999999996}]},{"osd":9,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.879},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.81899999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:32:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.90500000000000003}]},{"osd":11,"last update":"Fri Apr 5 23:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.752},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.84499999999999997}]}]},{"osd":2,"up_from":17,"seq":73014444087,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":966656,"data_stored":490528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76400000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.751,"5min":0.874,"15min":0.874},"last":0.76500000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55500000000000005,"15min":0.53800000000000003},"min":{"1min":0.37,"5min":0.311,"15min":0.311},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:31:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.67600000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.54500000000000004},"min":{"1min":0.43099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.72099999999999997}]},{"osd":5,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.503,"15min":0.47799999999999998},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86699999999999999,"5min":1.359,"15min":1.359},"last":0.61599999999999999}]},{"osd":6,"last update":"Fri Apr 5 23:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78500000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93899999999999995,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.66500000000000004}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73199999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.48399999999999999,"15min":0.47299999999999998},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70499999999999996,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Apr 5 23:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.80500000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.81699999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":3,"up_from":23,"seq":98784247858,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36104,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709048,"statfs":{"total":95995035648,"available":95958065152,"internally_reserved":0,"allocated":909312,"data_stored":429554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.45100000000000001},"min":{"1min":0.33500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.018,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51000000000000001,"15min":0.46300000000000002},"min":{"1min":0.379,"5min":0.313,"15min":0.313},"max":{"1min":1.0860000000000001,"5min":4.96,"15min":4.96},"last":0.379}]},{"osd":1,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64000000000000001,"15min":0.60599999999999998},"min":{"1min":0.44400000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91200000000000003,"5min":4.5590000000000002,"15min":4.5590000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.61599999999999999,"15min":0.59599999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71599999999999997,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.72799999999999998,"15min":0.67800000000000005},"min":{"1min":0.51400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.028,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71099999999999997,"15min":0.67300000000000004},"min":{"1min":0.57199999999999995,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94399999999999995,"5min":4.6710000000000003,"15min":4.6710000000000003},"last":0.53200000000000003}]},{"osd":4,"last update":"Fri Apr 5 23:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71299999999999997,"15min":0.66900000000000004},"min":{"1min":0.46700000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1479999999999999,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.71599999999999997,"15min":0.66600000000000004},"min":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.028,"5min":4.6289999999999996,"15min":4.6289999999999996},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Apr 5 23:32:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.752,"15min":0.69799999999999995},"min":{"1min":0.56000000000000005,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0860000000000001,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.77800000000000002,"15min":0.72599999999999998},"min":{"1min":0.42799999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.069,"5min":4.8929999999999998,"15min":4.8929999999999998},"last":0.47999999999999998}]},{"osd":6,"last update":"Fri Apr 5 23:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.878,"15min":0.88300000000000001},"min":{"1min":0.63400000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.228,"5min":4.8159999999999998,"15min":4.8159999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.78600000000000003,"15min":0.81399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":5.0979999999999999,"15min":5.0979999999999999},"last":0.72699999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.97499999999999998,"15min":1},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2929999999999999,"5min":4.9980000000000002,"15min":4.9980000000000002},"last":0.749},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.93500000000000005,"15min":0.96299999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1020000000000001,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.77200000000000002}]},{"osd":8,"last update":"Fri Apr 5 23:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":4.7569999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.0300000000000002,"5min":5.0300000000000002,"15min":5.0300000000000002},"last":0.72199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.57099999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.84199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]}]},{"osd":4,"up_from":30,"seq":128849018927,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.38600000000000001,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.443},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.40200000000000002,"15min":0.38500000000000001},"min":{"1min":0.27200000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Apr 5 23:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.46100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Apr 5 23:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.58799999999999997,"15min":0.56599999999999995},"min":{"1min":0.438,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.627},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57099999999999995,"15min":0.55900000000000005},"min":{"1min":0.35899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63800000000000001,"15min":0.60899999999999999},"min":{"1min":0.53300000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.77100000000000002}]},{"osd":6,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.61299999999999999,"15min":0.59499999999999997},"min":{"1min":0.435,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.56899999999999995,"15min":0.55500000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.60299999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.99299999999999999,"5min":1.087,"15min":1.087},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.754,"15min":0.746},"min":{"1min":0.53200000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0069999999999999,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.877}]},{"osd":9,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.67600000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.80800000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]}]},{"osd":5,"up_from":35,"seq":150323855401,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28508,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716644,"statfs":{"total":95995035648,"available":95965843456,"internally_reserved":0,"allocated":1519616,"data_stored":1022420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.40300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.78800000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53400000000000003,"15min":0.52400000000000002},"min":{"1min":0.36799999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72299999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.40600000000000003,"15min":0.40200000000000002},"min":{"1min":0.31,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66000000000000003,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.36199999999999999}]},{"osd":3,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75600000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86199999999999999,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.77400000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56399999999999995,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77500000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.46000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.66400000000000003}]},{"osd":6,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65700000000000003,"15min":0.65900000000000003},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.78900000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77900000000000003,"5min":1.153,"15min":1.153},"last":0.67700000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.375,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60799999999999998,"15min":0.60899999999999999},"min":{"1min":0.41899999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54500000000000004}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.81100000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.47099999999999997}]},{"osd":9,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.82399999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T23:32:50.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-05T23:32:50.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:50 smithi130 ceph-mon[79417]: pgmap v257: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 59 B/s, 0 keys/s, 1 objects/s recovering 2024-04-05T23:32:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:51.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:51 smithi016 ceph-mon[38248]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:51.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:51 smithi027 ceph-mon[78597]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:51.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:51 smithi130 ceph-mon[79417]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:52.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.310+0000 7f73d7200700 1 -- 172.21.15.16:0/689149166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d0106f70 msgr2=0x7f73d0107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:52.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.310+0000 7f73d7200700 1 --2- 172.21.15.16:0/689149166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d0106f70 0x7f73d0107350 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009990 tx=0x7f73c0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.310+0000 7f73d7200700 1 -- 172.21.15.16:0/689149166 shutdown_connections 2024-04-05T23:32:52.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.310+0000 7f73d7200700 1 --2- 172.21.15.16:0/689149166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d01018d0 0x7f73d010ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.310+0000 7f73d7200700 1 --2- 172.21.15.16:0/689149166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d0106f70 0x7f73d0107350 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.310+0000 7f73d7200700 1 --2- 172.21.15.16:0/689149166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d0100f30 0x7f73d0101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.310+0000 7f73d7200700 1 -- 172.21.15.16:0/689149166 >> 172.21.15.16:0/689149166 conn(0x7f73d00780d0 msgr2=0x7f73d00784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:52.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.310+0000 7f73d7200700 1 -- 172.21.15.16:0/689149166 shutdown_connections 2024-04-05T23:32:52.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.310+0000 7f73d7200700 1 -- 172.21.15.16:0/689149166 wait complete. 2024-04-05T23:32:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.311+0000 7f73d7200700 1 Processor -- start 2024-04-05T23:32:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73d7200700 1 -- start start 2024-04-05T23:32:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73d7200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d0100f30 0x7f73d0199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73d7200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01018d0 0x7f73d0199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73d7200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d0106f70 0x7f73d019de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73d7200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d019e420 con 0x7f73d01018d0 2024-04-05T23:32:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73d7200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d019e590 con 0x7f73d0100f30 2024-04-05T23:32:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73d7200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d019e750 con 0x7f73d0106f70 2024-04-05T23:32:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73cf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01018d0 0x7f73d0199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73cfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d0100f30 0x7f73d0199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73cf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01018d0 0x7f73d0199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55564/0 (socket says 172.21.15.16:55564) 2024-04-05T23:32:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73cf400700 1 -- 172.21.15.16:0/3946735285 learned_addr learned my addr 172.21.15.16:0/3946735285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.312+0000 7f73d5200700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d0106f70 0x7f73d019de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73cf400700 1 -- 172.21.15.16:0/3946735285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d0106f70 msgr2=0x7f73d019de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73cf400700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d0106f70 0x7f73d019de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73cf400700 1 -- 172.21.15.16:0/3946735285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d0100f30 msgr2=0x7f73d0199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73cf400700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d0100f30 0x7f73d0199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73cf400700 1 -- 172.21.15.16:0/3946735285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc009520 con 0x7f73d01018d0 2024-04-05T23:32:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73d5200700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d0106f70 0x7f73d019de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73cf400700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01018d0 0x7f73d0199a60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00eb30 tx=0x7f73bc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73ccc00700 1 -- 172.21.15.16:0/3946735285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc009880 con 0x7f73d01018d0 2024-04-05T23:32:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73ccc00700 1 -- 172.21.15.16:0/3946735285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73bc0044e0 con 0x7f73d01018d0 2024-04-05T23:32:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c00095f0 con 0x7f73d01018d0 2024-04-05T23:32:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73ccc00700 1 -- 172.21.15.16:0/3946735285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc003d80 con 0x7f73d01018d0 2024-04-05T23:32:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.313+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d01a60c0 con 0x7f73d01018d0 2024-04-05T23:32:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.314+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d01042b0 con 0x7f73d01018d0 2024-04-05T23:32:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.316+0000 7f73ccc00700 1 -- 172.21.15.16:0/3946735285 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f73bc003f20 con 0x7f73d01018d0 2024-04-05T23:32:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.318+0000 7f73ccc00700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73b40a1230 0x7f73b40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.319+0000 7f73ccc00700 1 -- 172.21.15.16:0/3946735285 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f73bc014070 con 0x7f73d01018d0 2024-04-05T23:32:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.319+0000 7f73cfe00700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73b40a1230 0x7f73b40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.319+0000 7f73cfe00700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73b40a1230 0x7f73b40a36f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f73c000b2e0 tx=0x7f73c0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.322+0000 7f73ccc00700 1 -- 172.21.15.16:0/3946735285 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f73bc096560 con 0x7f73d01018d0 2024-04-05T23:32:52.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.532+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73d019a830 con 0x7f73b40a1230 2024-04-05T23:32:52.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.544+0000 7f73ccc00700 1 -- 172.21.15.16:0/3946735285 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340337 (secure 0 0 0) 0x7f73d019a830 con 0x7f73b40a1230 2024-04-05T23:32:52.542 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73b40a1230 msgr2=0x7f73b40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73b40a1230 0x7f73b40a36f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f73c000b2e0 tx=0x7f73c0005fd0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01018d0 msgr2=0x7f73d0199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01018d0 0x7f73d0199a60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00eb30 tx=0x7f73bc00ee40 comp rx=0 tx=0).stop 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 shutdown_connections 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f73b40a1230 0x7f73b40a36f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d0106f70 0x7f73d019de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73d0100f30 0x7f73d0199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 --2- 172.21.15.16:0/3946735285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01018d0 0x7f73d0199a60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 >> 172.21.15.16:0/3946735285 conn(0x7f73d00780d0 msgr2=0x7f73d010c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 shutdown_connections 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:52.550+0000 7f73d7200700 1 -- 172.21.15.16:0/3946735285 wait complete. 2024-04-05T23:32:52.545 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-05T23:32:52.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:52 smithi027 ceph-mon[78597]: pgmap v258: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 54 B/s, 0 keys/s, 0 objects/s recovering 2024-04-05T23:32:52.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:52 smithi016 ceph-mon[38248]: pgmap v258: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 54 B/s, 0 keys/s, 0 objects/s recovering 2024-04-05T23:32:52.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:52 smithi130 ceph-mon[79417]: pgmap v258: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 54 B/s, 0 keys/s, 0 objects/s recovering 2024-04-05T23:32:52.924 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-04-05T23:32:51.303431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656662,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":555,"num_read_kb":402,"num_write":415,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":341,"ondisk_log_size":341,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365496,"kb_used_data":12564,"kb_used_omap":18,"kb_used_meta":352749,"kb_avail":1124576328,"statfs":{"total":1151940427776,"available":1151566159872,"internally_reserved":0,"allocated":12865536,"data_stored":7117871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":361215301},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":114,"num_read_kb":59,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003246"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371867+0000","last_change":"2024-04-05T23:32:04.536879+0000","last_active":"2024-04-05T23:32:37.371867+0000","last_peered":"2024-04-05T23:32:37.371867+0000","last_clean":"2024-04-05T23:32:37.371867+0000","last_became_active":"2024-04-05T23:32:04.536183+0000","last_became_peered":"2024-04-05T23:32:04.536183+0000","last_unstale":"2024-04-05T23:32:37.371867+0000","last_undegraded":"2024-04-05T23:32:37.371867+0000","last_fullsized":"2024-04-05T23:32:37.371867+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:13:22.589464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380649+0000","last_change":"2024-04-05T23:32:04.540791+0000","last_active":"2024-04-05T23:32:37.380649+0000","last_peered":"2024-04-05T23:32:37.380649+0000","last_clean":"2024-04-05T23:32:37.380649+0000","last_became_active":"2024-04-05T23:32:04.540455+0000","last_became_peered":"2024-04-05T23:32:04.540455+0000","last_unstale":"2024-04-05T23:32:37.380649+0000","last_undegraded":"2024-04-05T23:32:37.380649+0000","last_fullsized":"2024-04-05T23:32:37.380649+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:06:19.090672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380615+0000","last_change":"2024-04-05T23:32:04.543428+0000","last_active":"2024-04-05T23:32:37.380615+0000","last_peered":"2024-04-05T23:32:37.380615+0000","last_clean":"2024-04-05T23:32:37.380615+0000","last_became_active":"2024-04-05T23:32:04.543071+0000","last_became_peered":"2024-04-05T23:32:04.543071+0000","last_unstale":"2024-04-05T23:32:37.380615+0000","last_undegraded":"2024-04-05T23:32:37.380615+0000","last_fullsized":"2024-04-05T23:32:37.380615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:35:44.911749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378253+0000","last_change":"2024-04-05T23:32:04.531444+0000","last_active":"2024-04-05T23:32:37.378253+0000","last_peered":"2024-04-05T23:32:37.378253+0000","last_clean":"2024-04-05T23:32:37.378253+0000","last_became_active":"2024-04-05T23:32:04.529556+0000","last_became_peered":"2024-04-05T23:32:04.529556+0000","last_unstale":"2024-04-05T23:32:37.378253+0000","last_undegraded":"2024-04-05T23:32:37.378253+0000","last_fullsized":"2024-04-05T23:32:37.378253+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:48.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380525+0000","last_change":"2024-04-05T23:32:04.529565+0000","last_active":"2024-04-05T23:32:37.380525+0000","last_peered":"2024-04-05T23:32:37.380525+0000","last_clean":"2024-04-05T23:32:37.380525+0000","last_became_active":"2024-04-05T23:32:04.529226+0000","last_became_peered":"2024-04-05T23:32:04.529226+0000","last_unstale":"2024-04-05T23:32:37.380525+0000","last_undegraded":"2024-04-05T23:32:37.380525+0000","last_fullsized":"2024-04-05T23:32:37.380525+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:44:25.208996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375831+0000","last_change":"2024-04-05T23:32:04.546260+0000","last_active":"2024-04-05T23:32:37.375831+0000","last_peered":"2024-04-05T23:32:37.375831+0000","last_clean":"2024-04-05T23:32:37.375831+0000","last_became_active":"2024-04-05T23:32:04.545851+0000","last_became_peered":"2024-04-05T23:32:04.545851+0000","last_unstale":"2024-04-05T23:32:37.375831+0000","last_undegraded":"2024-04-05T23:32:37.375831+0000","last_fullsized":"2024-04-05T23:32:37.375831+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:59:16.735635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378362+0000","last_change":"2024-04-05T23:32:04.536348+0000","last_active":"2024-04-05T23:32:37.378362+0000","last_peered":"2024-04-05T23:32:37.378362+0000","last_clean":"2024-04-05T23:32:37.378362+0000","last_became_active":"2024-04-05T23:32:04.536015+0000","last_became_peered":"2024-04-05T23:32:04.536015+0000","last_unstale":"2024-04-05T23:32:37.378362+0000","last_undegraded":"2024-04-05T23:32:37.378362+0000","last_fullsized":"2024-04-05T23:32:37.378362+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:02:24.897957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371406+0000","last_change":"2024-04-05T23:32:04.532528+0000","last_active":"2024-04-05T23:32:37.371406+0000","last_peered":"2024-04-05T23:32:37.371406+0000","last_clean":"2024-04-05T23:32:37.371406+0000","last_became_active":"2024-04-05T23:32:04.532104+0000","last_became_peered":"2024-04-05T23:32:04.532104+0000","last_unstale":"2024-04-05T23:32:37.371406+0000","last_undegraded":"2024-04-05T23:32:37.371406+0000","last_fullsized":"2024-04-05T23:32:37.371406+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:06.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.381590+0000","last_change":"2024-04-05T23:32:04.531223+0000","last_active":"2024-04-05T23:32:37.381590+0000","last_peered":"2024-04-05T23:32:37.381590+0000","last_clean":"2024-04-05T23:32:37.381590+0000","last_became_active":"2024-04-05T23:32:04.530863+0000","last_became_peered":"2024-04-05T23:32:04.530863+0000","last_unstale":"2024-04-05T23:32:37.381590+0000","last_undegraded":"2024-04-05T23:32:37.381590+0000","last_fullsized":"2024-04-05T23:32:37.381590+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:03:35.912984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.376132+0000","last_change":"2024-04-05T23:32:04.541481+0000","last_active":"2024-04-05T23:32:37.376132+0000","last_peered":"2024-04-05T23:32:37.376132+0000","last_clean":"2024-04-05T23:32:37.376132+0000","last_became_active":"2024-04-05T23:32:04.541133+0000","last_became_peered":"2024-04-05T23:32:04.541133+0000","last_unstale":"2024-04-05T23:32:37.376132+0000","last_undegraded":"2024-04-05T23:32:37.376132+0000","last_fullsized":"2024-04-05T23:32:37.376132+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:01:22.825587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378586+0000","last_change":"2024-04-05T23:32:04.542390+0000","last_active":"2024-04-05T23:32:37.378586+0000","last_peered":"2024-04-05T23:32:37.378586+0000","last_clean":"2024-04-05T23:32:37.378586+0000","last_became_active":"2024-04-05T23:32:04.542026+0000","last_became_peered":"2024-04-05T23:32:04.542026+0000","last_unstale":"2024-04-05T23:32:37.378586+0000","last_undegraded":"2024-04-05T23:32:37.378586+0000","last_fullsized":"2024-04-05T23:32:37.378586+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:15:42.243807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378195+0000","last_change":"2024-04-05T23:32:04.540096+0000","last_active":"2024-04-05T23:32:37.378195+0000","last_peered":"2024-04-05T23:32:37.378195+0000","last_clean":"2024-04-05T23:32:37.378195+0000","last_became_active":"2024-04-05T23:32:04.539764+0000","last_became_peered":"2024-04-05T23:32:04.539764+0000","last_unstale":"2024-04-05T23:32:37.378195+0000","last_undegraded":"2024-04-05T23:32:37.378195+0000","last_fullsized":"2024-04-05T23:32:37.378195+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:57:36.442393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378551+0000","last_change":"2024-04-05T23:32:04.540839+0000","last_active":"2024-04-05T23:32:37.378551+0000","last_peered":"2024-04-05T23:32:37.378551+0000","last_clean":"2024-04-05T23:32:37.378551+0000","last_became_active":"2024-04-05T23:32:04.540511+0000","last_became_peered":"2024-04-05T23:32:04.540511+0000","last_unstale":"2024-04-05T23:32:37.378551+0000","last_undegraded":"2024-04-05T23:32:37.378551+0000","last_fullsized":"2024-04-05T23:32:37.378551+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:58:24.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380309+0000","last_change":"2024-04-05T23:32:04.539222+0000","last_active":"2024-04-05T23:32:37.380309+0000","last_peered":"2024-04-05T23:32:37.380309+0000","last_clean":"2024-04-05T23:32:37.380309+0000","last_became_active":"2024-04-05T23:32:04.538906+0000","last_became_peered":"2024-04-05T23:32:04.538906+0000","last_unstale":"2024-04-05T23:32:37.380309+0000","last_undegraded":"2024-04-05T23:32:37.380309+0000","last_fullsized":"2024-04-05T23:32:37.380309+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:05:24.429153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380277+0000","last_change":"2024-04-05T23:32:04.544155+0000","last_active":"2024-04-05T23:32:37.380277+0000","last_peered":"2024-04-05T23:32:37.380277+0000","last_clean":"2024-04-05T23:32:37.380277+0000","last_became_active":"2024-04-05T23:32:04.543852+0000","last_became_peered":"2024-04-05T23:32:04.543852+0000","last_unstale":"2024-04-05T23:32:37.380277+0000","last_undegraded":"2024-04-05T23:32:37.380277+0000","last_fullsized":"2024-04-05T23:32:37.380277+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:18:34.353542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380824+0000","last_change":"2024-04-05T23:32:04.533772+0000","last_active":"2024-04-05T23:32:37.380824+0000","last_peered":"2024-04-05T23:32:37.380824+0000","last_clean":"2024-04-05T23:32:37.380824+0000","last_became_active":"2024-04-05T23:32:04.533416+0000","last_became_peered":"2024-04-05T23:32:04.533416+0000","last_unstale":"2024-04-05T23:32:37.380824+0000","last_undegraded":"2024-04-05T23:32:37.380824+0000","last_fullsized":"2024-04-05T23:32:37.380824+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:36:55.072349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371943+0000","last_change":"2024-04-05T23:32:04.551902+0000","last_active":"2024-04-05T23:32:37.371943+0000","last_peered":"2024-04-05T23:32:37.371943+0000","last_clean":"2024-04-05T23:32:37.371943+0000","last_became_active":"2024-04-05T23:32:04.551595+0000","last_became_peered":"2024-04-05T23:32:04.551595+0000","last_unstale":"2024-04-05T23:32:37.371943+0000","last_undegraded":"2024-04-05T23:32:37.371943+0000","last_fullsized":"2024-04-05T23:32:37.371943+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:14:46.648789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375868+0000","last_change":"2024-04-05T23:32:04.542967+0000","last_active":"2024-04-05T23:32:37.375868+0000","last_peered":"2024-04-05T23:32:37.375868+0000","last_clean":"2024-04-05T23:32:37.375868+0000","last_became_active":"2024-04-05T23:32:04.541767+0000","last_became_peered":"2024-04-05T23:32:04.541767+0000","last_unstale":"2024-04-05T23:32:37.375868+0000","last_undegraded":"2024-04-05T23:32:37.375868+0000","last_fullsized":"2024-04-05T23:32:37.375868+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:27:46.150249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375102+0000","last_change":"2024-04-05T23:32:04.549512+0000","last_active":"2024-04-05T23:32:37.375102+0000","last_peered":"2024-04-05T23:32:37.375102+0000","last_clean":"2024-04-05T23:32:37.375102+0000","last_became_active":"2024-04-05T23:32:04.549184+0000","last_became_peered":"2024-04-05T23:32:04.549184+0000","last_unstale":"2024-04-05T23:32:37.375102+0000","last_undegraded":"2024-04-05T23:32:37.375102+0000","last_fullsized":"2024-04-05T23:32:37.375102+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:48:24.186408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:47.496852+0000","last_change":"2024-04-05T23:32:04.005188+0000","last_active":"2024-04-05T23:32:47.496852+0000","last_peered":"2024-04-05T23:32:47.496852+0000","last_clean":"2024-04-05T23:32:47.496852+0000","last_became_active":"2024-04-05T23:32:04.004818+0000","last_became_peered":"2024-04-05T23:32:04.004818+0000","last_unstale":"2024-04-05T23:32:47.496852+0000","last_undegraded":"2024-04-05T23:32:47.496852+0000","last_fullsized":"2024-04-05T23:32:47.496852+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:49:41.146761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.372092+0000","last_change":"2024-04-05T23:32:04.544881+0000","last_active":"2024-04-05T23:32:37.372092+0000","last_peered":"2024-04-05T23:32:37.372092+0000","last_clean":"2024-04-05T23:32:37.372092+0000","last_became_active":"2024-04-05T23:32:04.542149+0000","last_became_peered":"2024-04-05T23:32:04.542149+0000","last_unstale":"2024-04-05T23:32:37.372092+0000","last_undegraded":"2024-04-05T23:32:37.372092+0000","last_fullsized":"2024-04-05T23:32:37.372092+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:28:51.713053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.372234+0000","last_change":"2024-04-05T23:32:04.011450+0000","last_active":"2024-04-05T23:32:37.372234+0000","last_peered":"2024-04-05T23:32:37.372234+0000","last_clean":"2024-04-05T23:32:37.372234+0000","last_became_active":"2024-04-05T23:32:04.009607+0000","last_became_peered":"2024-04-05T23:32:04.009607+0000","last_unstale":"2024-04-05T23:32:37.372234+0000","last_undegraded":"2024-04-05T23:32:37.372234+0000","last_fullsized":"2024-04-05T23:32:37.372234+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:16:45.483108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.381468+0000","last_change":"2024-04-05T23:32:04.550370+0000","last_active":"2024-04-05T23:32:37.381468+0000","last_peered":"2024-04-05T23:32:37.381468+0000","last_clean":"2024-04-05T23:32:37.381468+0000","last_became_active":"2024-04-05T23:32:04.549980+0000","last_became_peered":"2024-04-05T23:32:04.549980+0000","last_unstale":"2024-04-05T23:32:37.381468+0000","last_undegraded":"2024-04-05T23:32:37.381468+0000","last_fullsized":"2024-04-05T23:32:37.381468+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:44:30.704366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380842+0000","last_change":"2024-04-05T23:32:04.004169+0000","last_active":"2024-04-05T23:32:37.380842+0000","last_peered":"2024-04-05T23:32:37.380842+0000","last_clean":"2024-04-05T23:32:37.380842+0000","last_became_active":"2024-04-05T23:32:04.003646+0000","last_became_peered":"2024-04-05T23:32:04.003646+0000","last_unstale":"2024-04-05T23:32:37.380842+0000","last_undegraded":"2024-04-05T23:32:37.380842+0000","last_fullsized":"2024-04-05T23:32:37.380842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:57:02.781815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.381679+0000","last_change":"2024-04-05T23:32:04.542066+0000","last_active":"2024-04-05T23:32:37.381679+0000","last_peered":"2024-04-05T23:32:37.381679+0000","last_clean":"2024-04-05T23:32:37.381679+0000","last_became_active":"2024-04-05T23:32:04.540735+0000","last_became_peered":"2024-04-05T23:32:04.540735+0000","last_unstale":"2024-04-05T23:32:37.381679+0000","last_undegraded":"2024-04-05T23:32:37.381679+0000","last_fullsized":"2024-04-05T23:32:37.381679+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:02:03.595888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.397068+0000","last_change":"2024-04-05T23:32:38.396641+0000","last_active":"2024-04-05T23:32:38.397068+0000","last_peered":"2024-04-05T23:32:38.397068+0000","last_clean":"2024-04-05T23:32:38.397068+0000","last_became_active":"2024-04-05T23:32:38.379708+0000","last_became_peered":"2024-04-05T23:32:38.379708+0000","last_unstale":"2024-04-05T23:32:38.397068+0000","last_undegraded":"2024-04-05T23:32:38.397068+0000","last_fullsized":"2024-04-05T23:32:38.397068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:00.370725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378242+0000","last_change":"2024-04-05T23:32:04.545780+0000","last_active":"2024-04-05T23:32:37.378242+0000","last_peered":"2024-04-05T23:32:37.378242+0000","last_clean":"2024-04-05T23:32:37.378242+0000","last_became_active":"2024-04-05T23:32:04.545279+0000","last_became_peered":"2024-04-05T23:32:04.545279+0000","last_unstale":"2024-04-05T23:32:37.378242+0000","last_undegraded":"2024-04-05T23:32:37.378242+0000","last_fullsized":"2024-04-05T23:32:37.378242+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:10.577280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:48.469685+0000","last_change":"2024-04-05T23:32:04.004189+0000","last_active":"2024-04-05T23:32:48.469685+0000","last_peered":"2024-04-05T23:32:48.469685+0000","last_clean":"2024-04-05T23:32:48.469685+0000","last_became_active":"2024-04-05T23:32:04.003841+0000","last_became_peered":"2024-04-05T23:32:04.003841+0000","last_unstale":"2024-04-05T23:32:48.469685+0000","last_undegraded":"2024-04-05T23:32:48.469685+0000","last_fullsized":"2024-04-05T23:32:48.469685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:33:26.898657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380547+0000","last_change":"2024-04-05T23:32:04.529538+0000","last_active":"2024-04-05T23:32:37.380547+0000","last_peered":"2024-04-05T23:32:37.380547+0000","last_clean":"2024-04-05T23:32:37.380547+0000","last_became_active":"2024-04-05T23:32:04.529231+0000","last_became_peered":"2024-04-05T23:32:04.529231+0000","last_unstale":"2024-04-05T23:32:37.380547+0000","last_undegraded":"2024-04-05T23:32:37.380547+0000","last_fullsized":"2024-04-05T23:32:37.380547+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:49:15.608966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375765+0000","last_change":"2024-04-05T23:32:04.002763+0000","last_active":"2024-04-05T23:32:37.375765+0000","last_peered":"2024-04-05T23:32:37.375765+0000","last_clean":"2024-04-05T23:32:37.375765+0000","last_became_active":"2024-04-05T23:32:04.002410+0000","last_became_peered":"2024-04-05T23:32:04.002410+0000","last_unstale":"2024-04-05T23:32:37.375765+0000","last_undegraded":"2024-04-05T23:32:37.375765+0000","last_fullsized":"2024-04-05T23:32:37.375765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:15.193813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.462329+0000","last_change":"2024-04-05T23:32:38.461447+0000","last_active":"2024-04-05T23:32:38.462329+0000","last_peered":"2024-04-05T23:32:38.462329+0000","last_clean":"2024-04-05T23:32:38.462329+0000","last_became_active":"2024-04-05T23:32:38.460968+0000","last_became_peered":"2024-04-05T23:32:38.460968+0000","last_unstale":"2024-04-05T23:32:38.462329+0000","last_undegraded":"2024-04-05T23:32:38.462329+0000","last_fullsized":"2024-04-05T23:32:38.462329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:42:21.093881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378913+0000","last_change":"2024-04-05T23:32:04.001937+0000","last_active":"2024-04-05T23:32:37.378913+0000","last_peered":"2024-04-05T23:32:37.378913+0000","last_clean":"2024-04-05T23:32:37.378913+0000","last_became_active":"2024-04-05T23:32:04.001508+0000","last_became_peered":"2024-04-05T23:32:04.001508+0000","last_unstale":"2024-04-05T23:32:37.378913+0000","last_undegraded":"2024-04-05T23:32:37.378913+0000","last_fullsized":"2024-04-05T23:32:37.378913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:09:20.682084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371478+0000","last_change":"2024-04-05T23:32:04.531576+0000","last_active":"2024-04-05T23:32:37.371478+0000","last_peered":"2024-04-05T23:32:37.371478+0000","last_clean":"2024-04-05T23:32:37.371478+0000","last_became_active":"2024-04-05T23:32:04.531234+0000","last_became_peered":"2024-04-05T23:32:04.531234+0000","last_unstale":"2024-04-05T23:32:37.371478+0000","last_undegraded":"2024-04-05T23:32:37.371478+0000","last_fullsized":"2024-04-05T23:32:37.371478+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:21:57.769861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377030+0000","last_change":"2024-04-05T23:32:04.001570+0000","last_active":"2024-04-05T23:32:37.377030+0000","last_peered":"2024-04-05T23:32:37.377030+0000","last_clean":"2024-04-05T23:32:37.377030+0000","last_became_active":"2024-04-05T23:32:04.001224+0000","last_became_peered":"2024-04-05T23:32:04.001224+0000","last_unstale":"2024-04-05T23:32:37.377030+0000","last_undegraded":"2024-04-05T23:32:37.377030+0000","last_fullsized":"2024-04-05T23:32:37.377030+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:44:41.070205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375868+0000","last_change":"2024-04-05T23:32:04.542029+0000","last_active":"2024-04-05T23:32:37.375868+0000","last_peered":"2024-04-05T23:32:37.375868+0000","last_clean":"2024-04-05T23:32:37.375868+0000","last_became_active":"2024-04-05T23:32:04.541684+0000","last_became_peered":"2024-04-05T23:32:04.541684+0000","last_unstale":"2024-04-05T23:32:37.375868+0000","last_undegraded":"2024-04-05T23:32:37.375868+0000","last_fullsized":"2024-04-05T23:32:37.375868+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:07:06.456917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":92,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:46.466090+0000","last_change":"2024-04-05T23:32:04.004986+0000","last_active":"2024-04-05T23:32:46.466090+0000","last_peered":"2024-04-05T23:32:46.466090+0000","last_clean":"2024-04-05T23:32:46.466090+0000","last_became_active":"2024-04-05T23:32:04.004606+0000","last_became_peered":"2024-04-05T23:32:04.004606+0000","last_unstale":"2024-04-05T23:32:46.466090+0000","last_undegraded":"2024-04-05T23:32:46.466090+0000","last_fullsized":"2024-04-05T23:32:46.466090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:23:41.060500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.381509+0000","last_change":"2024-04-05T23:32:04.541334+0000","last_active":"2024-04-05T23:32:37.381509+0000","last_peered":"2024-04-05T23:32:37.381509+0000","last_clean":"2024-04-05T23:32:37.381509+0000","last_became_active":"2024-04-05T23:32:04.540967+0000","last_became_peered":"2024-04-05T23:32:04.540967+0000","last_unstale":"2024-04-05T23:32:37.381509+0000","last_undegraded":"2024-04-05T23:32:37.381509+0000","last_fullsized":"2024-04-05T23:32:37.381509+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:16:33.045172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.466417+0000","last_change":"2024-04-05T23:32:38.466417+0000","last_active":"2024-04-05T23:32:38.466417+0000","last_peered":"2024-04-05T23:32:38.466417+0000","last_clean":"2024-04-05T23:32:38.466417+0000","last_became_active":"2024-04-05T23:32:38.462293+0000","last_became_peered":"2024-04-05T23:32:38.462293+0000","last_unstale":"2024-04-05T23:32:38.466417+0000","last_undegraded":"2024-04-05T23:32:38.466417+0000","last_fullsized":"2024-04-05T23:32:38.466417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:31.118636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.385818+0000","last_change":"2024-04-05T23:32:38.385818+0000","last_active":"2024-04-05T23:32:38.385818+0000","last_peered":"2024-04-05T23:32:38.385818+0000","last_clean":"2024-04-05T23:32:38.385818+0000","last_became_active":"2024-04-05T23:32:38.385423+0000","last_became_peered":"2024-04-05T23:32:38.385423+0000","last_unstale":"2024-04-05T23:32:38.385818+0000","last_undegraded":"2024-04-05T23:32:38.385818+0000","last_fullsized":"2024-04-05T23:32:38.385818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:07:11.374783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378141+0000","last_change":"2024-04-05T23:32:04.000697+0000","last_active":"2024-04-05T23:32:37.378141+0000","last_peered":"2024-04-05T23:32:37.378141+0000","last_clean":"2024-04-05T23:32:37.378141+0000","last_became_active":"2024-04-05T23:32:04.000377+0000","last_became_peered":"2024-04-05T23:32:04.000377+0000","last_unstale":"2024-04-05T23:32:37.378141+0000","last_undegraded":"2024-04-05T23:32:37.378141+0000","last_fullsized":"2024-04-05T23:32:37.378141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:55:11.227637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.383367+0000","last_change":"2024-04-05T23:32:38.383367+0000","last_active":"2024-04-05T23:32:38.383367+0000","last_peered":"2024-04-05T23:32:38.383367+0000","last_clean":"2024-04-05T23:32:38.383367+0000","last_became_active":"2024-04-05T23:32:38.383002+0000","last_became_peered":"2024-04-05T23:32:38.383002+0000","last_unstale":"2024-04-05T23:32:38.383367+0000","last_undegraded":"2024-04-05T23:32:38.383367+0000","last_fullsized":"2024-04-05T23:32:38.383367+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:30:59.947387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'30","reported_seq":128,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:46.478741+0000","last_change":"2024-04-05T23:32:04.007322+0000","last_active":"2024-04-05T23:32:46.478741+0000","last_peered":"2024-04-05T23:32:46.478741+0000","last_clean":"2024-04-05T23:32:46.478741+0000","last_became_active":"2024-04-05T23:32:04.006971+0000","last_became_peered":"2024-04-05T23:32:04.006971+0000","last_unstale":"2024-04-05T23:32:46.478741+0000","last_undegraded":"2024-04-05T23:32:46.478741+0000","last_fullsized":"2024-04-05T23:32:46.478741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:04:38.413482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59830,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":100,"num_write":32,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380649+0000","last_change":"2024-04-05T23:32:04.535342+0000","last_active":"2024-04-05T23:32:37.380649+0000","last_peered":"2024-04-05T23:32:37.380649+0000","last_clean":"2024-04-05T23:32:37.380649+0000","last_became_active":"2024-04-05T23:32:04.535053+0000","last_became_peered":"2024-04-05T23:32:04.535053+0000","last_unstale":"2024-04-05T23:32:37.380649+0000","last_undegraded":"2024-04-05T23:32:37.380649+0000","last_fullsized":"2024-04-05T23:32:37.380649+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:19:20.683836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.391798+0000","last_change":"2024-04-05T23:32:38.391418+0000","last_active":"2024-04-05T23:32:38.391798+0000","last_peered":"2024-04-05T23:32:38.391798+0000","last_clean":"2024-04-05T23:32:38.391798+0000","last_became_active":"2024-04-05T23:32:38.379643+0000","last_became_peered":"2024-04-05T23:32:38.379643+0000","last_unstale":"2024-04-05T23:32:38.391798+0000","last_undegraded":"2024-04-05T23:32:38.391798+0000","last_fullsized":"2024-04-05T23:32:38.391798+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:37:12.343153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378225+0000","last_change":"2024-04-05T23:32:04.536455+0000","last_active":"2024-04-05T23:32:37.378225+0000","last_peered":"2024-04-05T23:32:37.378225+0000","last_clean":"2024-04-05T23:32:37.378225+0000","last_became_active":"2024-04-05T23:32:04.533994+0000","last_became_peered":"2024-04-05T23:32:04.533994+0000","last_unstale":"2024-04-05T23:32:37.378225+0000","last_undegraded":"2024-04-05T23:32:37.378225+0000","last_fullsized":"2024-04-05T23:32:37.378225+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:38:18.952823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377668+0000","last_change":"2024-04-05T23:32:04.000708+0000","last_active":"2024-04-05T23:32:37.377668+0000","last_peered":"2024-04-05T23:32:37.377668+0000","last_clean":"2024-04-05T23:32:37.377668+0000","last_became_active":"2024-04-05T23:32:04.000354+0000","last_became_peered":"2024-04-05T23:32:04.000354+0000","last_unstale":"2024-04-05T23:32:37.377668+0000","last_undegraded":"2024-04-05T23:32:37.377668+0000","last_fullsized":"2024-04-05T23:32:37.377668+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:05:34.291607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378606+0000","last_change":"2024-04-05T23:32:04.532146+0000","last_active":"2024-04-05T23:32:37.378606+0000","last_peered":"2024-04-05T23:32:37.378606+0000","last_clean":"2024-04-05T23:32:37.378606+0000","last_became_active":"2024-04-05T23:32:04.531735+0000","last_became_peered":"2024-04-05T23:32:04.531735+0000","last_unstale":"2024-04-05T23:32:37.378606+0000","last_undegraded":"2024-04-05T23:32:37.378606+0000","last_fullsized":"2024-04-05T23:32:37.378606+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:52:44.898871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:46.473449+0000","last_change":"2024-04-05T23:32:04.000144+0000","last_active":"2024-04-05T23:32:46.473449+0000","last_peered":"2024-04-05T23:32:46.473449+0000","last_clean":"2024-04-05T23:32:46.473449+0000","last_became_active":"2024-04-05T23:32:03.999712+0000","last_became_peered":"2024-04-05T23:32:03.999712+0000","last_unstale":"2024-04-05T23:32:46.473449+0000","last_undegraded":"2024-04-05T23:32:46.473449+0000","last_fullsized":"2024-04-05T23:32:46.473449+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:43.775018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378466+0000","last_change":"2024-04-05T23:32:04.537623+0000","last_active":"2024-04-05T23:32:37.378466+0000","last_peered":"2024-04-05T23:32:37.378466+0000","last_clean":"2024-04-05T23:32:37.378466+0000","last_became_active":"2024-04-05T23:32:04.537399+0000","last_became_peered":"2024-04-05T23:32:04.537399+0000","last_unstale":"2024-04-05T23:32:37.378466+0000","last_undegraded":"2024-04-05T23:32:37.378466+0000","last_fullsized":"2024-04-05T23:32:37.378466+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:01:01.643133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380396+0000","last_change":"2024-04-05T23:32:04.006806+0000","last_active":"2024-04-05T23:32:37.380396+0000","last_peered":"2024-04-05T23:32:37.380396+0000","last_clean":"2024-04-05T23:32:37.380396+0000","last_became_active":"2024-04-05T23:32:04.006397+0000","last_became_peered":"2024-04-05T23:32:04.006397+0000","last_unstale":"2024-04-05T23:32:37.380396+0000","last_undegraded":"2024-04-05T23:32:37.380396+0000","last_fullsized":"2024-04-05T23:32:37.380396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:42:01.758863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377159+0000","last_change":"2024-04-05T23:32:04.545952+0000","last_active":"2024-04-05T23:32:37.377159+0000","last_peered":"2024-04-05T23:32:37.377159+0000","last_clean":"2024-04-05T23:32:37.377159+0000","last_became_active":"2024-04-05T23:32:04.545528+0000","last_became_peered":"2024-04-05T23:32:04.545528+0000","last_unstale":"2024-04-05T23:32:37.377159+0000","last_undegraded":"2024-04-05T23:32:37.377159+0000","last_fullsized":"2024-04-05T23:32:37.377159+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:33:06.074173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380812+0000","last_change":"2024-04-05T23:32:03.999841+0000","last_active":"2024-04-05T23:32:37.380812+0000","last_peered":"2024-04-05T23:32:37.380812+0000","last_clean":"2024-04-05T23:32:37.380812+0000","last_became_active":"2024-04-05T23:32:03.999418+0000","last_became_peered":"2024-04-05T23:32:03.999418+0000","last_unstale":"2024-04-05T23:32:37.380812+0000","last_undegraded":"2024-04-05T23:32:37.380812+0000","last_fullsized":"2024-04-05T23:32:37.380812+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:00:41.452167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371767+0000","last_change":"2024-04-05T23:32:04.530012+0000","last_active":"2024-04-05T23:32:37.371767+0000","last_peered":"2024-04-05T23:32:37.371767+0000","last_clean":"2024-04-05T23:32:37.371767+0000","last_became_active":"2024-04-05T23:32:04.529628+0000","last_became_peered":"2024-04-05T23:32:04.529628+0000","last_unstale":"2024-04-05T23:32:37.371767+0000","last_undegraded":"2024-04-05T23:32:37.371767+0000","last_fullsized":"2024-04-05T23:32:37.371767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:40:22.036170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375890+0000","last_change":"2024-04-05T23:32:04.006157+0000","last_active":"2024-04-05T23:32:37.375890+0000","last_peered":"2024-04-05T23:32:37.375890+0000","last_clean":"2024-04-05T23:32:37.375890+0000","last_became_active":"2024-04-05T23:32:04.005853+0000","last_became_peered":"2024-04-05T23:32:04.005853+0000","last_unstale":"2024-04-05T23:32:37.375890+0000","last_undegraded":"2024-04-05T23:32:37.375890+0000","last_fullsized":"2024-04-05T23:32:37.375890+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:48:39.849306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375866+0000","last_change":"2024-04-05T23:32:04.547367+0000","last_active":"2024-04-05T23:32:37.375866+0000","last_peered":"2024-04-05T23:32:37.375866+0000","last_clean":"2024-04-05T23:32:37.375866+0000","last_became_active":"2024-04-05T23:32:04.547040+0000","last_became_peered":"2024-04-05T23:32:04.547040+0000","last_unstale":"2024-04-05T23:32:37.375866+0000","last_undegraded":"2024-04-05T23:32:37.375866+0000","last_fullsized":"2024-04-05T23:32:37.375866+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:41:41.998984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:44.493445+0000","last_change":"2024-04-05T23:32:04.009851+0000","last_active":"2024-04-05T23:32:44.493445+0000","last_peered":"2024-04-05T23:32:44.493445+0000","last_clean":"2024-04-05T23:32:44.493445+0000","last_became_active":"2024-04-05T23:32:04.009460+0000","last_became_peered":"2024-04-05T23:32:04.009460+0000","last_unstale":"2024-04-05T23:32:44.493445+0000","last_undegraded":"2024-04-05T23:32:44.493445+0000","last_fullsized":"2024-04-05T23:32:44.493445+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:39:41.000439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.376044+0000","last_change":"2024-04-05T23:32:04.549621+0000","last_active":"2024-04-05T23:32:37.376044+0000","last_peered":"2024-04-05T23:32:37.376044+0000","last_clean":"2024-04-05T23:32:37.376044+0000","last_became_active":"2024-04-05T23:32:04.549325+0000","last_became_peered":"2024-04-05T23:32:04.549325+0000","last_unstale":"2024-04-05T23:32:37.376044+0000","last_undegraded":"2024-04-05T23:32:37.376044+0000","last_fullsized":"2024-04-05T23:32:37.376044+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:11:07.101463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377151+0000","last_change":"2024-04-05T23:32:04.546768+0000","last_active":"2024-04-05T23:32:37.377151+0000","last_peered":"2024-04-05T23:32:37.377151+0000","last_clean":"2024-04-05T23:32:37.377151+0000","last_became_active":"2024-04-05T23:32:04.546471+0000","last_became_peered":"2024-04-05T23:32:04.546471+0000","last_unstale":"2024-04-05T23:32:37.377151+0000","last_undegraded":"2024-04-05T23:32:37.377151+0000","last_fullsized":"2024-04-05T23:32:37.377151+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:48:22.728976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:49.476994+0000","last_change":"2024-04-05T23:32:04.007093+0000","last_active":"2024-04-05T23:32:49.476994+0000","last_peered":"2024-04-05T23:32:49.476994+0000","last_clean":"2024-04-05T23:32:49.476994+0000","last_became_active":"2024-04-05T23:32:04.006836+0000","last_became_peered":"2024-04-05T23:32:04.006836+0000","last_unstale":"2024-04-05T23:32:49.476994+0000","last_undegraded":"2024-04-05T23:32:49.476994+0000","last_fullsized":"2024-04-05T23:32:49.476994+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:20:42.489368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371662+0000","last_change":"2024-04-05T23:32:04.538204+0000","last_active":"2024-04-05T23:32:37.371662+0000","last_peered":"2024-04-05T23:32:37.371662+0000","last_clean":"2024-04-05T23:32:37.371662+0000","last_became_active":"2024-04-05T23:32:04.535254+0000","last_became_peered":"2024-04-05T23:32:04.535254+0000","last_unstale":"2024-04-05T23:32:37.371662+0000","last_undegraded":"2024-04-05T23:32:37.371662+0000","last_fullsized":"2024-04-05T23:32:37.371662+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:06:01.059339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377772+0000","last_change":"2024-04-05T23:32:03.997704+0000","last_active":"2024-04-05T23:32:37.377772+0000","last_peered":"2024-04-05T23:32:37.377772+0000","last_clean":"2024-04-05T23:32:37.377772+0000","last_became_active":"2024-04-05T23:32:03.997275+0000","last_became_peered":"2024-04-05T23:32:03.997275+0000","last_unstale":"2024-04-05T23:32:37.377772+0000","last_undegraded":"2024-04-05T23:32:37.377772+0000","last_fullsized":"2024-04-05T23:32:37.377772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:58.355104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":336,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378679+0000","last_change":"2024-04-05T23:31:10.395041+0000","last_active":"2024-04-05T23:32:37.378679+0000","last_peered":"2024-04-05T23:32:37.378679+0000","last_clean":"2024-04-05T23:32:37.378679+0000","last_became_active":"2024-04-05T23:31:10.394479+0000","last_became_peered":"2024-04-05T23:31:10.394479+0000","last_unstale":"2024-04-05T23:32:37.378679+0000","last_undegraded":"2024-04-05T23:32:37.378679+0000","last_fullsized":"2024-04-05T23:32:37.378679+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_clean_scrub_stamp":"2024-04-05T23:28:33.945697+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:54:10.683526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.385626+0000","last_change":"2024-04-05T23:32:38.385626+0000","last_active":"2024-04-05T23:32:38.385626+0000","last_peered":"2024-04-05T23:32:38.385626+0000","last_clean":"2024-04-05T23:32:38.385626+0000","last_became_active":"2024-04-05T23:32:38.384654+0000","last_became_peered":"2024-04-05T23:32:38.384654+0000","last_unstale":"2024-04-05T23:32:38.385626+0000","last_undegraded":"2024-04-05T23:32:38.385626+0000","last_fullsized":"2024-04-05T23:32:38.385626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:55:42.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378881+0000","last_change":"2024-04-05T23:32:04.005978+0000","last_active":"2024-04-05T23:32:37.378881+0000","last_peered":"2024-04-05T23:32:37.378881+0000","last_clean":"2024-04-05T23:32:37.378881+0000","last_became_active":"2024-04-05T23:32:04.005638+0000","last_became_peered":"2024-04-05T23:32:04.005638+0000","last_unstale":"2024-04-05T23:32:37.378881+0000","last_undegraded":"2024-04-05T23:32:37.378881+0000","last_fullsized":"2024-04-05T23:32:37.378881+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:39:35.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380727+0000","last_change":"2024-04-05T23:32:04.006727+0000","last_active":"2024-04-05T23:32:37.380727+0000","last_peered":"2024-04-05T23:32:37.380727+0000","last_clean":"2024-04-05T23:32:37.380727+0000","last_became_active":"2024-04-05T23:32:04.006414+0000","last_became_peered":"2024-04-05T23:32:04.006414+0000","last_unstale":"2024-04-05T23:32:37.380727+0000","last_undegraded":"2024-04-05T23:32:37.380727+0000","last_fullsized":"2024-04-05T23:32:37.380727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:41:28.661118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378376+0000","last_change":"2024-04-05T23:32:04.536893+0000","last_active":"2024-04-05T23:32:37.378376+0000","last_peered":"2024-04-05T23:32:37.378376+0000","last_clean":"2024-04-05T23:32:37.378376+0000","last_became_active":"2024-04-05T23:32:04.536629+0000","last_became_peered":"2024-04-05T23:32:04.536629+0000","last_unstale":"2024-04-05T23:32:37.378376+0000","last_undegraded":"2024-04-05T23:32:37.378376+0000","last_fullsized":"2024-04-05T23:32:37.378376+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:27:58.279768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'10","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375826+0000","last_change":"2024-04-05T23:32:04.010202+0000","last_active":"2024-04-05T23:32:37.375826+0000","last_peered":"2024-04-05T23:32:37.375826+0000","last_clean":"2024-04-05T23:32:37.375826+0000","last_became_active":"2024-04-05T23:32:04.009889+0000","last_became_peered":"2024-04-05T23:32:04.009889+0000","last_unstale":"2024-04-05T23:32:37.375826+0000","last_undegraded":"2024-04-05T23:32:37.375826+0000","last_fullsized":"2024-04-05T23:32:37.375826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:12:13.121449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371893+0000","last_change":"2024-04-05T23:32:04.533576+0000","last_active":"2024-04-05T23:32:37.371893+0000","last_peered":"2024-04-05T23:32:37.371893+0000","last_clean":"2024-04-05T23:32:37.371893+0000","last_became_active":"2024-04-05T23:32:04.533066+0000","last_became_peered":"2024-04-05T23:32:04.533066+0000","last_unstale":"2024-04-05T23:32:37.371893+0000","last_undegraded":"2024-04-05T23:32:37.371893+0000","last_fullsized":"2024-04-05T23:32:37.371893+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:01:44.019210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377470+0000","last_change":"2024-04-05T23:32:04.005096+0000","last_active":"2024-04-05T23:32:37.377470+0000","last_peered":"2024-04-05T23:32:37.377470+0000","last_clean":"2024-04-05T23:32:37.377470+0000","last_became_active":"2024-04-05T23:32:04.004702+0000","last_became_peered":"2024-04-05T23:32:04.004702+0000","last_unstale":"2024-04-05T23:32:37.377470+0000","last_undegraded":"2024-04-05T23:32:37.377470+0000","last_fullsized":"2024-04-05T23:32:37.377470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:33:09.934039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380697+0000","last_change":"2024-04-05T23:32:04.535623+0000","last_active":"2024-04-05T23:32:37.380697+0000","last_peered":"2024-04-05T23:32:37.380697+0000","last_clean":"2024-04-05T23:32:37.380697+0000","last_became_active":"2024-04-05T23:32:04.535223+0000","last_became_peered":"2024-04-05T23:32:04.535223+0000","last_unstale":"2024-04-05T23:32:37.380697+0000","last_undegraded":"2024-04-05T23:32:37.380697+0000","last_fullsized":"2024-04-05T23:32:37.380697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:24:43.543119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380978+0000","last_change":"2024-04-05T23:32:04.004097+0000","last_active":"2024-04-05T23:32:37.380978+0000","last_peered":"2024-04-05T23:32:37.380978+0000","last_clean":"2024-04-05T23:32:37.380978+0000","last_became_active":"2024-04-05T23:32:04.003713+0000","last_became_peered":"2024-04-05T23:32:04.003713+0000","last_unstale":"2024-04-05T23:32:37.380978+0000","last_undegraded":"2024-04-05T23:32:37.380978+0000","last_fullsized":"2024-04-05T23:32:37.380978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:47:43.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380375+0000","last_change":"2024-04-05T23:32:04.538843+0000","last_active":"2024-04-05T23:32:37.380375+0000","last_peered":"2024-04-05T23:32:37.380375+0000","last_clean":"2024-04-05T23:32:37.380375+0000","last_became_active":"2024-04-05T23:32:04.538472+0000","last_became_peered":"2024-04-05T23:32:04.538472+0000","last_unstale":"2024-04-05T23:32:37.380375+0000","last_undegraded":"2024-04-05T23:32:37.380375+0000","last_fullsized":"2024-04-05T23:32:37.380375+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:40:40.863815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371533+0000","last_change":"2024-04-05T23:32:04.013064+0000","last_active":"2024-04-05T23:32:37.371533+0000","last_peered":"2024-04-05T23:32:37.371533+0000","last_clean":"2024-04-05T23:32:37.371533+0000","last_became_active":"2024-04-05T23:32:04.012751+0000","last_became_peered":"2024-04-05T23:32:04.012751+0000","last_unstale":"2024-04-05T23:32:37.371533+0000","last_undegraded":"2024-04-05T23:32:37.371533+0000","last_fullsized":"2024-04-05T23:32:37.371533+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:06:10.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371497+0000","last_change":"2024-04-05T23:32:04.529014+0000","last_active":"2024-04-05T23:32:37.371497+0000","last_peered":"2024-04-05T23:32:37.371497+0000","last_clean":"2024-04-05T23:32:37.371497+0000","last_became_active":"2024-04-05T23:32:04.528622+0000","last_became_peered":"2024-04-05T23:32:04.528622+0000","last_unstale":"2024-04-05T23:32:37.371497+0000","last_undegraded":"2024-04-05T23:32:37.371497+0000","last_fullsized":"2024-04-05T23:32:37.371497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:46:26.390934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375037+0000","last_change":"2024-04-05T23:32:04.002651+0000","last_active":"2024-04-05T23:32:37.375037+0000","last_peered":"2024-04-05T23:32:37.375037+0000","last_clean":"2024-04-05T23:32:37.375037+0000","last_became_active":"2024-04-05T23:32:04.002346+0000","last_became_peered":"2024-04-05T23:32:04.002346+0000","last_unstale":"2024-04-05T23:32:37.375037+0000","last_undegraded":"2024-04-05T23:32:37.375037+0000","last_fullsized":"2024-04-05T23:32:37.375037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:26:48.176254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371637+0000","last_change":"2024-04-05T23:32:04.538785+0000","last_active":"2024-04-05T23:32:37.371637+0000","last_peered":"2024-04-05T23:32:37.371637+0000","last_clean":"2024-04-05T23:32:37.371637+0000","last_became_active":"2024-04-05T23:32:04.538463+0000","last_became_peered":"2024-04-05T23:32:04.538463+0000","last_unstale":"2024-04-05T23:32:37.371637+0000","last_undegraded":"2024-04-05T23:32:37.371637+0000","last_fullsized":"2024-04-05T23:32:37.371637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:44:38.941857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380620+0000","last_change":"2024-04-05T23:32:04.003760+0000","last_active":"2024-04-05T23:32:37.380620+0000","last_peered":"2024-04-05T23:32:37.380620+0000","last_clean":"2024-04-05T23:32:37.380620+0000","last_became_active":"2024-04-05T23:32:04.003389+0000","last_became_peered":"2024-04-05T23:32:04.003389+0000","last_unstale":"2024-04-05T23:32:37.380620+0000","last_undegraded":"2024-04-05T23:32:37.380620+0000","last_fullsized":"2024-04-05T23:32:37.380620+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:42:00.353651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.379178+0000","last_change":"2024-04-05T23:32:04.540051+0000","last_active":"2024-04-05T23:32:37.379178+0000","last_peered":"2024-04-05T23:32:37.379178+0000","last_clean":"2024-04-05T23:32:37.379178+0000","last_became_active":"2024-04-05T23:32:04.539736+0000","last_became_peered":"2024-04-05T23:32:04.539736+0000","last_unstale":"2024-04-05T23:32:37.379178+0000","last_undegraded":"2024-04-05T23:32:37.379178+0000","last_fullsized":"2024-04-05T23:32:37.379178+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:54:14.086904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:46.497151+0000","last_change":"2024-04-05T23:32:04.006492+0000","last_active":"2024-04-05T23:32:46.497151+0000","last_peered":"2024-04-05T23:32:46.497151+0000","last_clean":"2024-04-05T23:32:46.497151+0000","last_became_active":"2024-04-05T23:32:04.006154+0000","last_became_peered":"2024-04-05T23:32:04.006154+0000","last_unstale":"2024-04-05T23:32:46.497151+0000","last_undegraded":"2024-04-05T23:32:46.497151+0000","last_fullsized":"2024-04-05T23:32:46.497151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:53:23.431400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378846+0000","last_change":"2024-04-05T23:32:04.540757+0000","last_active":"2024-04-05T23:32:37.378846+0000","last_peered":"2024-04-05T23:32:37.378846+0000","last_clean":"2024-04-05T23:32:37.378846+0000","last_became_active":"2024-04-05T23:32:04.540359+0000","last_became_peered":"2024-04-05T23:32:04.540359+0000","last_unstale":"2024-04-05T23:32:37.378846+0000","last_undegraded":"2024-04-05T23:32:37.378846+0000","last_fullsized":"2024-04-05T23:32:37.378846+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:43:52.533489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378021+0000","last_change":"2024-04-05T23:32:04.002736+0000","last_active":"2024-04-05T23:32:37.378021+0000","last_peered":"2024-04-05T23:32:37.378021+0000","last_clean":"2024-04-05T23:32:37.378021+0000","last_became_active":"2024-04-05T23:32:04.002277+0000","last_became_peered":"2024-04-05T23:32:04.002277+0000","last_unstale":"2024-04-05T23:32:37.378021+0000","last_undegraded":"2024-04-05T23:32:37.378021+0000","last_fullsized":"2024-04-05T23:32:37.378021+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:56:39.001732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'1","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378607+0000","last_change":"2024-04-05T23:32:04.537076+0000","last_active":"2024-04-05T23:32:37.378607+0000","last_peered":"2024-04-05T23:32:37.378607+0000","last_clean":"2024-04-05T23:32:37.378607+0000","last_became_active":"2024-04-05T23:32:04.536707+0000","last_became_peered":"2024-04-05T23:32:04.536707+0000","last_unstale":"2024-04-05T23:32:37.378607+0000","last_undegraded":"2024-04-05T23:32:37.378607+0000","last_fullsized":"2024-04-05T23:32:37.378607+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:51:09.395408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.379112+0000","last_change":"2024-04-05T23:32:04.006122+0000","last_active":"2024-04-05T23:32:37.379112+0000","last_peered":"2024-04-05T23:32:37.379112+0000","last_clean":"2024-04-05T23:32:37.379112+0000","last_became_active":"2024-04-05T23:32:04.005683+0000","last_became_peered":"2024-04-05T23:32:04.005683+0000","last_unstale":"2024-04-05T23:32:37.379112+0000","last_undegraded":"2024-04-05T23:32:37.379112+0000","last_fullsized":"2024-04-05T23:32:37.379112+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:21:40.922614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378878+0000","last_change":"2024-04-05T23:32:04.533321+0000","last_active":"2024-04-05T23:32:37.378878+0000","last_peered":"2024-04-05T23:32:37.378878+0000","last_clean":"2024-04-05T23:32:37.378878+0000","last_became_active":"2024-04-05T23:32:04.532949+0000","last_became_peered":"2024-04-05T23:32:04.532949+0000","last_unstale":"2024-04-05T23:32:37.378878+0000","last_undegraded":"2024-04-05T23:32:37.378878+0000","last_fullsized":"2024-04-05T23:32:37.378878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:31.500132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371547+0000","last_change":"2024-04-05T23:32:04.536508+0000","last_active":"2024-04-05T23:32:37.371547+0000","last_peered":"2024-04-05T23:32:37.371547+0000","last_clean":"2024-04-05T23:32:37.371547+0000","last_became_active":"2024-04-05T23:32:04.536142+0000","last_became_peered":"2024-04-05T23:32:04.536142+0000","last_unstale":"2024-04-05T23:32:37.371547+0000","last_undegraded":"2024-04-05T23:32:37.371547+0000","last_fullsized":"2024-04-05T23:32:37.371547+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:18:28.497807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375905+0000","last_change":"2024-04-05T23:32:04.545511+0000","last_active":"2024-04-05T23:32:37.375905+0000","last_peered":"2024-04-05T23:32:37.375905+0000","last_clean":"2024-04-05T23:32:37.375905+0000","last_became_active":"2024-04-05T23:32:04.545191+0000","last_became_peered":"2024-04-05T23:32:04.545191+0000","last_unstale":"2024-04-05T23:32:37.375905+0000","last_undegraded":"2024-04-05T23:32:37.375905+0000","last_fullsized":"2024-04-05T23:32:37.375905+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:17:06.431429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375067+0000","last_change":"2024-04-05T23:32:04.549673+0000","last_active":"2024-04-05T23:32:37.375067+0000","last_peered":"2024-04-05T23:32:37.375067+0000","last_clean":"2024-04-05T23:32:37.375067+0000","last_became_active":"2024-04-05T23:32:04.549391+0000","last_became_peered":"2024-04-05T23:32:04.549391+0000","last_unstale":"2024-04-05T23:32:37.375067+0000","last_undegraded":"2024-04-05T23:32:37.375067+0000","last_fullsized":"2024-04-05T23:32:37.375067+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:30:22.689107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380491+0000","last_change":"2024-04-05T23:32:04.535735+0000","last_active":"2024-04-05T23:32:37.380491+0000","last_peered":"2024-04-05T23:32:37.380491+0000","last_clean":"2024-04-05T23:32:37.380491+0000","last_became_active":"2024-04-05T23:32:04.534461+0000","last_became_peered":"2024-04-05T23:32:04.534461+0000","last_unstale":"2024-04-05T23:32:37.380491+0000","last_undegraded":"2024-04-05T23:32:37.380491+0000","last_fullsized":"2024-04-05T23:32:37.380491+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:37:41.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378465+0000","last_change":"2024-04-05T23:32:04.532340+0000","last_active":"2024-04-05T23:32:37.378465+0000","last_peered":"2024-04-05T23:32:37.378465+0000","last_clean":"2024-04-05T23:32:37.378465+0000","last_became_active":"2024-04-05T23:32:04.530053+0000","last_became_peered":"2024-04-05T23:32:04.530053+0000","last_unstale":"2024-04-05T23:32:37.378465+0000","last_undegraded":"2024-04-05T23:32:37.378465+0000","last_fullsized":"2024-04-05T23:32:37.378465+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:16:11.168366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371640+0000","last_change":"2024-04-05T23:32:04.529972+0000","last_active":"2024-04-05T23:32:37.371640+0000","last_peered":"2024-04-05T23:32:37.371640+0000","last_clean":"2024-04-05T23:32:37.371640+0000","last_became_active":"2024-04-05T23:32:04.529551+0000","last_became_peered":"2024-04-05T23:32:04.529551+0000","last_unstale":"2024-04-05T23:32:37.371640+0000","last_undegraded":"2024-04-05T23:32:37.371640+0000","last_fullsized":"2024-04-05T23:32:37.371640+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:13.764971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.375667+0000","last_change":"2024-04-05T23:32:04.529166+0000","last_active":"2024-04-05T23:32:37.375667+0000","last_peered":"2024-04-05T23:32:37.375667+0000","last_clean":"2024-04-05T23:32:37.375667+0000","last_became_active":"2024-04-05T23:32:04.528785+0000","last_became_peered":"2024-04-05T23:32:04.528785+0000","last_unstale":"2024-04-05T23:32:37.375667+0000","last_undegraded":"2024-04-05T23:32:37.375667+0000","last_fullsized":"2024-04-05T23:32:37.375667+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:29:43.964265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371809+0000","last_change":"2024-04-05T23:32:04.540631+0000","last_active":"2024-04-05T23:32:37.371809+0000","last_peered":"2024-04-05T23:32:37.371809+0000","last_clean":"2024-04-05T23:32:37.371809+0000","last_became_active":"2024-04-05T23:32:04.540327+0000","last_became_peered":"2024-04-05T23:32:04.540327+0000","last_unstale":"2024-04-05T23:32:37.371809+0000","last_undegraded":"2024-04-05T23:32:37.371809+0000","last_fullsized":"2024-04-05T23:32:37.371809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:22:48.825324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.377032+0000","last_change":"2024-04-05T23:32:04.544845+0000","last_active":"2024-04-05T23:32:37.377032+0000","last_peered":"2024-04-05T23:32:37.377032+0000","last_clean":"2024-04-05T23:32:37.377032+0000","last_became_active":"2024-04-05T23:32:04.544472+0000","last_became_peered":"2024-04-05T23:32:04.544472+0000","last_unstale":"2024-04-05T23:32:37.377032+0000","last_undegraded":"2024-04-05T23:32:37.377032+0000","last_fullsized":"2024-04-05T23:32:37.377032+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:34:09.529644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.383681+0000","last_change":"2024-04-05T23:32:38.382577+0000","last_active":"2024-04-05T23:32:38.383681+0000","last_peered":"2024-04-05T23:32:38.383681+0000","last_clean":"2024-04-05T23:32:38.383681+0000","last_became_active":"2024-04-05T23:32:38.380286+0000","last_became_peered":"2024-04-05T23:32:38.380286+0000","last_unstale":"2024-04-05T23:32:38.383681+0000","last_undegraded":"2024-04-05T23:32:38.383681+0000","last_fullsized":"2024-04-05T23:32:38.383681+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:42:52.270935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.380994+0000","last_change":"2024-04-05T23:32:04.534631+0000","last_active":"2024-04-05T23:32:37.380994+0000","last_peered":"2024-04-05T23:32:37.380994+0000","last_clean":"2024-04-05T23:32:37.380994+0000","last_became_active":"2024-04-05T23:32:04.534330+0000","last_became_peered":"2024-04-05T23:32:04.534330+0000","last_unstale":"2024-04-05T23:32:37.380994+0000","last_undegraded":"2024-04-05T23:32:37.380994+0000","last_fullsized":"2024-04-05T23:32:37.380994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:20:47.592174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.378464+0000","last_change":"2024-04-05T23:32:04.531459+0000","last_active":"2024-04-05T23:32:37.378464+0000","last_peered":"2024-04-05T23:32:37.378464+0000","last_clean":"2024-04-05T23:32:37.378464+0000","last_became_active":"2024-04-05T23:32:04.530828+0000","last_became_peered":"2024-04-05T23:32:04.530828+0000","last_unstale":"2024-04-05T23:32:37.378464+0000","last_undegraded":"2024-04-05T23:32:37.378464+0000","last_fullsized":"2024-04-05T23:32:37.378464+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:51:34.090030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-05T23:32:37.371594+0000","last_change":"2024-04-05T23:32:04.527711+0000","last_active":"2024-04-05T23:32:37.371594+0000","last_peered":"2024-04-05T23:32:37.371594+0000","last_clean":"2024-04-05T23:32:37.371594+0000","last_became_active":"2024-04-05T23:32:04.527374+0000","last_became_peered":"2024-04-05T23:32:04.527374+0000","last_unstale":"2024-04-05T23:32:37.371594+0000","last_undegraded":"2024-04-05T23:32:37.371594+0000","last_fullsized":"2024-04-05T23:32:37.371594+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:15:10.764340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66060,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":419,"num_read_kb":283,"num_write":232,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":198180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579924,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":897024,"data_stored":430251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.56000000000000005}]},{"osd":1,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.625}]},{"osd":6,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.74099999999999999}]},{"osd":10,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":0.78800000000000003}]}]},{"osd":10,"up_from":67,"seq":287762808855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1589248,"data_stored":1081166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.503}]},{"osd":2,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.83299999999999996}]},{"osd":4,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.441},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.38800000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.77500000000000002}]},{"osd":6,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.74399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.61499999999999999}]},{"osd":9,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.68300000000000005}]},{"osd":11,"last update":"Fri Apr 5 23:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.73799999999999999}]}]},{"osd":9,"up_from":62,"seq":266287972379,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":430285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.443}]},{"osd":1,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.58699999999999997}]},{"osd":3,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.35299999999999998}]},{"osd":4,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.65800000000000003}]},{"osd":8,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.75600000000000001}]},{"osd":10,"last update":"Fri Apr 5 23:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.627},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.64300000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":8,"up_from":55,"seq":236223201312,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":430081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71699999999999997,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.36899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77800000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47299999999999998}]},{"osd":1,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.502},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.45800000000000002,"15min":0.45700000000000002},"min":{"1min":0.30199999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69699999999999995,"5min":0.748,"15min":0.748},"last":0.45000000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67800000000000005}]},{"osd":4,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.38900000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76200000000000001,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68799999999999994,"15min":0.69199999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.69999999999999996,"15min":0.70699999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90100000000000002,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.748}]},{"osd":7,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.72399999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.83099999999999996}]},{"osd":9,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.81699999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.84599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":7,"up_from":48,"seq":206158430242,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32592,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712560,"statfs":{"total":95995035648,"available":95961661440,"internally_reserved":0,"allocated":1507328,"data_stored":1022646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.76400000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.433,"15min":0.42299999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42399999999999999,"15min":0.41899999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.496}]},{"osd":2,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55700000000000005,"15min":0.55600000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48899999999999999}]},{"osd":3,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.621,"15min":0.60999999999999999},"min":{"1min":0.44700000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.60699999999999998}]},{"osd":4,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.32900000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.38100000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.625,"15min":0.623},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.438},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.65400000000000003,"15min":0.65000000000000002},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.51500000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.69399999999999995,"15min":0.68400000000000005},"min":{"1min":0.498,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68899999999999995,"15min":0.68100000000000005},"min":{"1min":0.502,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.47999999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.67600000000000005,"15min":0.68600000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72699999999999998,"15min":0.72799999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.89500000000000002,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.54300000000000004}]},{"osd":9,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.68700000000000006}]},{"osd":10,"last update":"Fri Apr 5 23:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.63},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.73599999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]},{"osd":6,"up_from":42,"seq":180388626471,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":429681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47799999999999998,"15min":0.47499999999999998},"min":{"1min":0.34000000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73499999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.505,"15min":0.5},"min":{"1min":0.36399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.79600000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.495}]},{"osd":1,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.45900000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.90400000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.66200000000000003}]},{"osd":2,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.627,"15min":0.627},"min":{"1min":0.39500000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92400000000000004,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.41799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85599999999999998,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.64400000000000002}]},{"osd":3,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78500000000000003,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67400000000000004,"15min":0.66900000000000004},"min":{"1min":0.47299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.92800000000000005,"5min":1.246,"15min":1.246},"last":0.73499999999999999}]},{"osd":4,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.50600000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.90200000000000002,"5min":1.095,"15min":1.095},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.754,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.77700000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.49099999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.016,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.73899999999999999},"min":{"1min":0.54900000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0509999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.91100000000000003}]},{"osd":7,"last update":"Fri Apr 5 23:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.77200000000000002,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.94599999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0089999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.79900000000000004}]},{"osd":8,"last update":"Fri Apr 5 23:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78400000000000003,"15min":0.79100000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.026,"5min":1.127,"15min":1.127},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.80000000000000004,"15min":0.79200000000000004},"min":{"1min":0.64000000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.94199999999999995}]},{"osd":9,"last update":"Fri Apr 5 23:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.755}]},{"osd":10,"last update":"Fri Apr 5 23:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.878}]}]},{"osd":1,"up_from":12,"seq":51539607610,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":954368,"data_stored":490424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.51600000000000001,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80900000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63500000000000001,"15min":0.68300000000000005},"min":{"1min":0.41399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.0069999999999999,"5min":4.4199999999999999,"15min":4.4199999999999999},"last":0.56699999999999995}]},{"osd":2,"last update":"Fri Apr 5 23:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55200000000000005,"15min":0.52000000000000002},"min":{"1min":0.34699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.57299999999999995,"15min":0.52100000000000002},"min":{"1min":0.53400000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.753}]},{"osd":3,"last update":"Fri Apr 5 23:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.56299999999999994,"15min":0.53400000000000003},"min":{"1min":0.44700000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59599999999999997}]},{"osd":4,"last update":"Fri Apr 5 23:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.42799999999999999,"15min":0.40500000000000003},"min":{"1min":0.33700000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.495},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.46800000000000003,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.877,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.61399999999999999,"15min":0.59299999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.68300000000000005,"15min":0.66200000000000003},"min":{"1min":0.52100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.79900000000000004}]},{"osd":6,"last update":"Fri Apr 5 23:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.68700000000000006,"15min":0.67900000000000005},"min":{"1min":0.5,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65100000000000002,"15min":0.64800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Apr 5 23:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.621,"15min":0.60599999999999998},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.38600000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.85499999999999998,"5min":1.002,"15min":1.002},"last":0.81899999999999995}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.624,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.72499999999999998,"15min":0.70899999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.083,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.95499999999999996}]},{"osd":9,"last update":"Fri Apr 5 23:31:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.86899999999999999}]},{"osd":10,"last update":"Fri Apr 5 23:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.93000000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36112,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709040,"statfs":{"total":95995035648,"available":95958056960,"internally_reserved":0,"allocated":917504,"data_stored":431293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 23:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.036,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53700000000000003,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82299999999999995,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.63200000000000001,"15min":0.58799999999999997},"min":{"1min":0.47999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0049999999999999,"5min":7.2060000000000004,"15min":7.2060000000000004},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:30:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.54200000000000004,"15min":0.52800000000000002},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.85699999999999998,"15min":0.93700000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82499999999999996,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67800000000000005,"15min":0.65600000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.052,"5min":1.395,"15min":1.395},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68899999999999995,"15min":0.66800000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.70999999999999996},"min":{"1min":0.55900000000000005,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.70099999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.88100000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.153,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.77800000000000002,"15min":0.77500000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.71099999999999997}]},{"osd":7,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.70699999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.92300000000000004,"15min":0.93999999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.98199999999999998,"15min":1.01},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1379999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.83299999999999996}]},{"osd":9,"last update":"Fri Apr 5 23:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.879},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.81899999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:32:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.90500000000000003}]},{"osd":11,"last update":"Fri Apr 5 23:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.752},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.84499999999999997}]}]},{"osd":2,"up_from":17,"seq":73014444087,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":966656,"data_stored":490528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76400000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.751,"5min":0.874,"15min":0.874},"last":0.76500000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55500000000000005,"15min":0.53800000000000003},"min":{"1min":0.37,"5min":0.311,"15min":0.311},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:31:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.67600000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.54500000000000004},"min":{"1min":0.43099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.72099999999999997}]},{"osd":5,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.503,"15min":0.47799999999999998},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86699999999999999,"5min":1.359,"15min":1.359},"last":0.61599999999999999}]},{"osd":6,"last update":"Fri Apr 5 23:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78500000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93899999999999995,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.66500000000000004}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73199999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.48399999999999999,"15min":0.47299999999999998},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70499999999999996,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Apr 5 23:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.80500000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.81699999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":3,"up_from":23,"seq":98784247858,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36104,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709048,"statfs":{"total":95995035648,"available":95958065152,"internally_reserved":0,"allocated":909312,"data_stored":429554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.45100000000000001},"min":{"1min":0.33500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.018,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51000000000000001,"15min":0.46300000000000002},"min":{"1min":0.379,"5min":0.313,"15min":0.313},"max":{"1min":1.0860000000000001,"5min":4.96,"15min":4.96},"last":0.379}]},{"osd":1,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64000000000000001,"15min":0.60599999999999998},"min":{"1min":0.44400000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91200000000000003,"5min":4.5590000000000002,"15min":4.5590000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.61599999999999999,"15min":0.59599999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71599999999999997,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.72799999999999998,"15min":0.67800000000000005},"min":{"1min":0.51400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.028,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71099999999999997,"15min":0.67300000000000004},"min":{"1min":0.57199999999999995,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94399999999999995,"5min":4.6710000000000003,"15min":4.6710000000000003},"last":0.53200000000000003}]},{"osd":4,"last update":"Fri Apr 5 23:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71299999999999997,"15min":0.66900000000000004},"min":{"1min":0.46700000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1479999999999999,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.71599999999999997,"15min":0.66600000000000004},"min":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.028,"5min":4.6289999999999996,"15min":4.6289999999999996},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Apr 5 23:32:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.752,"15min":0.69799999999999995},"min":{"1min":0.56000000000000005,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0860000000000001,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.77800000000000002,"15min":0.72599999999999998},"min":{"1min":0.42799999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.069,"5min":4.8929999999999998,"15min":4.8929999999999998},"last":0.47999999999999998}]},{"osd":6,"last update":"Fri Apr 5 23:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.878,"15min":0.88300000000000001},"min":{"1min":0.63400000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.228,"5min":4.8159999999999998,"15min":4.8159999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.78600000000000003,"15min":0.81399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":5.0979999999999999,"15min":5.0979999999999999},"last":0.72699999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.97499999999999998,"15min":1},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2929999999999999,"5min":4.9980000000000002,"15min":4.9980000000000002},"last":0.749},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.93500000000000005,"15min":0.96299999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1020000000000001,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.77200000000000002}]},{"osd":8,"last update":"Fri Apr 5 23:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":4.7569999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.0300000000000002,"5min":5.0300000000000002,"15min":5.0300000000000002},"last":0.72199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.57099999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.84199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]}]},{"osd":4,"up_from":30,"seq":128849018927,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":429542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.38600000000000001,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.443},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.40200000000000002,"15min":0.38500000000000001},"min":{"1min":0.27200000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Apr 5 23:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.46100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Apr 5 23:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.58799999999999997,"15min":0.56599999999999995},"min":{"1min":0.438,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.627},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57099999999999995,"15min":0.55900000000000005},"min":{"1min":0.35899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63800000000000001,"15min":0.60899999999999999},"min":{"1min":0.53300000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.77100000000000002}]},{"osd":6,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.61299999999999999,"15min":0.59499999999999997},"min":{"1min":0.435,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Fri Apr 5 23:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.56899999999999995,"15min":0.55500000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.60299999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.99299999999999999,"5min":1.087,"15min":1.087},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.754,"15min":0.746},"min":{"1min":0.53200000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0069999999999999,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.877}]},{"osd":9,"last update":"Fri Apr 5 23:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.67600000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.80800000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]}]},{"osd":5,"up_from":35,"seq":150323855402,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28508,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716644,"statfs":{"total":95995035648,"available":95965843456,"internally_reserved":0,"allocated":1519616,"data_stored":1022420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.40300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41499999999999998}]},{"osd":1,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53400000000000003,"15min":0.52400000000000002},"min":{"1min":0.36799999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72299999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.40600000000000003,"15min":0.40200000000000002},"min":{"1min":0.31,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66000000000000003,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.36799999999999999}]},{"osd":3,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75600000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86199999999999999,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51500000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56399999999999995,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77500000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.46000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.68600000000000005}]},{"osd":6,"last update":"Fri Apr 5 23:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65700000000000003,"15min":0.65900000000000003},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.78900000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77900000000000003,"5min":1.153,"15min":1.153},"last":0.48199999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.375,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60799999999999998,"15min":0.60899999999999999},"min":{"1min":0.41899999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.67100000000000004}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.81100000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.61299999999999999}]},{"osd":9,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.60199999999999998}]},{"osd":10,"last update":"Fri Apr 5 23:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T23:32:52.929 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-05T23:32:52.930 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-05T23:32:52.930 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-05T23:32:52.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-05T23:32:53.334 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:32:54.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.532+0000 7f3646800700 1 -- 172.21.15.16:0/1743996624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640106f70 msgr2=0x7f3640107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:54.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.532+0000 7f3646800700 1 --2- 172.21.15.16:0/1743996624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640106f70 0x7f3640107350 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3630009990 tx=0x7f3630009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.532+0000 7f3646800700 1 -- 172.21.15.16:0/1743996624 shutdown_connections 2024-04-05T23:32:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.532+0000 7f3646800700 1 --2- 172.21.15.16:0/1743996624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640106f70 0x7f3640107350 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.532+0000 7f3646800700 1 --2- 172.21.15.16:0/1743996624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36401018d0 0x7f364010ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.532+0000 7f3646800700 1 --2- 172.21.15.16:0/1743996624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640100f30 0x7f3640101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.532+0000 7f3646800700 1 -- 172.21.15.16:0/1743996624 >> 172.21.15.16:0/1743996624 conn(0x7f36400780d0 msgr2=0x7f36400784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.532+0000 7f3646800700 1 -- 172.21.15.16:0/1743996624 shutdown_connections 2024-04-05T23:32:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.532+0000 7f3646800700 1 -- 172.21.15.16:0/1743996624 wait complete. 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.533+0000 7f3646800700 1 Processor -- start 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.533+0000 7f3646800700 1 -- start start 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f3646800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640100f30 0x7f36400731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f3646800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36401018d0 0x7f3640073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f3646800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106f70 0x7f3640073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f3646800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36401a56f0 con 0x7f3640100f30 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f3646800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36401a5860 con 0x7f3640106f70 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f3646800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36401a5a20 con 0x7f36401018d0 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f363f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640100f30 0x7f36400731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f3637e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36401018d0 0x7f3640073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f363f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640100f30 0x7f36400731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55572/0 (socket says 172.21.15.16:55572) 2024-04-05T23:32:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f363f400700 1 -- 172.21.15.16:0/1445066824 learned_addr learned my addr 172.21.15.16:0/1445066824 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.534+0000 7f363fe00700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106f70 0x7f3640073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f363f400700 1 -- 172.21.15.16:0/1445066824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36401018d0 msgr2=0x7f3640073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f363f400700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36401018d0 0x7f3640073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f363f400700 1 -- 172.21.15.16:0/1445066824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106f70 msgr2=0x7f3640073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f363f400700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106f70 0x7f3640073c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f363f400700 1 -- 172.21.15.16:0/1445066824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36300095f0 con 0x7f3640100f30 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f363f400700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640100f30 0x7f36400731e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3630005910 tx=0x7f3630004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f363cc00700 1 -- 172.21.15.16:0/1445066824 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363001d070 con 0x7f3640100f30 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f363cc00700 1 -- 172.21.15.16:0/1445066824 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3630022470 con 0x7f3640100f30 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f363cc00700 1 -- 172.21.15.16:0/1445066824 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630017650 con 0x7f3640100f30 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36401a5ca0 con 0x7f3640100f30 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.535+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36401a6060 con 0x7f3640100f30 2024-04-05T23:32:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.536+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3640109490 con 0x7f3640100f30 2024-04-05T23:32:54.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.538+0000 7f363cc00700 1 -- 172.21.15.16:0/1445066824 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f36300225e0 con 0x7f3640100f30 2024-04-05T23:32:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.540+0000 7f363cc00700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f36240a11b0 0x7f36240a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.540+0000 7f363cc00700 1 -- 172.21.15.16:0/1445066824 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f36300cdcc0 con 0x7f3640100f30 2024-04-05T23:32:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.540+0000 7f3637e00700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f36240a11b0 0x7f36240a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.540+0000 7f3637e00700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f36240a11b0 0x7f36240a3670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3640074910 tx=0x7f3628009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.544+0000 7f363cc00700 1 -- 172.21.15.16:0/1445066824 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3630097390 con 0x7f3640100f30 2024-04-05T23:32:54.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:54 smithi016 ceph-mon[38248]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:54.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:54 smithi016 ceph-mon[38248]: pgmap v259: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-04-05T23:32:54.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:54 smithi027 ceph-mon[78597]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:54.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:54 smithi027 ceph-mon[78597]: pgmap v259: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-04-05T23:32:54.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.804+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3640102620 con 0x7f3640100f30 2024-04-05T23:32:54.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.804+0000 7f363cc00700 1 -- 172.21.15.16:0/1445066824 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f363009a9b0 con 0x7f3640100f30 2024-04-05T23:32:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:54.798 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f36240a11b0 msgr2=0x7f36240a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f36240a11b0 0x7f36240a3670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3640074910 tx=0x7f3628009260 comp rx=0 tx=0).stop 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640100f30 msgr2=0x7f36400731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640100f30 0x7f36400731e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3630005910 tx=0x7f3630004730 comp rx=0 tx=0).stop 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 shutdown_connections 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f36240a11b0 0x7f36240a3670 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3640100f30 0x7f36400731e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3640106f70 0x7f3640073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 --2- 172.21.15.16:0/1445066824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36401018d0 0x7f3640073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 >> 172.21.15.16:0/1445066824 conn(0x7f36400780d0 msgr2=0x7f36400ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 shutdown_connections 2024-04-05T23:32:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:54.807+0000 7f3646800700 1 -- 172.21.15.16:0/1445066824 wait complete. 2024-04-05T23:32:54.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:54 smithi130 ceph-mon[79417]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:32:54.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:54 smithi130 ceph-mon[79417]: pgmap v259: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-04-05T23:32:55.161 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-05T23:32:55.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:32:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.389+0000 7fc9c3600700 1 -- 172.21.15.16:0/4126112314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 msgr2=0x7fc9bc1011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.389+0000 7fc9c3600700 1 --2- 172.21.15.16:0/4126112314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 0x7fc9bc1011d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc9b00099c0 tx=0x7fc9b0009cd0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.389+0000 7fc9c3600700 1 -- 172.21.15.16:0/4126112314 shutdown_connections 2024-04-05T23:32:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.389+0000 7fc9c3600700 1 --2- 172.21.15.16:0/4126112314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 0x7fc9bc1011d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.389+0000 7fc9c3600700 1 --2- 172.21.15.16:0/4126112314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9bc102940 0x7fc9bc1091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.389+0000 7fc9c3600700 1 --2- 172.21.15.16:0/4126112314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9bc101fa0 0x7fc9bc102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.389+0000 7fc9c3600700 1 -- 172.21.15.16:0/4126112314 >> 172.21.15.16:0/4126112314 conn(0x7fc9bc0fc320 msgr2=0x7fc9bc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.389+0000 7fc9c3600700 1 -- 172.21.15.16:0/4126112314 shutdown_connections 2024-04-05T23:32:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.389+0000 7fc9c3600700 1 -- 172.21.15.16:0/4126112314 wait complete. 2024-04-05T23:32:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.390+0000 7fc9c3600700 1 Processor -- start 2024-04-05T23:32:55.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.390+0000 7fc9c3600700 1 -- start start 2024-04-05T23:32:55.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c3600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 0x7fc9bc198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:55.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c3600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9bc101fa0 0x7fc9bc1994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:55.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c3600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9bc102940 0x7fc9bc1a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c3600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc1a0a50 con 0x7fc9bc100db0 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c3600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc1a0b90 con 0x7fc9bc102940 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c3600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc1a0d20 con 0x7fc9bc101fa0 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c0c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 0x7fc9bc198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9bc102940 0x7fc9bc1a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9bbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9bc101fa0 0x7fc9bc1994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c0c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 0x7fc9bc198f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55590/0 (socket says 172.21.15.16:55590) 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c0c00700 1 -- 172.21.15.16:0/2455919975 learned_addr learned my addr 172.21.15.16:0/2455919975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c0c00700 1 -- 172.21.15.16:0/2455919975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9bc101fa0 msgr2=0x7fc9bc1994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c0c00700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9bc101fa0 0x7fc9bc1994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c0c00700 1 -- 172.21.15.16:0/2455919975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9bc102940 msgr2=0x7fc9bc1a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c0c00700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9bc102940 0x7fc9bc1a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.391+0000 7fc9c0c00700 1 -- 172.21.15.16:0/2455919975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b00095f0 con 0x7fc9bc100db0 2024-04-05T23:32:55.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.392+0000 7fc9c0c00700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 0x7fc9bc198f70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0006010 tx=0x7fc9b000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:55.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.392+0000 7fc9b9600700 1 -- 172.21.15.16:0/2455919975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b001d070 con 0x7fc9bc100db0 2024-04-05T23:32:55.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.392+0000 7fc9b9600700 1 -- 172.21.15.16:0/2455919975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9b0005020 con 0x7fc9bc100db0 2024-04-05T23:32:55.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.392+0000 7fc9b9600700 1 -- 172.21.15.16:0/2455919975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0004b10 con 0x7fc9bc100db0 2024-04-05T23:32:55.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.392+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc1a0fa0 con 0x7fc9bc100db0 2024-04-05T23:32:55.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.392+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc1a1310 con 0x7fc9bc100db0 2024-04-05T23:32:55.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.393+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc067bc0 con 0x7fc9bc100db0 2024-04-05T23:32:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.395+0000 7fc9b9600700 1 -- 172.21.15.16:0/2455919975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fc9b0005190 con 0x7fc9bc100db0 2024-04-05T23:32:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.396+0000 7fc9b9600700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc9a40a1230 0x7fc9a40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.396+0000 7fc9b9600700 1 -- 172.21.15.16:0/2455919975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc9b00cd450 con 0x7fc9bc100db0 2024-04-05T23:32:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.397+0000 7fc9bbe00700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc9a40a1230 0x7fc9a40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.397+0000 7fc9bbe00700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc9a40a1230 0x7fc9a40a36f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009c60 tx=0x7fc9a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.401+0000 7fc9b9600700 1 -- 172.21.15.16:0/2455919975 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc9bc067bc0 con 0x7fc9bc100db0 2024-04-05T23:32:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.620+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9bc067b50 con 0x7fc9bc100db0 2024-04-05T23:32:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.622+0000 7fc9b9600700 1 -- 172.21.15.16:0/2455919975 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22519 (secure 0 0 0) 0x7fc9b0096ac0 con 0x7fc9bc100db0 2024-04-05T23:32:55.616 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:55.616 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:38.369016+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:32:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc9a40a1230 msgr2=0x7fc9a40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc9a40a1230 0x7fc9a40a36f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009c60 tx=0x7fc9a8009260 comp rx=0 tx=0).stop 2024-04-05T23:32:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 msgr2=0x7fc9bc198f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 0x7fc9bc198f70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0006010 tx=0x7fc9b000f690 comp rx=0 tx=0).stop 2024-04-05T23:32:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 shutdown_connections 2024-04-05T23:32:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc9a40a1230 0x7fc9a40a36f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc100db0 0x7fc9bc198f70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9bc102940 0x7fc9bc1a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 --2- 172.21.15.16:0/2455919975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9bc101fa0 0x7fc9bc1994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 >> 172.21.15.16:0/2455919975 conn(0x7fc9bc0fc320 msgr2=0x7fc9bc073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 shutdown_connections 2024-04-05T23:32:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.625+0000 7fc9c3600700 1 -- 172.21.15.16:0/2455919975 wait complete. 2024-04-05T23:32:55.638 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-05T23:32:02.641673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-05T23:32:03.972521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:32:55.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:32:55.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:55 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1445066824' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-05T23:32:55.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:55 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1445066824' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-05T23:32:55.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:55 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1445066824' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-05T23:32:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.847+0000 7fbfab800700 1 -- 172.21.15.16:0/646858324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa4100e70 msgr2=0x7fbfa4101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.847+0000 7fbfab800700 1 --2- 172.21.15.16:0/646858324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa4100e70 0x7fbfa4101290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbf980098f0 tx=0x7fbf98009c00 comp rx=0 tx=0).stop 2024-04-05T23:32:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.847+0000 7fbfab800700 1 -- 172.21.15.16:0/646858324 shutdown_connections 2024-04-05T23:32:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.847+0000 7fbfab800700 1 --2- 172.21.15.16:0/646858324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa4102a10 0x7fbfa4109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.847+0000 7fbfab800700 1 --2- 172.21.15.16:0/646858324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa4100e70 0x7fbfa4101290 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.847+0000 7fbfab800700 1 --2- 172.21.15.16:0/646858324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4102050 0x7fbfa41024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.847+0000 7fbfab800700 1 -- 172.21.15.16:0/646858324 >> 172.21.15.16:0/646858324 conn(0x7fbfa40fc430 msgr2=0x7fbfa40fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.847+0000 7fbfab800700 1 -- 172.21.15.16:0/646858324 shutdown_connections 2024-04-05T23:32:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.847+0000 7fbfab800700 1 -- 172.21.15.16:0/646858324 wait complete. 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.848+0000 7fbfab800700 1 Processor -- start 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.848+0000 7fbfab800700 1 -- start start 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.848+0000 7fbfab800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa4100e70 0x7fbfa4074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.848+0000 7fbfab800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4102050 0x7fbfa40732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfab800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa4102a10 0x7fbfa4073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfab800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4073fb0 con 0x7fbfa4100e70 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfab800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4074120 con 0x7fbfa4102a10 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfab800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa41a0f40 con 0x7fbfa4102050 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa8e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa4100e70 0x7fbfa4074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa8e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa4100e70 0x7fbfa4074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55610/0 (socket says 172.21.15.16:55610) 2024-04-05T23:32:55.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa8e00700 1 -- 172.21.15.16:0/3312563504 learned_addr learned my addr 172.21.15.16:0/3312563504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa9800700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa4102a10 0x7fbfa4073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa3e00700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4102050 0x7fbfa40732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa8e00700 1 -- 172.21.15.16:0/3312563504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4102050 msgr2=0x7fbfa40732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa8e00700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4102050 0x7fbfa40732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa8e00700 1 -- 172.21.15.16:0/3312563504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa4102a10 msgr2=0x7fbfa4073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa8e00700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa4102a10 0x7fbfa4073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa8e00700 1 -- 172.21.15.16:0/3312563504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c009d00 con 0x7fbfa4100e70 2024-04-05T23:32:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.849+0000 7fbfa8e00700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa4100e70 0x7fbfa4074c40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800fb70 tx=0x7fbf9800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.850+0000 7fbfa1600700 1 -- 172.21.15.16:0/3312563504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9801c070 con 0x7fbfa4100e70 2024-04-05T23:32:55.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.850+0000 7fbfa1600700 1 -- 172.21.15.16:0/3312563504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf9800f7d0 con 0x7fbfa4100e70 2024-04-05T23:32:55.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.850+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf98009520 con 0x7fbfa4100e70 2024-04-05T23:32:55.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.850+0000 7fbfa1600700 1 -- 172.21.15.16:0/3312563504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf980039b0 con 0x7fbfa4100e70 2024-04-05T23:32:55.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.850+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa41a1400 con 0x7fbfa4100e70 2024-04-05T23:32:55.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.851+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa41083c0 con 0x7fbfa4100e70 2024-04-05T23:32:55.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.853+0000 7fbfa1600700 1 -- 172.21.15.16:0/3312563504 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fbf98020030 con 0x7fbfa4100e70 2024-04-05T23:32:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.854+0000 7fbfa1600700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf8c0a1230 0x7fbf8c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.854+0000 7fbfa1600700 1 -- 172.21.15.16:0/3312563504 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbf980cdad0 con 0x7fbfa4100e70 2024-04-05T23:32:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.854+0000 7fbfa3e00700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf8c0a1230 0x7fbf8c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.855+0000 7fbfa3e00700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf8c0a1230 0x7fbf8c0a36f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4100b90 tx=0x7fbf90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:55.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:55.858+0000 7fbfa1600700 1 -- 172.21.15.16:0/3312563504 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbf98097140 con 0x7fbfa4100e70 2024-04-05T23:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.076+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbfa4107da0 con 0x7fbfa4100e70 2024-04-05T23:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.077+0000 7fbfa1600700 1 -- 172.21.15.16:0/3312563504 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbf9809a760 con 0x7fbfa4100e70 2024-04-05T23:32:56.069 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:32:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf8c0a1230 msgr2=0x7fbf8c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf8c0a1230 0x7fbf8c0a36f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4100b90 tx=0x7fbf90009260 comp rx=0 tx=0).stop 2024-04-05T23:32:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa4100e70 msgr2=0x7fbfa4074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa4100e70 0x7fbfa4074c40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800fb70 tx=0x7fbf9800b790 comp rx=0 tx=0).stop 2024-04-05T23:32:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 shutdown_connections 2024-04-05T23:32:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbf8c0a1230 0x7fbf8c0a36f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa4100e70 0x7fbfa4074c40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa4102a10 0x7fbfa4073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 --2- 172.21.15.16:0/3312563504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4102050 0x7fbfa40732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 >> 172.21.15.16:0/3312563504 conn(0x7fbfa40fc430 msgr2=0x7fbfa4105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 shutdown_connections 2024-04-05T23:32:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.079+0000 7fbfab800700 1 -- 172.21.15.16:0/3312563504 wait complete. 2024-04-05T23:32:56.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-05T23:32:56.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.300+0000 7fe6f7000700 1 -- 172.21.15.16:0/3996795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 msgr2=0x7fe6f0101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.300+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3996795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 0x7fe6f0101290 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009990 tx=0x7fe6e0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.300+0000 7fe6f7000700 1 -- 172.21.15.16:0/3996795110 shutdown_connections 2024-04-05T23:32:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.300+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3996795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 0x7fe6f0101290 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.300+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3996795110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6f0102a10 0x7fe6f0109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.300+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3996795110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6f0102050 0x7fe6f01024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.300+0000 7fe6f7000700 1 -- 172.21.15.16:0/3996795110 >> 172.21.15.16:0/3996795110 conn(0x7fe6f00fc430 msgr2=0x7fe6f00fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.301+0000 7fe6f7000700 1 -- 172.21.15.16:0/3996795110 shutdown_connections 2024-04-05T23:32:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.301+0000 7fe6f7000700 1 -- 172.21.15.16:0/3996795110 wait complete. 2024-04-05T23:32:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.301+0000 7fe6f7000700 1 Processor -- start 2024-04-05T23:32:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6f7000700 1 -- start start 2024-04-05T23:32:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6f7000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 0x7fe6f01990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6f7000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6f0102050 0x7fe6f0199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6f7000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6f0102a10 0x7fe6f01a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6f7000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f01a0bb0 con 0x7fe6f0100e70 2024-04-05T23:32:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6f7000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f01a0cf0 con 0x7fe6f0102050 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6f7000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f01a0e80 con 0x7fe6f0102a10 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6e7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 0x7fe6f01990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6efe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6f0102050 0x7fe6f0199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6e7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 0x7fe6f01990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55636/0 (socket says 172.21.15.16:55636) 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6e7e00700 1 -- 172.21.15.16:0/3280119633 learned_addr learned my addr 172.21.15.16:0/3280119633 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.302+0000 7fe6f5000700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6f0102a10 0x7fe6f01a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6e7e00700 1 -- 172.21.15.16:0/3280119633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6f0102a10 msgr2=0x7fe6f01a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6e7e00700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6f0102a10 0x7fe6f01a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6e7e00700 1 -- 172.21.15.16:0/3280119633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6f0102050 msgr2=0x7fe6f0199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6e7e00700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6f0102050 0x7fe6f0199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6e7e00700 1 -- 172.21.15.16:0/3280119633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e00095f0 con 0x7fe6f0100e70 2024-04-05T23:32:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6efe00700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6f0102050 0x7fe6f0199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:56.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6e7e00700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 0x7fe6f01990d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000b680 tx=0x7fe6e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:56.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6ed600700 1 -- 172.21.15.16:0/3280119633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e001d070 con 0x7fe6f0100e70 2024-04-05T23:32:56.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6ed600700 1 -- 172.21.15.16:0/3280119633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6e0021410 con 0x7fe6f0100e70 2024-04-05T23:32:56.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f01a1100 con 0x7fe6f0100e70 2024-04-05T23:32:56.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6ed600700 1 -- 172.21.15.16:0/3280119633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0017650 con 0x7fe6f0100e70 2024-04-05T23:32:56.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.303+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f01a1520 con 0x7fe6f0100e70 2024-04-05T23:32:56.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.304+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f0067c30 con 0x7fe6f0100e70 2024-04-05T23:32:56.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.306+0000 7fe6ed600700 1 -- 172.21.15.16:0/3280119633 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fe6e0021580 con 0x7fe6f0100e70 2024-04-05T23:32:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.307+0000 7fe6ed600700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe6d40a11b0 0x7fe6d40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.308+0000 7fe6ed600700 1 -- 172.21.15.16:0/3280119633 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe6e00ce4e0 con 0x7fe6f0100e70 2024-04-05T23:32:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.308+0000 7fe6efe00700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe6d40a11b0 0x7fe6d40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.308+0000 7fe6efe00700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe6d40a11b0 0x7fe6d40a3670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0100b90 tx=0x7fe6d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.310+0000 7fe6ed600700 1 -- 172.21.15.16:0/3280119633 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe6e0097bb0 con 0x7fe6f0100e70 2024-04-05T23:32:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.532+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe6f010c8c0 con 0x7fe6f0100e70 2024-04-05T23:32:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.533+0000 7fe6ed600700 1 -- 172.21.15.16:0/3280119633 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe6e009b1d0 con 0x7fe6f0100e70 2024-04-05T23:32:56.526 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-05T23:32:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe6d40a11b0 msgr2=0x7fe6d40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe6d40a11b0 0x7fe6d40a3670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0100b90 tx=0x7fe6d8009260 comp rx=0 tx=0).stop 2024-04-05T23:32:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 msgr2=0x7fe6f01990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 0x7fe6f01990d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000b680 tx=0x7fe6e0004730 comp rx=0 tx=0).stop 2024-04-05T23:32:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 shutdown_connections 2024-04-05T23:32:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe6d40a11b0 0x7fe6d40a3670 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6f0100e70 0x7fe6f01990d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6f0102a10 0x7fe6f01a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 --2- 172.21.15.16:0/3280119633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6f0102050 0x7fe6f0199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 >> 172.21.15.16:0/3280119633 conn(0x7fe6f00fc430 msgr2=0x7fe6f0073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 shutdown_connections 2024-04-05T23:32:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.535+0000 7fe6f7000700 1 -- 172.21.15.16:0/3280119633 wait complete. 2024-04-05T23:32:56.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-05T23:32:56.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:56 smithi027 ceph-mon[78597]: pgmap v260: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-05T23:32:56.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:56 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2455919975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:56.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:56 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3312563504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:32:56.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.756+0000 7f1944e00700 1 -- 172.21.15.16:0/3821836257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1940100e70 msgr2=0x7f1940101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.756+0000 7f1944e00700 1 --2- 172.21.15.16:0/3821836257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1940100e70 0x7f1940101290 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1930009990 tx=0x7f1930009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.756+0000 7f1944e00700 1 -- 172.21.15.16:0/3821836257 shutdown_connections 2024-04-05T23:32:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.756+0000 7f1944e00700 1 --2- 172.21.15.16:0/3821836257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1940102a10 0x7f1940109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.756+0000 7f1944e00700 1 --2- 172.21.15.16:0/3821836257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1940100e70 0x7f1940101290 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.756+0000 7f1944e00700 1 --2- 172.21.15.16:0/3821836257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1940102050 0x7f19401024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.756+0000 7f1944e00700 1 -- 172.21.15.16:0/3821836257 >> 172.21.15.16:0/3821836257 conn(0x7f19400fc430 msgr2=0x7f19400fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.756+0000 7f1944e00700 1 -- 172.21.15.16:0/3821836257 shutdown_connections 2024-04-05T23:32:56.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.756+0000 7f1944e00700 1 -- 172.21.15.16:0/3821836257 wait complete. 2024-04-05T23:32:56.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.757+0000 7f1944e00700 1 Processor -- start 2024-04-05T23:32:56.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.757+0000 7f1944e00700 1 -- start start 2024-04-05T23:32:56.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.757+0000 7f1944e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1940100e70 0x7f1940074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:56.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.757+0000 7f1944e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1940102050 0x7f19400732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:56.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.757+0000 7f1944e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1940102a10 0x7f1940073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:56.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.757+0000 7f1944e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1940073fb0 con 0x7f1940102050 2024-04-05T23:32:56.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.757+0000 7f1944e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1940074120 con 0x7f1940102a10 2024-04-05T23:32:56.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.757+0000 7f1944e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19401a0e90 con 0x7f1940100e70 2024-04-05T23:32:56.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1940102050 0x7f19400732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:56.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1940102050 0x7f19400732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55656/0 (socket says 172.21.15.16:55656) 2024-04-05T23:32:56.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193d400700 1 -- 172.21.15.16:0/76206184 learned_addr learned my addr 172.21.15.16:0/76206184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193e800700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1940102a10 0x7f1940073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193de00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1940100e70 0x7f1940074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193d400700 1 -- 172.21.15.16:0/76206184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1940100e70 msgr2=0x7f1940074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193d400700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1940100e70 0x7f1940074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193d400700 1 -- 172.21.15.16:0/76206184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1940102a10 msgr2=0x7f1940073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193d400700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1940102a10 0x7f1940073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193d400700 1 -- 172.21.15.16:0/76206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19300095f0 con 0x7f1940102050 2024-04-05T23:32:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193de00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1940100e70 0x7f1940074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:32:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.758+0000 7f193d400700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1940102050 0x7f19400732d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e970 tx=0x7f192c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.759+0000 7f1936a00700 1 -- 172.21.15.16:0/76206184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c009790 con 0x7f1940102050 2024-04-05T23:32:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.759+0000 7f1936a00700 1 -- 172.21.15.16:0/76206184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f192c00eec0 con 0x7f1940102050 2024-04-05T23:32:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.759+0000 7f1936a00700 1 -- 172.21.15.16:0/76206184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c003c90 con 0x7f1940102050 2024-04-05T23:32:56.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.759+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19401a1120 con 0x7f1940102050 2024-04-05T23:32:56.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.759+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19401a15e0 con 0x7f1940102050 2024-04-05T23:32:56.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.760+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19401083c0 con 0x7f1940102050 2024-04-05T23:32:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.762+0000 7f1936a00700 1 -- 172.21.15.16:0/76206184 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f192c00cc70 con 0x7f1940102050 2024-04-05T23:32:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.763+0000 7f1936a00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f19240a11b0 0x7f19240a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.763+0000 7f1936a00700 1 -- 172.21.15.16:0/76206184 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f192c014070 con 0x7f1940102050 2024-04-05T23:32:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.763+0000 7f193de00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f19240a11b0 0x7f19240a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.764+0000 7f193de00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f19240a11b0 0x7f19240a3670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1930006010 tx=0x7f193000b770 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.767+0000 7f1936a00700 1 -- 172.21.15.16:0/76206184 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f192c096300 con 0x7f1940102050 2024-04-05T23:32:56.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:56 smithi016 ceph-mon[38248]: pgmap v260: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-05T23:32:56.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:56 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2455919975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:56.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:56 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3312563504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:32:56.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:56 smithi130 ceph-mon[79417]: pgmap v260: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-05T23:32:56.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:56 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2455919975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:56.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:56 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3312563504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:32:56.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.983+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f194010c6e0 con 0x7f1940102050 2024-04-05T23:32:56.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.984+0000 7f1936a00700 1 -- 172.21.15.16:0/76206184 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f192c099920 con 0x7f1940102050 2024-04-05T23:32:56.977 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-05T23:32:56.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f19240a11b0 msgr2=0x7f19240a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f19240a11b0 0x7f19240a3670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1930006010 tx=0x7f193000b770 comp rx=0 tx=0).stop 2024-04-05T23:32:56.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1940102050 msgr2=0x7f19400732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:56.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1940102050 0x7f19400732d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e970 tx=0x7f192c00ec80 comp rx=0 tx=0).stop 2024-04-05T23:32:56.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 shutdown_connections 2024-04-05T23:32:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f19240a11b0 0x7f19240a3670 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1940102a10 0x7f1940073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1940100e70 0x7f1940074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 --2- 172.21.15.16:0/76206184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1940102050 0x7f19400732d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 >> 172.21.15.16:0/76206184 conn(0x7f19400fc430 msgr2=0x7f1940105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.986+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 shutdown_connections 2024-04-05T23:32:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:56.987+0000 7f1944e00700 1 -- 172.21.15.16:0/76206184 wait complete. 2024-04-05T23:32:56.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:32:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.205+0000 7f2d19800700 1 -- 172.21.15.16:0/2565424111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d14101fb0 msgr2=0x7f2d14102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.205+0000 7f2d19800700 1 --2- 172.21.15.16:0/2565424111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d14101fb0 0x7f2d14102410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009920 tx=0x7f2d00009c30 comp rx=0 tx=0).stop 2024-04-05T23:32:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.205+0000 7f2d19800700 1 -- 172.21.15.16:0/2565424111 shutdown_connections 2024-04-05T23:32:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.205+0000 7f2d19800700 1 --2- 172.21.15.16:0/2565424111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d14102950 0x7f2d14109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.205+0000 7f2d19800700 1 --2- 172.21.15.16:0/2565424111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d14100dc0 0x7f2d141011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.205+0000 7f2d19800700 1 --2- 172.21.15.16:0/2565424111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d14101fb0 0x7f2d14102410 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.205+0000 7f2d19800700 1 -- 172.21.15.16:0/2565424111 >> 172.21.15.16:0/2565424111 conn(0x7f2d140fc350 msgr2=0x7f2d140fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.205+0000 7f2d19800700 1 -- 172.21.15.16:0/2565424111 shutdown_connections 2024-04-05T23:32:57.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.205+0000 7f2d19800700 1 -- 172.21.15.16:0/2565424111 wait complete. 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.206+0000 7f2d19800700 1 Processor -- start 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.206+0000 7f2d19800700 1 -- start start 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.206+0000 7f2d19800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d14100dc0 0x7f2d14196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.206+0000 7f2d19800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d14101fb0 0x7f2d14197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.206+0000 7f2d19800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d14102950 0x7f2d1419e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.206+0000 7f2d19800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1419e8f0 con 0x7f2d14100dc0 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.206+0000 7f2d19800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1419ea30 con 0x7f2d14102950 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.206+0000 7f2d19800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1419ebc0 con 0x7f2d14101fb0 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d14100dc0 0x7f2d14196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d14100dc0 0x7f2d14196dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55674/0 (socket says 172.21.15.16:55674) 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12a00700 1 -- 172.21.15.16:0/4014455537 learned_addr learned my addr 172.21.15.16:0/4014455537 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d13400700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d14102950 0x7f2d1419e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:57.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12000700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d14101fb0 0x7f2d14197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12a00700 1 -- 172.21.15.16:0/4014455537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d14101fb0 msgr2=0x7f2d14197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12a00700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d14101fb0 0x7f2d14197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12a00700 1 -- 172.21.15.16:0/4014455537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d14102950 msgr2=0x7f2d1419e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12a00700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d14102950 0x7f2d1419e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12a00700 1 -- 172.21.15.16:0/4014455537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d00009520 con 0x7f2d14100dc0 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d12a00700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d14100dc0 0x7f2d14196dc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400e970 tx=0x7f2d0400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d0b400700 1 -- 172.21.15.16:0/4014455537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d04009790 con 0x7f2d14100dc0 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d0b400700 1 -- 172.21.15.16:0/4014455537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d0400eec0 con 0x7f2d14100dc0 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.207+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1419eea0 con 0x7f2d14100dc0 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.208+0000 7f2d0b400700 1 -- 172.21.15.16:0/4014455537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d04003cf0 con 0x7f2d14100dc0 2024-04-05T23:32:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.208+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1419f2d0 con 0x7f2d14100dc0 2024-04-05T23:32:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.209+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d141082e0 con 0x7f2d14100dc0 2024-04-05T23:32:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.210+0000 7f2d0b400700 1 -- 172.21.15.16:0/4014455537 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f2d0400ccd0 con 0x7f2d14100dc0 2024-04-05T23:32:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.212+0000 7f2d0b400700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2cf80a1230 0x7f2cf80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.212+0000 7f2d0b400700 1 -- 172.21.15.16:0/4014455537 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2d04014070 con 0x7f2d14100dc0 2024-04-05T23:32:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.212+0000 7f2d12000700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2cf80a1230 0x7f2cf80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.213+0000 7f2d12000700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2cf80a1230 0x7f2cf80a36f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000fb70 tx=0x7f2d00019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.216+0000 7f2d0b400700 1 -- 172.21.15.16:0/4014455537 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2d04096270 con 0x7f2d14100dc0 2024-04-05T23:32:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.488+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d1410c670 con 0x7f2d14100dc0 2024-04-05T23:32:57.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.488+0000 7f2d0b400700 1 -- 172.21.15.16:0/4014455537 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10902 (secure 0 0 0) 0x7f2d04099890 con 0x7f2d14100dc0 2024-04-05T23:32:57.481 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:57.482 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:32:11.436182+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:32:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2cf80a1230 msgr2=0x7f2cf80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2cf80a1230 0x7f2cf80a36f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000fb70 tx=0x7f2d00019040 comp rx=0 tx=0).stop 2024-04-05T23:32:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d14100dc0 msgr2=0x7f2d14196dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d14100dc0 0x7f2d14196dc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400e970 tx=0x7f2d0400ec80 comp rx=0 tx=0).stop 2024-04-05T23:32:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 shutdown_connections 2024-04-05T23:32:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2cf80a1230 0x7f2cf80a36f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d14100dc0 0x7f2d14196dc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d14102950 0x7f2d1419e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 --2- 172.21.15.16:0/4014455537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d14101fb0 0x7f2d14197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 >> 172.21.15.16:0/4014455537 conn(0x7f2d140fc350 msgr2=0x7f2d14105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 shutdown_connections 2024-04-05T23:32:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.491+0000 7f2d19800700 1 -- 172.21.15.16:0/4014455537 wait complete. 2024-04-05T23:32:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-04-05T23:32:57.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:32:57.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.708+0000 7fb414000700 1 -- 172.21.15.16:0/332774347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c100db0 msgr2=0x7fb40c1011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.708+0000 7fb414000700 1 --2- 172.21.15.16:0/332774347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c100db0 0x7fb40c1011d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb4000099c0 tx=0x7fb400009cd0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.708+0000 7fb414000700 1 -- 172.21.15.16:0/332774347 shutdown_connections 2024-04-05T23:32:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.708+0000 7fb414000700 1 --2- 172.21.15.16:0/332774347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c100db0 0x7fb40c1011d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.708+0000 7fb414000700 1 --2- 172.21.15.16:0/332774347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb40c102940 0x7fb40c1091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.708+0000 7fb414000700 1 --2- 172.21.15.16:0/332774347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c101fa0 0x7fb40c102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.708+0000 7fb414000700 1 -- 172.21.15.16:0/332774347 >> 172.21.15.16:0/332774347 conn(0x7fb40c0fc320 msgr2=0x7fb40c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.708+0000 7fb414000700 1 -- 172.21.15.16:0/332774347 shutdown_connections 2024-04-05T23:32:57.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.708+0000 7fb414000700 1 -- 172.21.15.16:0/332774347 wait complete. 2024-04-05T23:32:57.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.709+0000 7fb414000700 1 Processor -- start 2024-04-05T23:32:57.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.709+0000 7fb414000700 1 -- start start 2024-04-05T23:32:57.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb414000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c100db0 0x7fb40c196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:57.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb414000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb40c101fa0 0x7fb40c1972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:57.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb414000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c102940 0x7fb40c19e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:57.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb414000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c19e8c0 con 0x7fb40c102940 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb414000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c19ea00 con 0x7fb40c100db0 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb414000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c19eb90 con 0x7fb40c101fa0 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb412000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c102940 0x7fb40c19e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb412000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c102940 0x7fb40c19e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55688/0 (socket says 172.21.15.16:55688) 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb412000700 1 -- 172.21.15.16:0/2306584826 learned_addr learned my addr 172.21.15.16:0/2306584826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb410c00700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb40c101fa0 0x7fb40c1972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb411600700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c100db0 0x7fb40c196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb412000700 1 -- 172.21.15.16:0/2306584826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb40c101fa0 msgr2=0x7fb40c1972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb412000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb40c101fa0 0x7fb40c1972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb412000700 1 -- 172.21.15.16:0/2306584826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c100db0 msgr2=0x7fb40c196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb412000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c100db0 0x7fb40c196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.710+0000 7fb412000700 1 -- 172.21.15.16:0/2306584826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8009520 con 0x7fb40c102940 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.711+0000 7fb412000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c102940 0x7fb40c19e2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb40400ef30 tx=0x7fb40400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.711+0000 7fb40a000700 1 -- 172.21.15.16:0/2306584826 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb404018070 con 0x7fb40c102940 2024-04-05T23:32:57.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.711+0000 7fb40a000700 1 -- 172.21.15.16:0/2306584826 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb40400f040 con 0x7fb40c102940 2024-04-05T23:32:57.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.711+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4000095f0 con 0x7fb40c102940 2024-04-05T23:32:57.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.711+0000 7fb40a000700 1 -- 172.21.15.16:0/2306584826 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40400cd00 con 0x7fb40c102940 2024-04-05T23:32:57.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.711+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c19f190 con 0x7fb40c102940 2024-04-05T23:32:57.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.712+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c108280 con 0x7fb40c102940 2024-04-05T23:32:57.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.714+0000 7fb40a000700 1 -- 172.21.15.16:0/2306584826 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fb404003a00 con 0x7fb40c102940 2024-04-05T23:32:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.715+0000 7fb40a000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb3f40a1230 0x7fb3f40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:57.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.716+0000 7fb40a000700 1 -- 172.21.15.16:0/2306584826 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb4040cc540 con 0x7fb40c102940 2024-04-05T23:32:57.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.716+0000 7fb411600700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb3f40a1230 0x7fb3f40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:57.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.716+0000 7fb411600700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb3f40a1230 0x7fb3f40a36f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb400009990 tx=0x7fb400000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:57.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.719+0000 7fb40a000700 1 -- 172.21.15.16:0/2306584826 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb404095bb0 con 0x7fb40c102940 2024-04-05T23:32:57.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:57 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3280119633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:32:57.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:57 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/76206184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:32:57.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:57 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3280119633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:32:57.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:57 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/76206184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:32:57.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:57 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3280119633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:32:57.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:57 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/76206184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:32:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.943+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb40c067b50 con 0x7fb40c102940 2024-04-05T23:32:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.945+0000 7fb40a000700 1 -- 172.21.15.16:0/2306584826 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22519 (secure 0 0 0) 0x7fb4040991d0 con 0x7fb40c102940 2024-04-05T23:32:57.938 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:57.938 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:38.369016+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:32:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb3f40a1230 msgr2=0x7fb3f40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb3f40a1230 0x7fb3f40a36f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb400009990 tx=0x7fb400000bc0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c102940 msgr2=0x7fb40c19e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c102940 0x7fb40c19e2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb40400ef30 tx=0x7fb40400c3a0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 shutdown_connections 2024-04-05T23:32:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb3f40a1230 0x7fb3f40a36f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40c102940 0x7fb40c19e2f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c100db0 0x7fb40c196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 --2- 172.21.15.16:0/2306584826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb40c101fa0 0x7fb40c1972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.947+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 >> 172.21.15.16:0/2306584826 conn(0x7fb40c0fc320 msgr2=0x7fb40c1053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.948+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 shutdown_connections 2024-04-05T23:32:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:57.948+0000 7fb414000700 1 -- 172.21.15.16:0/2306584826 wait complete. 2024-04-05T23:32:57.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-05T23:32:02.641673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-05T23:32:03.972521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:32:57.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:32:58.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.169+0000 7f5b5d200700 1 -- 172.21.15.16:0/58583054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58102a10 msgr2=0x7f5b58109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.169+0000 7f5b5d200700 1 --2- 172.21.15.16:0/58583054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58102a10 0x7f5b58109330 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000b0a0 tx=0x7f5b5000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.169+0000 7f5b5d200700 1 -- 172.21.15.16:0/58583054 shutdown_connections 2024-04-05T23:32:58.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.169+0000 7f5b5d200700 1 --2- 172.21.15.16:0/58583054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58102a10 0x7f5b58109330 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.169+0000 7f5b5d200700 1 --2- 172.21.15.16:0/58583054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b58100e70 0x7f5b58101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.169+0000 7f5b5d200700 1 --2- 172.21.15.16:0/58583054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b58102050 0x7f5b581024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.169+0000 7f5b5d200700 1 -- 172.21.15.16:0/58583054 >> 172.21.15.16:0/58583054 conn(0x7f5b580fc430 msgr2=0x7f5b580fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:58.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.169+0000 7f5b5d200700 1 -- 172.21.15.16:0/58583054 shutdown_connections 2024-04-05T23:32:58.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.169+0000 7f5b5d200700 1 -- 172.21.15.16:0/58583054 wait complete. 2024-04-05T23:32:58.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.170+0000 7f5b5d200700 1 Processor -- start 2024-04-05T23:32:58.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b5d200700 1 -- start start 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b5d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b58100e70 0x7f5b58196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b5d200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58102050 0x7f5b58197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b5d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b58102a10 0x7f5b5819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b5d200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5819ea30 con 0x7f5b58102050 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b5d200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5819eb70 con 0x7f5b58100e70 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b5d200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5819ed00 con 0x7f5b58102a10 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b55e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b58100e70 0x7f5b58196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b55e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b58100e70 0x7f5b58196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:55458/0 (socket says 172.21.15.16:55458) 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b55400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58102050 0x7f5b58197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b55e00700 1 -- 172.21.15.16:0/1272150272 learned_addr learned my addr 172.21.15.16:0/1272150272 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.171+0000 7f5b56800700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b58102a10 0x7f5b5819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b55400700 1 -- 172.21.15.16:0/1272150272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b58102a10 msgr2=0x7f5b5819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b55400700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b58102a10 0x7f5b5819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b55400700 1 -- 172.21.15.16:0/1272150272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b58100e70 msgr2=0x7f5b58196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b55400700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b58100e70 0x7f5b58196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b55400700 1 -- 172.21.15.16:0/1272150272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b48009520 con 0x7f5b58102050 2024-04-05T23:32:58.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b55e00700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b58100e70 0x7f5b58196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:58.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b55400700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58102050 0x7f5b58197440 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400e9c0 tx=0x7f5b4400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:58.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b4ea00700 1 -- 172.21.15.16:0/1272150272 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44004220 con 0x7f5b58102050 2024-04-05T23:32:58.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b50009d00 con 0x7f5b58102050 2024-04-05T23:32:58.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b4ea00700 1 -- 172.21.15.16:0/1272150272 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b44004380 con 0x7f5b58102050 2024-04-05T23:32:58.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b4ea00700 1 -- 172.21.15.16:0/1272150272 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44003be0 con 0x7f5b58102050 2024-04-05T23:32:58.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.172+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5819f2d0 con 0x7f5b58102050 2024-04-05T23:32:58.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.174+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b581082a0 con 0x7f5b58102050 2024-04-05T23:32:58.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.175+0000 7f5b4ea00700 1 -- 172.21.15.16:0/1272150272 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f5b44003d40 con 0x7f5b58102050 2024-04-05T23:32:58.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.177+0000 7f5b4ea00700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b3c0a11b0 0x7f5b3c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:58.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.177+0000 7f5b4ea00700 1 -- 172.21.15.16:0/1272150272 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5b44014070 con 0x7f5b58102050 2024-04-05T23:32:58.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.177+0000 7f5b55e00700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b3c0a11b0 0x7f5b3c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:58.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.177+0000 7f5b55e00700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b3c0a11b0 0x7f5b3c0a3670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009fd0 tx=0x7f5b48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.180+0000 7f5b4ea00700 1 -- 172.21.15.16:0/1272150272 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5b44096320 con 0x7f5b58102050 2024-04-05T23:32:58.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.393+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b5810c9e0 con 0x7f5b58102050 2024-04-05T23:32:58.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.394+0000 7f5b4ea00700 1 -- 172.21.15.16:0/1272150272 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b44099940 con 0x7f5b58102050 2024-04-05T23:32:58.387 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:32:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b3c0a11b0 msgr2=0x7f5b3c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b3c0a11b0 0x7f5b3c0a3670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009fd0 tx=0x7f5b48009260 comp rx=0 tx=0).stop 2024-04-05T23:32:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58102050 msgr2=0x7f5b58197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58102050 0x7f5b58197440 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400e9c0 tx=0x7f5b4400ecd0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 shutdown_connections 2024-04-05T23:32:58.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f5b3c0a11b0 0x7f5b3c0a3670 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b58102a10 0x7f5b5819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b58100e70 0x7f5b58196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 --2- 172.21.15.16:0/1272150272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b58102050 0x7f5b58197440 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 >> 172.21.15.16:0/1272150272 conn(0x7f5b580fc430 msgr2=0x7f5b581053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 shutdown_connections 2024-04-05T23:32:58.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.396+0000 7f5b5d200700 1 -- 172.21.15.16:0/1272150272 wait complete. 2024-04-05T23:32:58.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-05T23:32:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.617+0000 7feddfa00700 1 -- 172.21.15.16:0/1789136601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd8104270 msgr2=0x7fedd8106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.617+0000 7feddfa00700 1 --2- 172.21.15.16:0/1789136601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd8104270 0x7fedd8106660 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009990 tx=0x7fedc4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.617+0000 7feddfa00700 1 -- 172.21.15.16:0/1789136601 shutdown_connections 2024-04-05T23:32:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.617+0000 7feddfa00700 1 --2- 172.21.15.16:0/1789136601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedd8106ba0 0x7fedd8109010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.617+0000 7feddfa00700 1 --2- 172.21.15.16:0/1789136601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedd8101940 0x7fedd8103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.617+0000 7feddfa00700 1 --2- 172.21.15.16:0/1789136601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd8104270 0x7fedd8106660 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.617+0000 7feddfa00700 1 -- 172.21.15.16:0/1789136601 >> 172.21.15.16:0/1789136601 conn(0x7fedd80fb330 msgr2=0x7fedd80fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.617+0000 7feddfa00700 1 -- 172.21.15.16:0/1789136601 shutdown_connections 2024-04-05T23:32:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.617+0000 7feddfa00700 1 -- 172.21.15.16:0/1789136601 wait complete. 2024-04-05T23:32:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.618+0000 7feddfa00700 1 Processor -- start 2024-04-05T23:32:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddfa00700 1 -- start start 2024-04-05T23:32:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddfa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedd8101940 0x7fedd8196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddfa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedd8104270 0x7fedd8197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddfa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd8106ba0 0x7fedd819e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddfa00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd819e960 con 0x7fedd8106ba0 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddfa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd819eaa0 con 0x7fedd8101940 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddfa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd819ec30 con 0x7fedd8104270 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddd000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedd8101940 0x7fedd8196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddd000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedd8101940 0x7fedd8196e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:55474/0 (socket says 172.21.15.16:55474) 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddd000700 1 -- 172.21.15.16:0/630240034 learned_addr learned my addr 172.21.15.16:0/630240034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7feddda00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd8106ba0 0x7fedd819e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.619+0000 7fedd7e00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedd8104270 0x7fedd8197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7feddd000700 1 -- 172.21.15.16:0/630240034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedd8104270 msgr2=0x7fedd8197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7feddd000700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedd8104270 0x7fedd8197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7feddd000700 1 -- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd8106ba0 msgr2=0x7fedd819e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7feddd000700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd8106ba0 0x7fedd819e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7feddd000700 1 -- 172.21.15.16:0/630240034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc40095f0 con 0x7fedd8101940 2024-04-05T23:32:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7feddd000700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedd8101940 0x7fedd8196e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00e9a0 tx=0x7fedcc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7fedd5600700 1 -- 172.21.15.16:0/630240034 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc009790 con 0x7fedd8101940 2024-04-05T23:32:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7fedd5600700 1 -- 172.21.15.16:0/630240034 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedcc010430 con 0x7fedd8101940 2024-04-05T23:32:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd819ef10 con 0x7fedd8101940 2024-04-05T23:32:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7fedd5600700 1 -- 172.21.15.16:0/630240034 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc018680 con 0x7fedd8101940 2024-04-05T23:32:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.620+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd819f330 con 0x7fedd8101940 2024-04-05T23:32:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.621+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd80fceb0 con 0x7fedd8101940 2024-04-05T23:32:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.623+0000 7fedd5600700 1 -- 172.21.15.16:0/630240034 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fedcc0105a0 con 0x7fedd8101940 2024-04-05T23:32:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.624+0000 7fedd5600700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedc00a11b0 0x7fedc00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.625+0000 7fedd5600700 1 -- 172.21.15.16:0/630240034 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fedcc014070 con 0x7fedd8101940 2024-04-05T23:32:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.625+0000 7fedd7e00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedc00a11b0 0x7fedc00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.625+0000 7fedd7e00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedc00a11b0 0x7fedc00a3670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fedc400b2e0 tx=0x7fedc4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.627+0000 7fedd5600700 1 -- 172.21.15.16:0/630240034 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fedcc096440 con 0x7fedd8101940 2024-04-05T23:32:58.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:58 smithi027 ceph-mon[78597]: pgmap v261: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-05T23:32:58.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:58 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4014455537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:32:58.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:58 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2306584826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:58.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:58 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1272150272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:32:58.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:58 smithi016 ceph-mon[38248]: pgmap v261: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-05T23:32:58.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:58 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4014455537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:32:58.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:58 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2306584826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:58.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:58 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1272150272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:32:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.841+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fedd810c6b0 con 0x7fedd8101940 2024-04-05T23:32:58.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:58 smithi130 ceph-mon[79417]: pgmap v261: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-05T23:32:58.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:58 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4014455537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:32:58.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:58 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2306584826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:32:58.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:58 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1272150272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:32:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.841+0000 7fedd5600700 1 -- 172.21.15.16:0/630240034 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7fedcc099a60 con 0x7fedd8101940 2024-04-05T23:32:58.836 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-05T23:32:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedc00a11b0 msgr2=0x7fedc00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedc00a11b0 0x7fedc00a3670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fedc400b2e0 tx=0x7fedc4005fd0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedd8101940 msgr2=0x7fedd8196e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedd8101940 0x7fedd8196e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00e9a0 tx=0x7fedcc00ecb0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 shutdown_connections 2024-04-05T23:32:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedc00a11b0 0x7fedc00a3670 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd8106ba0 0x7fedd819e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedd8101940 0x7fedd8196e30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 --2- 172.21.15.16:0/630240034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedd8104270 0x7fedd8197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 >> 172.21.15.16:0/630240034 conn(0x7fedd80fb330 msgr2=0x7fedd8103830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 shutdown_connections 2024-04-05T23:32:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:58.844+0000 7feddfa00700 1 -- 172.21.15.16:0/630240034 wait complete. 2024-04-05T23:32:58.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-05T23:32:59.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.065+0000 7f1d6ea00700 1 -- 172.21.15.16:0/3481299843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 msgr2=0x7f1d68101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.065+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/3481299843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 0x7f1d68101280 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009990 tx=0x7f1d58009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.065+0000 7f1d6ea00700 1 -- 172.21.15.16:0/3481299843 shutdown_connections 2024-04-05T23:32:59.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.065+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/3481299843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d68102a00 0x7f1d68109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.065+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/3481299843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 0x7f1d68101280 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.065+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/3481299843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d68102040 0x7f1d681024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.065+0000 7f1d6ea00700 1 -- 172.21.15.16:0/3481299843 >> 172.21.15.16:0/3481299843 conn(0x7f1d680fc440 msgr2=0x7f1d680fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.065+0000 7f1d6ea00700 1 -- 172.21.15.16:0/3481299843 shutdown_connections 2024-04-05T23:32:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.065+0000 7f1d6ea00700 1 -- 172.21.15.16:0/3481299843 wait complete. 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.066+0000 7f1d6ea00700 1 Processor -- start 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.066+0000 7f1d6ea00700 1 -- start start 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d6ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 0x7f1d68199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d6ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d68102040 0x7f1d681995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d6ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d68102a00 0x7f1d681a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d6ea00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d681a0bc0 con 0x7f1d68102a00 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d6ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d681a0d00 con 0x7f1d68102040 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d6ea00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d681a0e90 con 0x7f1d68100e60 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d5fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 0x7f1d68199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d5fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 0x7f1d68199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42336/0 (socket says 172.21.15.16:42336) 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d5fe00700 1 -- 172.21.15.16:0/2707370485 learned_addr learned my addr 172.21.15.16:0/2707370485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d6ca00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d68102a00 0x7f1d681a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d67e00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d68102040 0x7f1d681995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d5fe00700 1 -- 172.21.15.16:0/2707370485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d68102040 msgr2=0x7f1d681995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d5fe00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d68102040 0x7f1d681995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d5fe00700 1 -- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d68102a00 msgr2=0x7f1d681a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d5fe00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d68102a00 0x7f1d681a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.067+0000 7f1d5fe00700 1 -- 172.21.15.16:0/2707370485 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d580095f0 con 0x7f1d68100e60 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.068+0000 7f1d5fe00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 0x7f1d68199090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800b680 tx=0x7f1d58004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.068+0000 7f1d65600700 1 -- 172.21.15.16:0/2707370485 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5801d070 con 0x7f1d68100e60 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.068+0000 7f1d65600700 1 -- 172.21.15.16:0/2707370485 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d58021410 con 0x7f1d68100e60 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.068+0000 7f1d65600700 1 -- 172.21.15.16:0/2707370485 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58017650 con 0x7f1d68100e60 2024-04-05T23:32:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.068+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d681a1110 con 0x7f1d68100e60 2024-04-05T23:32:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.068+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d681a14e0 con 0x7f1d68100e60 2024-04-05T23:32:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.069+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d68067c30 con 0x7f1d68100e60 2024-04-05T23:32:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.071+0000 7f1d65600700 1 -- 172.21.15.16:0/2707370485 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f1d58021580 con 0x7f1d68100e60 2024-04-05T23:32:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.072+0000 7f1d65600700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1d4c0a11b0 0x7f1d4c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.073+0000 7f1d65600700 1 -- 172.21.15.16:0/2707370485 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1d580ce590 con 0x7f1d68100e60 2024-04-05T23:32:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.073+0000 7f1d67e00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1d4c0a11b0 0x7f1d4c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:59.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.073+0000 7f1d67e00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1d4c0a11b0 0x7f1d4c0a3670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d68100b80 tx=0x7f1d50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:59.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.075+0000 7f1d65600700 1 -- 172.21.15.16:0/2707370485 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1d58097c00 con 0x7f1d68100e60 2024-04-05T23:32:59.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.292+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1d6810c8b0 con 0x7f1d68100e60 2024-04-05T23:32:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.292+0000 7f1d65600700 1 -- 172.21.15.16:0/2707370485 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1d5809b220 con 0x7f1d68100e60 2024-04-05T23:32:59.285 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-05T23:32:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1d4c0a11b0 msgr2=0x7f1d4c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1d4c0a11b0 0x7f1d4c0a3670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d68100b80 tx=0x7f1d50009260 comp rx=0 tx=0).stop 2024-04-05T23:32:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 msgr2=0x7f1d68199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 0x7f1d68199090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800b680 tx=0x7f1d58004730 comp rx=0 tx=0).stop 2024-04-05T23:32:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 shutdown_connections 2024-04-05T23:32:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f1d4c0a11b0 0x7f1d4c0a3670 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d68102a00 0x7f1d681a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d68100e60 0x7f1d68199090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 --2- 172.21.15.16:0/2707370485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d68102040 0x7f1d681995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 >> 172.21.15.16:0/2707370485 conn(0x7f1d680fc440 msgr2=0x7f1d68073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:59.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 shutdown_connections 2024-04-05T23:32:59.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.295+0000 7f1d6ea00700 1 -- 172.21.15.16:0/2707370485 wait complete. 2024-04-05T23:32:59.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:32:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.514+0000 7f330b200700 1 -- 172.21.15.16:0/2542404898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3304101840 msgr2=0x7f3304103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.514+0000 7f330b200700 1 --2- 172.21.15.16:0/2542404898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3304101840 0x7f3304103c30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009990 tx=0x7f32f4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.515+0000 7f330b200700 1 -- 172.21.15.16:0/2542404898 shutdown_connections 2024-04-05T23:32:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.515+0000 7f330b200700 1 --2- 172.21.15.16:0/2542404898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3304106aa0 0x7f3304108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.515+0000 7f330b200700 1 --2- 172.21.15.16:0/2542404898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3304101840 0x7f3304103c30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.515+0000 7f330b200700 1 --2- 172.21.15.16:0/2542404898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3304104170 0x7f3304106560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.515+0000 7f330b200700 1 -- 172.21.15.16:0/2542404898 >> 172.21.15.16:0/2542404898 conn(0x7f33040fb230 msgr2=0x7f33040fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:59.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.515+0000 7f330b200700 1 -- 172.21.15.16:0/2542404898 shutdown_connections 2024-04-05T23:32:59.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.515+0000 7f330b200700 1 -- 172.21.15.16:0/2542404898 wait complete. 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f330b200700 1 Processor -- start 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f330b200700 1 -- start start 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f330b200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3304101840 0x7f3304101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f330b200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3304104170 0x7f33040ff7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f330b200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3304106aa0 0x7f33040ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f330b200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33041004d0 con 0x7f3304106aa0 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f330b200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3304100640 con 0x7f3304101840 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f330b200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33041a0da0 con 0x7f3304104170 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f3309200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3304106aa0 0x7f33040ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f3309200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3304106aa0 0x7f33040ffd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55746/0 (socket says 172.21.15.16:55746) 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f3309200700 1 -- 172.21.15.16:0/1127527876 learned_addr learned my addr 172.21.15.16:0/1127527876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.516+0000 7f3303e00700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3304101840 0x7f3304101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3303400700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3304104170 0x7f33040ff7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3309200700 1 -- 172.21.15.16:0/1127527876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3304104170 msgr2=0x7f33040ff7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3309200700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3304104170 0x7f33040ff7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3309200700 1 -- 172.21.15.16:0/1127527876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3304101840 msgr2=0x7f3304101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3309200700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3304101840 0x7f3304101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3309200700 1 -- 172.21.15.16:0/1127527876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32fc009c40 con 0x7f3304106aa0 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3303400700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3304104170 0x7f33040ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3309200700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3304106aa0 0x7f33040ffd30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00c370 tx=0x7f32fc00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3300c00700 1 -- 172.21.15.16:0/1127527876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc017070 con 0x7f3304106aa0 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3300c00700 1 -- 172.21.15.16:0/1127527876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32fc00f040 con 0x7f3304106aa0 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f3300c00700 1 -- 172.21.15.16:0/1127527876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc00e050 con 0x7f3304106aa0 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.517+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f40095f0 con 0x7f3304106aa0 2024-04-05T23:32:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.518+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33041a1210 con 0x7f3304106aa0 2024-04-05T23:32:59.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.519+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33040fcdb0 con 0x7f3304106aa0 2024-04-05T23:32:59.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.520+0000 7f3300c00700 1 -- 172.21.15.16:0/1127527876 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f32fc00cb00 con 0x7f3304106aa0 2024-04-05T23:32:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.522+0000 7f3300c00700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f32e80a11b0 0x7f32e80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:32:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.522+0000 7f3300c00700 1 -- 172.21.15.16:0/1127527876 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f32fc0cc3e0 con 0x7f3304106aa0 2024-04-05T23:32:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.522+0000 7f3303e00700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f32e80a11b0 0x7f32e80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:32:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.522+0000 7f3303e00700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f32e80a11b0 0x7f32e80a3670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f32f400b2e0 tx=0x7f32f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:32:59.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.526+0000 7f3300c00700 1 -- 172.21.15.16:0/1127527876 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f32fc0959c0 con 0x7f3304106aa0 2024-04-05T23:32:59.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:59 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/630240034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:32:59.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:32:59 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2707370485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:32:59.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:59 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/630240034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:32:59.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:32:59 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2707370485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:32:59.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.786+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3304067310 con 0x7f3304106aa0 2024-04-05T23:32:59.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.786+0000 7f3300c00700 1 -- 172.21.15.16:0/1127527876 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10902 (secure 0 0 0) 0x7f32fc098fe0 con 0x7f3304106aa0 2024-04-05T23:32:59.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:32:59.779 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:32:11.436182+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f32e80a11b0 msgr2=0x7f32e80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f32e80a11b0 0x7f32e80a3670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f32f400b2e0 tx=0x7f32f4005fd0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3304106aa0 msgr2=0x7f33040ffd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3304106aa0 0x7f33040ffd30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00c370 tx=0x7f32fc00c680 comp rx=0 tx=0).stop 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 shutdown_connections 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f32e80a11b0 0x7f32e80a3670 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3304106aa0 0x7f33040ffd30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3304101840 0x7f3304101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 --2- 172.21.15.16:0/1127527876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3304104170 0x7f33040ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 >> 172.21.15.16:0/1127527876 conn(0x7f33040fb230 msgr2=0x7f3304102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 shutdown_connections 2024-04-05T23:32:59.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:32:59.789+0000 7f330b200700 1 -- 172.21.15.16:0/1127527876 wait complete. 2024-04-05T23:32:59.783 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-04-05T23:32:59.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:32:59.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:59 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/630240034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:32:59.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:32:59 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2707370485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:32:59.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.006+0000 7fa1baa00700 1 -- 172.21.15.16:0/1437672470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b4101fa0 msgr2=0x7fa1b4102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:32:59.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.006+0000 7fa1baa00700 1 --2- 172.21.15.16:0/1437672470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b4101fa0 0x7fa1b4102400 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00099c0 tx=0x7fa1a0009cd0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.006+0000 7fa1baa00700 1 -- 172.21.15.16:0/1437672470 shutdown_connections 2024-04-05T23:33:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.006+0000 7fa1baa00700 1 --2- 172.21.15.16:0/1437672470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1b4102940 0x7fa1b41091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.006+0000 7fa1baa00700 1 --2- 172.21.15.16:0/1437672470 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b4100db0 0x7fa1b41011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.006+0000 7fa1baa00700 1 --2- 172.21.15.16:0/1437672470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b4101fa0 0x7fa1b4102400 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.006+0000 7fa1baa00700 1 -- 172.21.15.16:0/1437672470 >> 172.21.15.16:0/1437672470 conn(0x7fa1b40fc320 msgr2=0x7fa1b40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.006+0000 7fa1baa00700 1 -- 172.21.15.16:0/1437672470 shutdown_connections 2024-04-05T23:33:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.007+0000 7fa1baa00700 1 -- 172.21.15.16:0/1437672470 wait complete. 2024-04-05T23:33:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.008+0000 7fa1baa00700 1 Processor -- start 2024-04-05T23:33:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.008+0000 7fa1baa00700 1 -- start start 2024-04-05T23:33:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1baa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b4100db0 0x7fa1b4074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1baa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1b4101fa0 0x7fa1b40732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1baa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b4102940 0x7fa1b4073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1baa00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b4073fb0 con 0x7fa1b4102940 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1baa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b4074120 con 0x7fa1b4101fa0 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1baa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b41a0d50 con 0x7fa1b4100db0 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1b4101fa0 0x7fa1b40732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b3e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b4100db0 0x7fa1b4074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1b4101fa0 0x7fa1b40732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:55528/0 (socket says 172.21.15.16:55528) 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b3400700 1 -- 172.21.15.16:0/3470423745 learned_addr learned my addr 172.21.15.16:0/3470423745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b8a00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b4102940 0x7fa1b4073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b3e00700 1 -- 172.21.15.16:0/3470423745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1b4101fa0 msgr2=0x7fa1b40732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b3e00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1b4101fa0 0x7fa1b40732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b3e00700 1 -- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b4102940 msgr2=0x7fa1b4073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b3e00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b4102940 0x7fa1b4073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.009+0000 7fa1b3e00700 1 -- 172.21.15.16:0/3470423745 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a00095f0 con 0x7fa1b4100db0 2024-04-05T23:33:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.010+0000 7fa1b3e00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b4100db0 0x7fa1b4074c40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400d740 tx=0x7fa1a400da50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.010+0000 7fa1b0c00700 1 -- 172.21.15.16:0/3470423745 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4004220 con 0x7fa1b4100db0 2024-04-05T23:33:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.010+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b41a0f80 con 0x7fa1b4100db0 2024-04-05T23:33:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.010+0000 7fa1b0c00700 1 -- 172.21.15.16:0/3470423745 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1a4004380 con 0x7fa1b4100db0 2024-04-05T23:33:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.010+0000 7fa1b0c00700 1 -- 172.21.15.16:0/3470423745 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4003cf0 con 0x7fa1b4100db0 2024-04-05T23:33:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.010+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b41a1350 con 0x7fa1b4100db0 2024-04-05T23:33:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.011+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b4108160 con 0x7fa1b4100db0 2024-04-05T23:33:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.013+0000 7fa1b0c00700 1 -- 172.21.15.16:0/3470423745 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fa1a4004870 con 0x7fa1b4100db0 2024-04-05T23:33:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.015+0000 7fa1b0c00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1980a11b0 0x7fa1980a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.015+0000 7fa1b0c00700 1 -- 172.21.15.16:0/3470423745 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa1a40cc5c0 con 0x7fa1b4100db0 2024-04-05T23:33:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.015+0000 7fa1b3400700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1980a11b0 0x7fa1980a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.015+0000 7fa1b3400700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1980a11b0 0x7fa1980a3670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0006010 tx=0x7fa1a00036f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.017+0000 7fa1b0c00700 1 -- 172.21.15.16:0/3470423745 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa1a4095ba0 con 0x7fa1b4100db0 2024-04-05T23:33:00.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.238+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1b410c7a0 con 0x7fa1b4100db0 2024-04-05T23:33:00.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.241+0000 7fa1b0c00700 1 -- 172.21.15.16:0/3470423745 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22519 (secure 0 0 0) 0x7fa1a401f090 con 0x7fa1b4100db0 2024-04-05T23:33:00.233 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:00.233 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:38.369016+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:33:00.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1980a11b0 msgr2=0x7fa1980a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1980a11b0 0x7fa1980a3670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0006010 tx=0x7fa1a00036f0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b4100db0 msgr2=0x7fa1b4074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b4100db0 0x7fa1b4074c40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400d740 tx=0x7fa1a400da50 comp rx=0 tx=0).stop 2024-04-05T23:33:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 shutdown_connections 2024-04-05T23:33:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa1980a11b0 0x7fa1980a3670 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1b4102940 0x7fa1b4073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b4100db0 0x7fa1b4074c40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 --2- 172.21.15.16:0/3470423745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1b4101fa0 0x7fa1b40732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.242+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 >> 172.21.15.16:0/3470423745 conn(0x7fa1b40fc320 msgr2=0x7fa1b41052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.243+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 shutdown_connections 2024-04-05T23:33:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.243+0000 7fa1baa00700 1 -- 172.21.15.16:0/3470423745 wait complete. 2024-04-05T23:33:00.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:33:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.459+0000 7f51e4000700 1 -- 172.21.15.16:0/1577177127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc102950 msgr2=0x7f51dc109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.459+0000 7f51e4000700 1 --2- 172.21.15.16:0/1577177127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc102950 0x7f51dc109200 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f51d400b0d0 tx=0x7f51d400b3e0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.459+0000 7f51e4000700 1 -- 172.21.15.16:0/1577177127 shutdown_connections 2024-04-05T23:33:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.459+0000 7f51e4000700 1 --2- 172.21.15.16:0/1577177127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc102950 0x7f51dc109200 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.459+0000 7f51e4000700 1 --2- 172.21.15.16:0/1577177127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc100dc0 0x7f51dc1011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.459+0000 7f51e4000700 1 --2- 172.21.15.16:0/1577177127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51dc101fb0 0x7f51dc102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.459+0000 7f51e4000700 1 -- 172.21.15.16:0/1577177127 >> 172.21.15.16:0/1577177127 conn(0x7f51dc0fc350 msgr2=0x7f51dc0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.459+0000 7f51e4000700 1 -- 172.21.15.16:0/1577177127 shutdown_connections 2024-04-05T23:33:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.459+0000 7f51e4000700 1 -- 172.21.15.16:0/1577177127 wait complete. 2024-04-05T23:33:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.460+0000 7f51e4000700 1 Processor -- start 2024-04-05T23:33:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e4000700 1 -- start start 2024-04-05T23:33:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e4000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100dc0 0x7f51dc074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e4000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51dc101fb0 0x7f51dc0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e4000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc102950 0x7f51dc073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e4000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc073fb0 con 0x7f51dc100dc0 2024-04-05T23:33:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e4000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc074120 con 0x7f51dc102950 2024-04-05T23:33:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e4000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc1a0eb0 con 0x7f51dc101fb0 2024-04-05T23:33:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e1600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100dc0 0x7f51dc074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e1600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100dc0 0x7f51dc074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55772/0 (socket says 172.21.15.16:55772) 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e2000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc102950 0x7f51dc073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e1600700 1 -- 172.21.15.16:0/3590248868 learned_addr learned my addr 172.21.15.16:0/3590248868 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.461+0000 7f51e0c00700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51dc101fb0 0x7f51dc0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51e1600700 1 -- 172.21.15.16:0/3590248868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51dc101fb0 msgr2=0x7f51dc0732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51e1600700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51dc101fb0 0x7f51dc0732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51e1600700 1 -- 172.21.15.16:0/3590248868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc102950 msgr2=0x7f51dc073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51e1600700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc102950 0x7f51dc073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51e1600700 1 -- 172.21.15.16:0/3590248868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d4009d00 con 0x7f51dc100dc0 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51e2000700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc102950 0x7f51dc073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51e1600700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100dc0 0x7f51dc074c40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f51d000e970 tx=0x7f51d000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51da000700 1 -- 172.21.15.16:0/3590248868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0009790 con 0x7f51dc100dc0 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51da000700 1 -- 172.21.15.16:0/3590248868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51d000c780 con 0x7f51dc100dc0 2024-04-05T23:33:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51dc1a1140 con 0x7f51dc100dc0 2024-04-05T23:33:00.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51da000700 1 -- 172.21.15.16:0/3590248868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0003be0 con 0x7f51dc100dc0 2024-04-05T23:33:00.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.462+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51dc1a1650 con 0x7f51dc100dc0 2024-04-05T23:33:00.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.463+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51dc108210 con 0x7f51dc100dc0 2024-04-05T23:33:00.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.465+0000 7f51da000700 1 -- 172.21.15.16:0/3590248868 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f51d000cbf0 con 0x7f51dc100dc0 2024-04-05T23:33:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.467+0000 7f51da000700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f51c40a11b0 0x7f51c40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.467+0000 7f51da000700 1 -- 172.21.15.16:0/3590248868 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f51d0014070 con 0x7f51dc100dc0 2024-04-05T23:33:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.467+0000 7f51e0c00700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f51c40a11b0 0x7f51c40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.468+0000 7f51e0c00700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f51c40a11b0 0x7f51c40a3670 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f51c8007be0 tx=0x7f51c80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.471+0000 7f51da000700 1 -- 172.21.15.16:0/3590248868 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f51d0096270 con 0x7f51dc100dc0 2024-04-05T23:33:00.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:00 smithi016 ceph-mon[38248]: pgmap v262: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-05T23:33:00.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:00 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1127527876' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:00.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:00 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3470423745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:00.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:00 smithi027 ceph-mon[78597]: pgmap v262: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-05T23:33:00.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:00 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1127527876' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:00.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:00 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3470423745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:00.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.738+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51dc067b50 con 0x7f51dc100dc0 2024-04-05T23:33:00.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.739+0000 7f51da000700 1 -- 172.21.15.16:0/3590248868 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10902 (secure 0 0 0) 0x7f51d0099890 con 0x7f51dc100dc0 2024-04-05T23:33:00.732 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:00.732 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:32:11.436182+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f51c40a11b0 msgr2=0x7f51c40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f51c40a11b0 0x7f51c40a3670 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f51c8007be0 tx=0x7f51c80073d0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100dc0 msgr2=0x7f51dc074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100dc0 0x7f51dc074c40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f51d000e970 tx=0x7f51d000ec80 comp rx=0 tx=0).stop 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 shutdown_connections 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f51c40a11b0 0x7f51c40a3670 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc100dc0 0x7f51dc074c40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc102950 0x7f51dc073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 --2- 172.21.15.16:0/3590248868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51dc101fb0 0x7f51dc0732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 >> 172.21.15.16:0/3590248868 conn(0x7f51dc0fc350 msgr2=0x7f51dc105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 shutdown_connections 2024-04-05T23:33:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.742+0000 7f51e4000700 1 -- 172.21.15.16:0/3590248868 wait complete. 2024-04-05T23:33:00.736 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-04-05T23:33:00.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-05T23:32:04.511660+0000', 'modified': '2024-04-05T23:32:11.436182+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24566, 'mds_1': 24550, 'mds_2': 24526, 'mds_3': 24476, 'mds_4': 24472}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14529': {'gid': 14529, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6841/2579712996', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 2579712996}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 2579712996}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24452': {'gid': 24452, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6833/1368666940', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1368666940}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 1368666940}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24472': {'gid': 24472, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.130:6833/477825490', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 477825490}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 477825490}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24476': {'gid': 24476, 'name': 'd', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.16:6837/2898043660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 2898043660}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 2898043660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24488': {'gid': 24488, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6835/686908192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6834', 'nonce': 686908192}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 686908192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24518': {'gid': 24518, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6837/541968377', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6836', 'nonce': 541968377}, {'type': 'v1', 'addr': '172.21.15.27:6837', 'nonce': 541968377}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6835/3125278540', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 3125278540}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 3125278540}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24526': {'gid': 24526, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.130:6837/3873050281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 3873050281}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 3873050281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.130:6839/2970827239', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 2970827239}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 2970827239}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24566': {'gid': 24566, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6839/491598025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6838', 'nonce': 491598025}, {'type': 'v1', 'addr': '172.21.15.27:6839', 'nonce': 491598025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-05T23:33:00.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-05T23:33:00.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-04-05T23:33:00.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:00 smithi130 ceph-mon[79417]: pgmap v262: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-05T23:33:00.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:00 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1127527876' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:00.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:00 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3470423745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.960+0000 7f5847400700 1 -- 172.21.15.16:0/1331502424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 msgr2=0x7f5840073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.960+0000 7f5847400700 1 --2- 172.21.15.16:0/1331502424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 0x7f5840073370 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5834009990 tx=0x7f5834009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.960+0000 7f5847400700 1 -- 172.21.15.16:0/1331502424 shutdown_connections 2024-04-05T23:33:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.960+0000 7f5847400700 1 --2- 172.21.15.16:0/1331502424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 0x7f5840073370 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.960+0000 7f5847400700 1 --2- 172.21.15.16:0/1331502424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5840108d70 0x7f584010b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.960+0000 7f5847400700 1 --2- 172.21.15.16:0/1331502424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5840073940 0x7f5840073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.960+0000 7f5847400700 1 -- 172.21.15.16:0/1331502424 >> 172.21.15.16:0/1331502424 conn(0x7f58400fc440 msgr2=0x7f58400fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.960+0000 7f5847400700 1 -- 172.21.15.16:0/1331502424 shutdown_connections 2024-04-05T23:33:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.960+0000 7f5847400700 1 -- 172.21.15.16:0/1331502424 wait complete. 2024-04-05T23:33:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.961+0000 7f5847400700 1 Processor -- start 2024-04-05T23:33:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f5847400700 1 -- start start 2024-04-05T23:33:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f5847400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5840073940 0x7f5840199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f5847400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 0x7f58401995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f5847400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5840108d70 0x7f58401a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f5847400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58401a0ba0 con 0x7f5840074f20 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f5847400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58401a0ce0 con 0x7f5840108d70 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f5847400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58401a0e70 con 0x7f5840073940 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f583fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 0x7f58401995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f583fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 0x7f58401995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55790/0 (socket says 172.21.15.16:55790) 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f583fe00700 1 -- 172.21.15.16:0/2769265741 learned_addr learned my addr 172.21.15.16:0/2769265741 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f5845400700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5840108d70 0x7f58401a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.962+0000 7f5844a00700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5840073940 0x7f5840199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f583fe00700 1 -- 172.21.15.16:0/2769265741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5840073940 msgr2=0x7f5840199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f583fe00700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5840073940 0x7f5840199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f583fe00700 1 -- 172.21.15.16:0/2769265741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5840108d70 msgr2=0x7f58401a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f583fe00700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5840108d70 0x7f58401a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f583fe00700 1 -- 172.21.15.16:0/2769265741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340095f0 con 0x7f5840074f20 2024-04-05T23:33:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f583fe00700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 0x7f58401995b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f582c00e970 tx=0x7f582c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f583d600700 1 -- 172.21.15.16:0/2769265741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c009790 con 0x7f5840074f20 2024-04-05T23:33:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f583d600700 1 -- 172.21.15.16:0/2769265741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f582c00c7e0 con 0x7f5840074f20 2024-04-05T23:33:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f583d600700 1 -- 172.21.15.16:0/2769265741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c003c70 con 0x7f5840074f20 2024-04-05T23:33:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f5847400700 1 -- 172.21.15.16:0/2769265741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58401a1150 con 0x7f5840074f20 2024-04-05T23:33:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.963+0000 7f5847400700 1 -- 172.21.15.16:0/2769265741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58401a1610 con 0x7f5840074f20 2024-04-05T23:33:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.967+0000 7f583d600700 1 -- 172.21.15.16:0/2769265741 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f582c00cc50 con 0x7f5840074f20 2024-04-05T23:33:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f585ae67b80 1 Processor -- start 2024-04-05T23:33:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f583d600700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58280a11b0 0x7f58280a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f585ae67b80 1 -- start start 2024-04-05T23:33:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f583d600700 1 -- 172.21.15.16:0/2769265741 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f582c014070 con 0x7f5840074f20 2024-04-05T23:33:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f5845400700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58280a11b0 0x7f58280a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664834c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664834d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834da60 0x55664834dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f585ae67b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566482b2b90 con 0x55664834da60 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f585ae67b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566482a3940 con 0x55664834ce70 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.968+0000 7f585ae67b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566482a1300 con 0x55664834c510 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f5844a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664834c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f583fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664834d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f5845400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834da60 0x55664834dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f5844a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664834c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42414/0 (socket says 172.21.15.16:42414) 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f5844a00700 1 -- 172.21.15.16:0/153151615 learned_addr learned my addr 172.21.15.16:0/153151615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f583fe00700 1 -- 172.21.15.16:0/153151615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 msgr2=0x55664834c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f583fe00700 1 --2- 172.21.15.16:0/153151615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664834c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f583fe00700 1 -- 172.21.15.16:0/153151615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834da60 msgr2=0x55664834dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f583fe00700 1 --2- 172.21.15.16:0/153151615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834da60 0x55664834dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f583fe00700 1 -- 172.21.15.16:0/153151615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55664811e360 con 0x55664834ce70 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f5845400700 1 --2- 172.21.15.16:0/153151615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834da60 0x55664834dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f5844a00700 1 --2- 172.21.15.16:0/153151615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664834c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f5845400700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58280a11b0 0x7f58280a3670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5838009c70 tx=0x7f5838004320 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f583fe00700 1 --2- 172.21.15.16:0/153151615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664834d4d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f582c0ca610 tx=0x7f582c099980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43ba8adc08fc03aa server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.969+0000 7f5831c00700 1 -- 172.21.15.16:0/153151615 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c09dd80 con 0x55664834ce70 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f5831c00700 1 -- 172.21.15.16:0/153151615 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f582c0a23d0 con 0x55664834ce70 2024-04-05T23:33:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f5831c00700 1 -- 172.21.15.16:0/153151615 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c09a3b0 con 0x55664834ce70 2024-04-05T23:33:00.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f585ae67b80 1 -- 172.21.15.16:0/153151615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 msgr2=0x55664834d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f585ae67b80 1 --2- 172.21.15.16:0/153151615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664834d4d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f582c0ca610 tx=0x7f582c099980 comp rx=0 tx=0).stop 2024-04-05T23:33:00.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f585ae67b80 1 -- 172.21.15.16:0/153151615 shutdown_connections 2024-04-05T23:33:00.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f585ae67b80 1 --2- 172.21.15.16:0/153151615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834da60 0x55664834dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f585ae67b80 1 --2- 172.21.15.16:0/153151615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664834c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f585ae67b80 1 --2- 172.21.15.16:0/153151615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664834d4d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f585ae67b80 1 -- 172.21.15.16:0/153151615 >> 172.21.15.16:0/153151615 conn(0x5566482c01d0 msgr2=0x556648347050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f585ae67b80 1 -- 172.21.15.16:0/153151615 shutdown_connections 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.970+0000 7f585ae67b80 1 -- 172.21.15.16:0/153151615 wait complete. 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.971+0000 7f585ae67b80 1 -- start start 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.971+0000 7f585ae67b80 1 Processor -- start 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x556648356110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x556648356650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 0x55664835d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f585ae67b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55664829cbf0 con 0x55664834ce70 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f585ae67b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556648238d30 con 0x55664834da60 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f585ae67b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566482389d0 con 0x55664834c510 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f585ae67b80 10 client.0 _ll_register_callbacks cb 0x5566480a4350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f583fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x556648356650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f583fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x556648356650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55802/0 (socket says 172.21.15.16:55802) 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f583fe00700 1 -- 172.21.15.16:0/3177969955 learned_addr learned my addr 172.21.15.16:0/3177969955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f5844a00700 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x556648356110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.972+0000 7f5845400700 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 0x55664835d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f583fe00700 1 -- 172.21.15.16:0/3177969955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 msgr2=0x556648356110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f583fe00700 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x556648356110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f583fe00700 1 -- 172.21.15.16:0/3177969955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 msgr2=0x55664835d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f583fe00700 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 0x55664835d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f583fe00700 1 -- 172.21.15.16:0/3177969955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584010be20 con 0x55664834ce70 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f583fe00700 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x556648356650 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f582c099c50 tx=0x7f582c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc05f84ba7cc1c26 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f5826000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f5826000700 1 -- 172.21.15.16:0/3177969955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c09dd80 con 0x55664834ce70 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f5826000700 1 -- 172.21.15.16:0/3177969955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f582c09a3c0 con 0x55664834ce70 2024-04-05T23:33:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f585ae67b80 1 -- 172.21.15.16:0/3177969955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55664833f440 con 0x55664834ce70 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f5826000700 1 -- 172.21.15.16:0/3177969955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f582c0a1f40 con 0x55664834ce70 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f585ae67b80 10 client.14940 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.973+0000 7f585ae67b80 1 -- 172.21.15.16:0/3177969955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f58401a1610 con 0x55664834ce70 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.974+0000 7f5826000700 1 -- 172.21.15.16:0/3177969955 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55664833f440 con 0x55664834ce70 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.974+0000 7f5826000700 1 -- 172.21.15.16:0/3177969955 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f582c095d40 con 0x55664834ce70 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.974+0000 7f585ae67b80 10 client.14940 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.974+0000 7f585ae67b80 10 client.14940 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.974+0000 7f585ae67b80 20 client.14940 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.974+0000 7f585ae67b80 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556648360f30 0x556648366230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.974+0000 7f585ae67b80 4 client.14940 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.974+0000 7f585ae67b80 1 -- 172.21.15.16:0/3177969955 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566482956b0 con 0x556648360f30 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.974+0000 7f5844a00700 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556648360f30 0x556648366230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.975+0000 7f5844a00700 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556648360f30 0x556648366230 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d3f14d08be2f497e server_cookie=593d49c2a8777d81 in_seq=0 out_seq=0 2024-04-05T23:33:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.975+0000 7f5826000700 10 client.14940 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:33:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.976+0000 7f5826000700 1 -- 172.21.15.16:0/3177969955 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5566482956b0 con 0x556648360f30 2024-04-05T23:33:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.976+0000 7f5826000700 10 client.14940 handle_command_reply: tid=0 2024-04-05T23:33:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.976+0000 7f585ae67b80 1 client.14940 shutdown 2024-04-05T23:33:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.976+0000 7f585ae67b80 1 -- 172.21.15.16:0/3177969955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 msgr2=0x556648356650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.976+0000 7f585ae67b80 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x556648356650 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f582c099c50 tx=0x7f582c003680 comp rx=0 tx=0).stop 2024-04-05T23:33:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.976+0000 7f585ae67b80 1 -- 172.21.15.16:0/3177969955 shutdown_connections 2024-04-05T23:33:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.976+0000 7f585ae67b80 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 0x55664835d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.976+0000 7f585ae67b80 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x556648356110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.976+0000 7f585ae67b80 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556648360f30 0x556648366230 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.977+0000 7f585ae67b80 1 --2- 172.21.15.16:0/3177969955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x556648356650 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.977+0000 7f5826000700 0 client.14940 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:33:00.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.977+0000 7f585ae67b80 1 -- 172.21.15.16:0/3177969955 >> 172.21.15.16:0/3177969955 conn(0x5566482c01d0 msgr2=0x556648341410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:00.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.977+0000 7f585ae67b80 1 -- 172.21.15.16:0/3177969955 shutdown_connections 2024-04-05T23:33:00.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.977+0000 7f585ae67b80 1 -- 172.21.15.16:0/3177969955 wait complete. 2024-04-05T23:33:00.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.977+0000 7f585ae67b80 20 client.14940 trim_cache size 0 max 16384 2024-04-05T23:33:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.995+0000 7f585ae67b80 1 Processor -- start 2024-04-05T23:33:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.995+0000 7f585ae67b80 1 -- start start 2024-04-05T23:33:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834c510 0x556648354080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664835d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834da60 0x556648355620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f585ae67b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556648120650 con 0x55664834c510 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f585ae67b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55664825aad0 con 0x55664834ce70 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f585ae67b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556648299850 con 0x55664834da60 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5844a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834c510 0x556648354080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5844a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834c510 0x556648354080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55814/0 (socket says 172.21.15.16:55814) 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5844a00700 1 -- 172.21.15.16:0/419695059 learned_addr learned my addr 172.21.15.16:0/419695059 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f583fe00700 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664835d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5844a00700 1 -- 172.21.15.16:0/419695059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834da60 msgr2=0x556648355620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5845400700 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834da60 0x556648355620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5844a00700 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834da60 0x556648355620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5844a00700 1 -- 172.21.15.16:0/419695059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 msgr2=0x55664835d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5844a00700 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664835d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5844a00700 1 -- 172.21.15.16:0/419695059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566480cab10 con 0x55664834c510 2024-04-05T23:33:00.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f583fe00700 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664835d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:00.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5845400700 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834da60 0x556648355620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:33:00.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.996+0000 7f5844a00700 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834c510 0x556648354080 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f58340098e0 tx=0x7f5834005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14de415c43b1914c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f5832600700 1 -- 172.21.15.16:0/419695059 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583401d040 con 0x55664834c510 2024-04-05T23:33:00.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f5832600700 1 -- 172.21.15.16:0/419695059 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f583401e040 con 0x55664834c510 2024-04-05T23:33:00.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f5832600700 1 -- 172.21.15.16:0/419695059 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583400f6c0 con 0x55664834c510 2024-04-05T23:33:00.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f585ae67b80 1 -- 172.21.15.16:0/419695059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834c510 msgr2=0x556648354080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f585ae67b80 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834c510 0x556648354080 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f58340098e0 tx=0x7f5834005fd0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f585ae67b80 1 -- 172.21.15.16:0/419695059 shutdown_connections 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f585ae67b80 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834c510 0x556648354080 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f585ae67b80 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834da60 0x556648355620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f585ae67b80 1 --2- 172.21.15.16:0/419695059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834ce70 0x55664835d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f585ae67b80 1 -- 172.21.15.16:0/419695059 >> 172.21.15.16:0/419695059 conn(0x5566482c01d0 msgr2=0x556648347030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.997+0000 7f585ae67b80 1 -- 172.21.15.16:0/419695059 shutdown_connections 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.998+0000 7f585ae67b80 1 -- 172.21.15.16:0/419695059 wait complete. 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.998+0000 7f585ae67b80 1 -- start start 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.998+0000 7f585ae67b80 1 Processor -- start 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.998+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664835cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.998+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x55664835d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.998+0000 7f585ae67b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 0x5566483599f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.998+0000 7f585ae67b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556648359f70 con 0x55664834ce70 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.998+0000 7f585ae67b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55664835a0e0 con 0x55664834da60 2024-04-05T23:33:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.998+0000 7f585ae67b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55664835a2a0 con 0x55664834c510 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f585ae67b80 10 client.0 _ll_register_callbacks cb 0x7f582c005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f583fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x55664835d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5844a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664835cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5844a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664835cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42432/0 (socket says 172.21.15.16:42432) 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f583fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x55664835d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55828/0 (socket says 172.21.15.16:55828) 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5844a00700 1 -- 172.21.15.16:0/980303294 learned_addr learned my addr 172.21.15.16:0/980303294 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5845400700 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 0x5566483599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5844a00700 1 -- 172.21.15.16:0/980303294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 msgr2=0x5566483599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5844a00700 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 0x5566483599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5844a00700 1 -- 172.21.15.16:0/980303294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 msgr2=0x55664835d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5844a00700 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x55664835d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5844a00700 1 -- 172.21.15.16:0/980303294 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583400b850 con 0x55664834c510 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f583fe00700 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x55664835d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:00.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5844a00700 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664835cf00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f583400b680 tx=0x7f5834029160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=492cca70128b2608 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:00.999+0000 7f5826000700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.000+0000 7f5826000700 1 -- 172.21.15.16:0/980303294 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583401d070 con 0x55664834c510 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.000+0000 7f585ae67b80 1 -- 172.21.15.16:0/980303294 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f582c0a23d0 con 0x55664834c510 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.000+0000 7f5826000700 1 -- 172.21.15.16:0/980303294 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f583400bb30 con 0x55664834c510 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.000+0000 7f5826000700 1 -- 172.21.15.16:0/980303294 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5834040030 con 0x55664834c510 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.000+0000 7f585ae67b80 10 client.24994 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.000+0000 7f585ae67b80 1 -- 172.21.15.16:0/980303294 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556648262e50 con 0x55664834c510 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.000+0000 7f5826000700 1 -- 172.21.15.16:0/980303294 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f582c0a23d0 con 0x55664834c510 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.000+0000 7f5826000700 1 -- 172.21.15.16:0/980303294 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f5834038020 con 0x55664834c510 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.000+0000 7f585ae67b80 10 client.24994 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f585ae67b80 10 client.24994 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f585ae67b80 20 client.24994 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f585ae67b80 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5566483b9c10 0x5566483b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:00.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f585ae67b80 4 client.24994 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f585ae67b80 1 -- 172.21.15.16:0/980303294 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5566480ebca0 con 0x5566483b9c10 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f583fe00700 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5566483b9c10 0x5566483b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f583fe00700 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5566483b9c10 0x5566483b6580 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ce38703dfdc38ff1 server_cookie=f1c4e96b2661040f in_seq=0 out_seq=0 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f5826000700 10 client.24994 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f5826000700 1 -- 172.21.15.16:0/980303294 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5566480ebca0 con 0x5566483b9c10 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.001+0000 7f5826000700 10 client.24994 handle_command_reply: tid=0 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 client.24994 shutdown 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 -- 172.21.15.16:0/980303294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 msgr2=0x55664835cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664835cf00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f583400b680 tx=0x7f5834029160 comp rx=0 tx=0).stop 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 -- 172.21.15.16:0/980303294 shutdown_connections 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55664834da60 0x5566483599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55664834c510 0x55664835cf00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5566483b9c10 0x5566483b6580 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:00.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 --2- 172.21.15.16:0/980303294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55664834ce70 0x55664835d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 -- 172.21.15.16:0/980303294 >> 172.21.15.16:0/980303294 conn(0x5566482c01d0 msgr2=0x556648355620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f5826000700 0 client.24994 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 -- 172.21.15.16:0/980303294 shutdown_connections 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 1 -- 172.21.15.16:0/980303294 wait complete. 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.002+0000 7f585ae67b80 20 client.24994 trim_cache size 0 max 16384 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24472, 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:01.000 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:01.001 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:01.001 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 50.586806537000001, 2024-04-05T23:33:01.001 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:01.001 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:01.001 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:01.001 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 85.919591634 2024-04-05T23:33:01.001 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:01.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 -- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58280a11b0 msgr2=0x7f58280a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58280a11b0 0x7f58280a3670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5838009c70 tx=0x7f5838004320 comp rx=0 tx=0).stop 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 -- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 msgr2=0x7f58401995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 0x7f58401995b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f582c00e970 tx=0x7f582c00ec80 comp rx=0 tx=0).stop 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 -- 172.21.15.16:0/2769265741 shutdown_connections 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58280a11b0 0x7f58280a3670 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5840108d70 0x7f58401a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5840073940 0x7f5840199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 --2- 172.21.15.16:0/2769265741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5840074f20 0x7f58401995b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 -- 172.21.15.16:0/2769265741 >> 172.21.15.16:0/2769265741 conn(0x7f58400fc440 msgr2=0x7f5840104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 -- 172.21.15.16:0/2769265741 shutdown_connections 2024-04-05T23:33:01.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.003+0000 7f5847400700 1 -- 172.21.15.16:0/2769265741 wait complete. 2024-04-05T23:33:01.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-05T23:33:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.221+0000 7fb7b0c00700 1 -- 172.21.15.16:0/3058693487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac102050 msgr2=0x7fb7ac1024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.221+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/3058693487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac102050 0x7fb7ac1024d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb798009990 tx=0x7fb798009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.221+0000 7fb7b0c00700 1 -- 172.21.15.16:0/3058693487 shutdown_connections 2024-04-05T23:33:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.221+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/3058693487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ac102a10 0x7fb7ac109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.221+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/3058693487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7ac100e70 0x7fb7ac101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.221+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/3058693487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac102050 0x7fb7ac1024d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.221+0000 7fb7b0c00700 1 -- 172.21.15.16:0/3058693487 >> 172.21.15.16:0/3058693487 conn(0x7fb7ac0fc430 msgr2=0x7fb7ac0fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.221+0000 7fb7b0c00700 1 -- 172.21.15.16:0/3058693487 shutdown_connections 2024-04-05T23:33:01.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.221+0000 7fb7b0c00700 1 -- 172.21.15.16:0/3058693487 wait complete. 2024-04-05T23:33:01.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.222+0000 7fb7b0c00700 1 Processor -- start 2024-04-05T23:33:01.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.222+0000 7fb7b0c00700 1 -- start start 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.222+0000 7fb7b0c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac100e70 0x7fb7ac074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.222+0000 7fb7b0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ac102050 0x7fb7ac0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.222+0000 7fb7b0c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7ac102a10 0x7fb7ac073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.222+0000 7fb7b0c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac073fb0 con 0x7fb7ac100e70 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.222+0000 7fb7b0c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac074120 con 0x7fb7ac102050 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.222+0000 7fb7b0c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac1a0ec0 con 0x7fb7ac102a10 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac100e70 0x7fb7ac074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ac102050 0x7fb7ac0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac100e70 0x7fb7ac074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55850/0 (socket says 172.21.15.16:55850) 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9e00700 1 -- 172.21.15.16:0/444212888 learned_addr learned my addr 172.21.15.16:0/444212888 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7aa800700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7ac102a10 0x7fb7ac073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9e00700 1 -- 172.21.15.16:0/444212888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7ac102a10 msgr2=0x7fb7ac073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7ac102a10 0x7fb7ac073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9e00700 1 -- 172.21.15.16:0/444212888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ac102050 msgr2=0x7fb7ac0732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ac102050 0x7fb7ac0732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9e00700 1 -- 172.21.15.16:0/444212888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c009520 con 0x7fb7ac100e70 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7aa800700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7ac102a10 0x7fb7ac073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9400700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ac102050 0x7fb7ac0732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac100e70 0x7fb7ac074c40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00ea50 tx=0x7fb79c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a2a00700 1 -- 172.21.15.16:0/444212888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c00cb70 con 0x7fb7ac100e70 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a2a00700 1 -- 172.21.15.16:0/444212888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb79c00ccd0 con 0x7fb7ac100e70 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7b0c00700 1 -- 172.21.15.16:0/444212888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7980095f0 con 0x7fb7ac100e70 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.223+0000 7fb7a2a00700 1 -- 172.21.15.16:0/444212888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c018680 con 0x7fb7ac100e70 2024-04-05T23:33:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.224+0000 7fb7b0c00700 1 -- 172.21.15.16:0/444212888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac1a1520 con 0x7fb7ac100e70 2024-04-05T23:33:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.227+0000 7fb7a2a00700 1 -- 172.21.15.16:0/444212888 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fb79c010430 con 0x7fb7ac100e70 2024-04-05T23:33:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7c46cbb80 1 Processor -- start 2024-04-05T23:33:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7a2a00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7900a1230 0x7fb7900a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7c46cbb80 1 -- start start 2024-04-05T23:33:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7a2a00700 1 -- 172.21.15.16:0/444212888 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb79c014070 con 0x7fb7ac100e70 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7aa800700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7900a1230 0x7fb7900a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a537930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a5384d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a538ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a49db90 con 0x55a55a537510 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a48e940 con 0x55a55a538a60 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.228+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a48c300 con 0x55a55a537e70 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a537930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a5384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7aa800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a538ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a5384d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42458/0 (socket says 172.21.15.16:42458) 2024-04-05T23:33:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9400700 1 -- 172.21.15.16:0/1143660744 learned_addr learned my addr 172.21.15.16:0/1143660744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9400700 1 -- 172.21.15.16:0/1143660744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 msgr2=0x55a55a538ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9400700 1 --2- 172.21.15.16:0/1143660744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a538ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9400700 1 -- 172.21.15.16:0/1143660744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 msgr2=0x55a55a537930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9400700 1 --2- 172.21.15.16:0/1143660744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a537930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9400700 1 -- 172.21.15.16:0/1143660744 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a55a309360 con 0x55a55a537e70 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7aa800700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7900a1230 0x7fb7900a36f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007770 tx=0x7fb7a4007b30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.229+0000 7fb7a9400700 1 --2- 172.21.15.16:0/1143660744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a5384d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb798009990 tx=0x7fb798005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80d3424457fc4357 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb797000700 1 -- 172.21.15.16:0/1143660744 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79801d030 con 0x55a55a537e70 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb797000700 1 -- 172.21.15.16:0/1143660744 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb79801e040 con 0x55a55a537e70 2024-04-05T23:33:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb797000700 1 -- 172.21.15.16:0/1143660744 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798004c30 con 0x55a55a537e70 2024-04-05T23:33:01.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1143660744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 msgr2=0x55a55a5384d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/1143660744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a5384d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb798009990 tx=0x7fb798005e80 comp rx=0 tx=0).stop 2024-04-05T23:33:01.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1143660744 shutdown_connections 2024-04-05T23:33:01.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/1143660744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a537930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/1143660744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a538ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/1143660744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a5384d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1143660744 >> 172.21.15.16:0/1143660744 conn(0x55a55a4ab1d0 msgr2=0x55a55a532050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.230+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1143660744 shutdown_connections 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.231+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1143660744 wait complete. 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.231+0000 7fb7c46cbb80 1 -- start start 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.231+0000 7fb7c46cbb80 1 Processor -- start 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 0x55a55a5410c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a537e70 0x55a55a541600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a538a60 0x55a55a5485f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a487bf0 con 0x55a55a538a60 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a423d30 con 0x55a55a537e70 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a4239d0 con 0x55a55a537510 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7c46cbb80 10 client.0 _ll_register_callbacks cb 0x55a55a28f350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7aa800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a538a60 0x55a55a5485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7aa800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a538a60 0x55a55a5485f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55866/0 (socket says 172.21.15.16:55866) 2024-04-05T23:33:01.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7aa800700 1 -- 172.21.15.16:0/2949476246 learned_addr learned my addr 172.21.15.16:0/2949476246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 0x55a55a5410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7a9400700 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a537e70 0x55a55a541600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7aa800700 1 -- 172.21.15.16:0/2949476246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 msgr2=0x55a55a5410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7aa800700 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 0x55a55a5410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7aa800700 1 -- 172.21.15.16:0/2949476246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a537e70 msgr2=0x55a55a541600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7aa800700 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a537e70 0x55a55a541600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.232+0000 7fb7aa800700 1 -- 172.21.15.16:0/2949476246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac109c60 con 0x55a55a538a60 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.233+0000 7fb7aa800700 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a538a60 0x55a55a5485f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40094d0 tx=0x7fb7a4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe9118d9d5c3670 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.233+0000 7fb78b400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.233+0000 7fb78b400700 1 -- 172.21.15.16:0/2949476246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a400e050 con 0x55a55a538a60 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.233+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2949476246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a55a52a440 con 0x55a55a538a60 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.233+0000 7fb78b400700 1 -- 172.21.15.16:0/2949476246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb7a4023430 con 0x55a55a538a60 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.233+0000 7fb78b400700 1 -- 172.21.15.16:0/2949476246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7a40235f0 con 0x55a55a538a60 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.233+0000 7fb7c46cbb80 10 client.14958 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.233+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2949476246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a55a2b5b10 con 0x55a55a538a60 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.233+0000 7fb78b400700 1 -- 172.21.15.16:0/2949476246 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a55a52a440 con 0x55a55a538a60 2024-04-05T23:33:01.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb78b400700 1 -- 172.21.15.16:0/2949476246 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fb7a402c790 con 0x55a55a538a60 2024-04-05T23:33:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb7c46cbb80 10 client.14958 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb7c46cbb80 10 client.14958 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-05T23:33:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb7c46cbb80 20 client.14958 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55a55a54b790 0x55a55a5515d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb7c46cbb80 4 client.14958 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2949476246 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a55a4806b0 con 0x55a55a54b790 2024-04-05T23:33:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55a55a54b790 0x55a55a5515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55a55a54b790 0x55a55a5515d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dee50f4e250f0d51 server_cookie=ef64ddd1d0ae5c66 in_seq=0 out_seq=0 2024-04-05T23:33:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.234+0000 7fb78b400700 10 client.14958 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:33:01.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb78b400700 1 -- 172.21.15.16:0/2949476246 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a55a4806b0 con 0x55a55a54b790 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb78b400700 10 client.14958 handle_command_reply: tid=0 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb7c46cbb80 1 client.14958 shutdown 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2949476246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a538a60 msgr2=0x55a55a5485f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a538a60 0x55a55a5485f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40094d0 tx=0x7fb7a4004b40 comp rx=0 tx=0).stop 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2949476246 shutdown_connections 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55a55a54b790 0x55a55a5515d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a538a60 0x55a55a5485f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 0x55a55a5410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2949476246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a537e70 0x55a55a541600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.235+0000 7fb78b400700 0 client.14958 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.236+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2949476246 >> 172.21.15.16:0/2949476246 conn(0x55a55a4ab1d0 msgr2=0x55a55a52c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.236+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2949476246 shutdown_connections 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.236+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2949476246 wait complete. 2024-04-05T23:33:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.236+0000 7fb7c46cbb80 20 client.14958 trim_cache size 0 max 16384 2024-04-05T23:33:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.254+0000 7fb7c46cbb80 1 Processor -- start 2024-04-05T23:33:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.254+0000 7fb7c46cbb80 1 -- start start 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.254+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a53efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.254+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a548510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.254+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a53fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.254+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a30b650 con 0x55a55a537510 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.254+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a426e10 con 0x55a55a538a60 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.254+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a532f40 con 0x55a55a537e70 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a53efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a53efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55882/0 (socket says 172.21.15.16:55882) 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9e00700 1 -- 172.21.15.16:0/1881603329 learned_addr learned my addr 172.21.15.16:0/1881603329 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9400700 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a548510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7aa800700 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a53fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9e00700 1 -- 172.21.15.16:0/1881603329 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 msgr2=0x55a55a548510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a548510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9e00700 1 -- 172.21.15.16:0/1881603329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 msgr2=0x55a55a53fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a53fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9e00700 1 -- 172.21.15.16:0/1881603329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a55a44de50 con 0x55a55a537510 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7aa800700 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a53fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a53efc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb79c096760 tx=0x7fb79c0cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c41f4cd4b5d685 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb797a00700 1 -- 172.21.15.16:0/1881603329 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0af040 con 0x55a55a537510 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb797a00700 1 -- 172.21.15.16:0/1881603329 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb79c0a26e0 con 0x55a55a537510 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.255+0000 7fb797a00700 1 -- 172.21.15.16:0/1881603329 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0a2250 con 0x55a55a537510 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.256+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1881603329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 msgr2=0x55a55a53efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.256+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a53efc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb79c096760 tx=0x7fb79c0cc040 comp rx=0 tx=0).stop 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.256+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1881603329 shutdown_connections 2024-04-05T23:33:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.256+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537510 0x55a55a53efc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.256+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a53fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.256+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/1881603329 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537e70 0x55a55a548510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.256+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1881603329 >> 172.21.15.16:0/1881603329 conn(0x55a55a4ab1d0 msgr2=0x55a55a531ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.256+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1881603329 shutdown_connections 2024-04-05T23:33:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.256+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/1881603329 wait complete. 2024-04-05T23:33:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7c46cbb80 1 -- start start 2024-04-05T23:33:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7c46cbb80 1 Processor -- start 2024-04-05T23:33:01.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 0x55a55a543070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537e70 0x55a55a548450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7c46cbb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a543610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a5a0bc0 con 0x55a55a537e70 2024-04-05T23:33:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a5a0d00 con 0x55a55a538a60 2024-04-05T23:33:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7c46cbb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55a5a0e40 con 0x55a55a537510 2024-04-05T23:33:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7c46cbb80 10 client.0 _ll_register_callbacks cb 0x7fb7a401a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7a9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537e70 0x55a55a548450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.257+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 0x55a55a543070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7aa800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a543610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7a9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537e70 0x55a55a548450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55888/0 (socket says 172.21.15.16:55888) 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7a9400700 1 -- 172.21.15.16:0/2975287368 learned_addr learned my addr 172.21.15.16:0/2975287368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7a9e00700 1 -- 172.21.15.16:0/2975287368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 msgr2=0x55a55a543610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a543610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7a9e00700 1 -- 172.21.15.16:0/2975287368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537e70 msgr2=0x55a55a548450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537e70 0x55a55a548450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7a9e00700 1 -- 172.21.15.16:0/2975287368 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac1a1520 con 0x55a55a537510 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7aa800700 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a543610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7a9400700 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537e70 0x55a55a548450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7a9e00700 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 0x55a55a543070 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb79c096cd0 tx=0x7fb79c010bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8742aed7b8c12714 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb78b400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb78b400700 1 -- 172.21.15.16:0/2975287368 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0af070 con 0x55a55a537510 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2975287368 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb79801e040 con 0x55a55a537510 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.258+0000 7fb78b400700 1 -- 172.21.15.16:0/2975287368 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb79c0a2250 con 0x55a55a537510 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb78b400700 1 -- 172.21.15.16:0/2975287368 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb79c0a2410 con 0x55a55a537510 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb7c46cbb80 10 client.25012 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2975287368 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a55a30dc80 con 0x55a55a537510 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb78b400700 1 -- 172.21.15.16:0/2975287368 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb79801e040 con 0x55a55a537510 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb78b400700 1 -- 172.21.15.16:0/2975287368 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fb79c081060 con 0x55a55a537510 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb7c46cbb80 10 client.25012 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb7c46cbb80 10 client.25012 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb7c46cbb80 20 client.25012 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55a55a5a59a0 0x55a55a5a2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb7c46cbb80 4 client.25012 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2975287368 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a55a2d6ca0 con 0x55a55a5a59a0 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.259+0000 7fb7a9400700 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55a55a5a59a0 0x55a55a5a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7a9400700 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55a55a5a59a0 0x55a55a5a2330 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=86d6aa5bd33c66cd server_cookie=b7c818888469864 in_seq=0 out_seq=0 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb78b400700 10 client.25012 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb78b400700 1 -- 172.21.15.16:0/2975287368 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55a55a2d6ca0 con 0x55a55a5a59a0 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb78b400700 10 client.25012 handle_command_reply: tid=0 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 client.25012 shutdown 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2975287368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 msgr2=0x55a55a543070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 0x55a55a543070 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb79c096cd0 tx=0x7fb79c010bc0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2975287368 shutdown_connections 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55a55a5a59a0 0x55a55a5a2330 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a55a538a60 0x55a55a543610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a55a537510 0x55a55a543070 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 --2- 172.21.15.16:0/2975287368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a55a537e70 0x55a55a548450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb78b400700 0 client.25012 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2975287368 >> 172.21.15.16:0/2975287368 conn(0x55a55a4ab1d0 msgr2=0x55a55a53ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2975287368 shutdown_connections 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.260+0000 7fb7c46cbb80 1 -- 172.21.15.16:0/2975287368 wait complete. 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7c46cbb80 20 client.25012 trim_cache size 0 max 16384 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24476, 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 51.851819855000002, 2024-04-05T23:33:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 79.547910423000005 2024-04-05T23:33:01.261 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 -- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7900a1230 msgr2=0x7fb7900a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7900a1230 0x7fb7900a36f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007770 tx=0x7fb7a4007b30 comp rx=0 tx=0).stop 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 -- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac100e70 msgr2=0x7fb7ac074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac100e70 0x7fb7ac074c40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00ea50 tx=0x7fb79c00ed60 comp rx=0 tx=0).stop 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 -- 172.21.15.16:0/444212888 shutdown_connections 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fb7900a1230 0x7fb7900a36f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac100e70 0x7fb7ac074c40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7ac102a10 0x7fb7ac073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 --2- 172.21.15.16:0/444212888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ac102050 0x7fb7ac0732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 -- 172.21.15.16:0/444212888 >> 172.21.15.16:0/444212888 conn(0x7fb7ac0fc430 msgr2=0x7fb7ac105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 -- 172.21.15.16:0/444212888 shutdown_connections 2024-04-05T23:33:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.261+0000 7fb7b0c00700 1 -- 172.21.15.16:0/444212888 wait complete. 2024-04-05T23:33:01.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-05T23:33:01.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.481+0000 7f635c000700 1 -- 172.21.15.16:0/1145259656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354102040 msgr2=0x7f63541024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.481+0000 7f635c000700 1 --2- 172.21.15.16:0/1145259656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354102040 0x7f63541024c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6340009990 tx=0x7f6340009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.481+0000 7f635c000700 1 -- 172.21.15.16:0/1145259656 shutdown_connections 2024-04-05T23:33:01.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.481+0000 7f635c000700 1 --2- 172.21.15.16:0/1145259656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6354102a00 0x7f6354109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.481+0000 7f635c000700 1 --2- 172.21.15.16:0/1145259656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6354100e60 0x7f6354101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.481+0000 7f635c000700 1 --2- 172.21.15.16:0/1145259656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354102040 0x7f63541024c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.481+0000 7f635c000700 1 -- 172.21.15.16:0/1145259656 >> 172.21.15.16:0/1145259656 conn(0x7f63540fc440 msgr2=0x7f63540fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.481+0000 7f635c000700 1 -- 172.21.15.16:0/1145259656 shutdown_connections 2024-04-05T23:33:01.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.481+0000 7f635c000700 1 -- 172.21.15.16:0/1145259656 wait complete. 2024-04-05T23:33:01.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.482+0000 7f635c000700 1 Processor -- start 2024-04-05T23:33:01.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.482+0000 7f635c000700 1 -- start start 2024-04-05T23:33:01.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.482+0000 7f635c000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354100e60 0x7f6354074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.482+0000 7f635c000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6354102040 0x7f63540732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.482+0000 7f635c000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6354102a00 0x7f6354073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.482+0000 7f635c000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6354073fb0 con 0x7f6354100e60 2024-04-05T23:33:01.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.482+0000 7f635c000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6354074120 con 0x7f6354102040 2024-04-05T23:33:01.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.482+0000 7f635c000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541a0f20 con 0x7f6354102a00 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6359600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354100e60 0x7f6354074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6359600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354100e60 0x7f6354074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55908/0 (socket says 172.21.15.16:55908) 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6359600700 1 -- 172.21.15.16:0/1172491139 learned_addr learned my addr 172.21.15.16:0/1172491139 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6358c00700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6354102040 0x7f63540732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f635a000700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6354102a00 0x7f6354073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6359600700 1 -- 172.21.15.16:0/1172491139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6354102a00 msgr2=0x7f6354073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6359600700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6354102a00 0x7f6354073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6359600700 1 -- 172.21.15.16:0/1172491139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6354102040 msgr2=0x7f63540732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6359600700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6354102040 0x7f63540732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6359600700 1 -- 172.21.15.16:0/1172491139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63400095f0 con 0x7f6354100e60 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6359600700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354100e60 0x7f6354074c40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f634800e970 tx=0x7f634800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.483+0000 7f6352000700 1 -- 172.21.15.16:0/1172491139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348009790 con 0x7f6354100e60 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.484+0000 7f6352000700 1 -- 172.21.15.16:0/1172491139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f634800c780 con 0x7f6354100e60 2024-04-05T23:33:01.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.484+0000 7f635c000700 1 -- 172.21.15.16:0/1172491139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63541a11b0 con 0x7f6354100e60 2024-04-05T23:33:01.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.484+0000 7f6352000700 1 -- 172.21.15.16:0/1172491139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348003be0 con 0x7f6354100e60 2024-04-05T23:33:01.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.484+0000 7f635c000700 1 -- 172.21.15.16:0/1172491139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63541a1710 con 0x7f6354100e60 2024-04-05T23:33:01.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.487+0000 7f6352000700 1 -- 172.21.15.16:0/1172491139 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f634800cbf0 con 0x7f6354100e60 2024-04-05T23:33:01.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.488+0000 7f6352000700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f633c0a1230 0x7f633c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.488+0000 7f636fa8cb80 1 Processor -- start 2024-04-05T23:33:01.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.488+0000 7f6352000700 1 -- 172.21.15.16:0/1172491139 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6348014070 con 0x7f6354100e60 2024-04-05T23:33:01.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.488+0000 7f636fa8cb80 1 -- start start 2024-04-05T23:33:01.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.488+0000 7f6358c00700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f633c0a1230 0x7f633c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314510 0x5601bf314930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf3154d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf315a60 0x5601bf315ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf27ab90 con 0x5601bf315a60 2024-04-05T23:33:01.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf26b940 con 0x5601bf314e70 2024-04-05T23:33:01.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf269300 con 0x5601bf314510 2024-04-05T23:33:01.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f635a000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf315a60 0x5601bf315ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f6359600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314510 0x5601bf314930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f635a000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf315a60 0x5601bf315ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55920/0 (socket says 172.21.15.16:55920) 2024-04-05T23:33:01.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f635a000700 1 -- 172.21.15.16:0/3394252493 learned_addr learned my addr 172.21.15.16:0/3394252493 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f6359600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314510 0x5601bf314930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42510/0 (socket says 172.21.15.16:42510) 2024-04-05T23:33:01.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f6358c00700 1 --2- 172.21.15.16:0/3394252493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf3154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f6358c00700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f633c0a1230 0x7f633c0a36f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6340000c00 tx=0x7f6340005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f635a000700 1 -- 172.21.15.16:0/3394252493 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314510 msgr2=0x5601bf314930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f635a000700 1 --2- 172.21.15.16:0/3394252493 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314510 0x5601bf314930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f635a000700 1 -- 172.21.15.16:0/3394252493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 msgr2=0x5601bf3154d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f635a000700 1 --2- 172.21.15.16:0/3394252493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf3154d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f635a000700 1 -- 172.21.15.16:0/3394252493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601bf0e6360 con 0x5601bf315a60 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.489+0000 7f6359600700 1 --2- 172.21.15.16:0/3394252493 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314510 0x5601bf314930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.490+0000 7f635a000700 1 --2- 172.21.15.16:0/3394252493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf315a60 0x5601bf315ee0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f634c003f90 tx=0x7f634c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2269b6f20efa4d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.490+0000 7f6346600700 1 -- 172.21.15.16:0/3394252493 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c0046f0 con 0x5601bf315a60 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.490+0000 7f6346600700 1 -- 172.21.15.16:0/3394252493 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f634c00f040 con 0x5601bf315a60 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.490+0000 7f6346600700 1 -- 172.21.15.16:0/3394252493 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c015770 con 0x5601bf315a60 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3394252493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf315a60 msgr2=0x5601bf315ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3394252493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf315a60 0x5601bf315ee0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f634c003f90 tx=0x7f634c009c20 comp rx=0 tx=0).stop 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3394252493 shutdown_connections 2024-04-05T23:33:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3394252493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf315a60 0x5601bf315ee0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3394252493 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314510 0x5601bf314930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3394252493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf3154d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3394252493 >> 172.21.15.16:0/3394252493 conn(0x5601bf2881d0 msgr2=0x5601bf30f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3394252493 shutdown_connections 2024-04-05T23:33:01.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3394252493 wait complete. 2024-04-05T23:33:01.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 -- start start 2024-04-05T23:33:01.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.491+0000 7f636fa8cb80 1 Processor -- start 2024-04-05T23:33:01.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf31e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 0x5601bf31e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 0x5601bf325610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf264bf0 con 0x5601bf314510 2024-04-05T23:33:01.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf200d30 con 0x5601bf315a60 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf2009d0 con 0x5601bf314e70 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f6359600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf31e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f636fa8cb80 10 client.0 _ll_register_callbacks cb 0x5601bf06c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f6358c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 0x5601bf31e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f6358c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 0x5601bf31e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42516/0 (socket says 172.21.15.16:42516) 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f6358c00700 1 -- 172.21.15.16:0/3258368141 learned_addr learned my addr 172.21.15.16:0/3258368141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f6359600700 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf31e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55934/0 (socket says 172.21.15.16:55934) 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.492+0000 7f635a000700 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 0x5601bf325610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f6359600700 1 -- 172.21.15.16:0/3258368141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 msgr2=0x5601bf31e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f6359600700 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 0x5601bf31e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f6359600700 1 -- 172.21.15.16:0/3258368141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 msgr2=0x5601bf325610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f6359600700 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 0x5601bf325610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f6359600700 1 -- 172.21.15.16:0/3258368141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63541a1710 con 0x5601bf314510 2024-04-05T23:33:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f6359600700 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf31e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f63480ca540 tx=0x7f634800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a79756fee34f13e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f633aa00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f633aa00700 1 -- 172.21.15.16:0/3258368141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634809dcb0 con 0x5601bf314510 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f633aa00700 1 -- 172.21.15.16:0/3258368141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f63480a2300 con 0x5601bf314510 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3258368141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601bf307440 con 0x5601bf314510 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f633aa00700 1 -- 172.21.15.16:0/3258368141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63480b81d0 con 0x5601bf314510 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f636fa8cb80 10 client.14976 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3258368141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f634c003bf0 con 0x5601bf314510 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.493+0000 7f633aa00700 1 -- 172.21.15.16:0/3258368141 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f634807c020 con 0x5601bf314510 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.494+0000 7f633aa00700 1 -- 172.21.15.16:0/3258368141 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f63480a1e70 con 0x5601bf314510 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.494+0000 7f636fa8cb80 10 client.14976 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.494+0000 7f636fa8cb80 10 client.14976 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.494+0000 7f636fa8cb80 20 client.14976 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.494+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5601bf3292a0 0x5601bf32e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.494+0000 7f636fa8cb80 4 client.14976 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:01.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.494+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3258368141 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601bf25d6b0 con 0x5601bf3292a0 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.494+0000 7f6358c00700 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5601bf3292a0 0x5601bf32e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.495+0000 7f6358c00700 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5601bf3292a0 0x5601bf32e5c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d56c6758aa3b947d server_cookie=d397786a2c6a762b in_seq=0 out_seq=0 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.495+0000 7f633aa00700 10 client.14976 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.495+0000 7f633aa00700 1 -- 172.21.15.16:0/3258368141 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5601bf25d6b0 con 0x5601bf3292a0 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.495+0000 7f633aa00700 10 client.14976 handle_command_reply: tid=0 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.495+0000 7f636fa8cb80 1 client.14976 shutdown 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3258368141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 msgr2=0x5601bf31e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf31e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f63480ca540 tx=0x7f634800c9e0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3258368141 shutdown_connections 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5601bf3292a0 0x5601bf32e5c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf31e090 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 0x5601bf325610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f633aa00700 0 client.14976 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:33:01.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/3258368141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 0x5601bf31e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3258368141 >> 172.21.15.16:0/3258368141 conn(0x5601bf2881d0 msgr2=0x5601bf309410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3258368141 shutdown_connections 2024-04-05T23:33:01.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 1 -- 172.21.15.16:0/3258368141 wait complete. 2024-04-05T23:33:01.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.496+0000 7f636fa8cb80 20 client.14976 trim_cache size 0 max 16384 2024-04-05T23:33:01.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f636fa8cb80 1 Processor -- start 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f636fa8cb80 1 -- start start 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf3148f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 0x5601bf325cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 0x5601bf31d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf0e8650 con 0x5601bf314510 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf31ef90 con 0x5601bf315a60 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf25d6b0 con 0x5601bf314e70 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f6359600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf3148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f6359600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf3148f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55936/0 (socket says 172.21.15.16:55936) 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f6359600700 1 -- 172.21.15.16:0/927416479 learned_addr learned my addr 172.21.15.16:0/927416479 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f635a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 0x5601bf31d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f6358c00700 1 --2- 172.21.15.16:0/927416479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 0x5601bf325cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f6359600700 1 -- 172.21.15.16:0/927416479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 msgr2=0x5601bf325cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f6359600700 1 --2- 172.21.15.16:0/927416479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 0x5601bf325cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f6359600700 1 -- 172.21.15.16:0/927416479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 msgr2=0x5601bf31d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.515+0000 7f6359600700 1 --2- 172.21.15.16:0/927416479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 0x5601bf31d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.516+0000 7f6359600700 1 -- 172.21.15.16:0/927416479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601bf092b10 con 0x5601bf314510 2024-04-05T23:33:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.516+0000 7f6359600700 1 --2- 172.21.15.16:0/927416479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf3148f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f63480968a0 tx=0x7f6348003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8e1a9cc4b526ff server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.516+0000 7f6347000700 1 -- 172.21.15.16:0/927416479 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63480b81d0 con 0x5601bf314510 2024-04-05T23:33:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.516+0000 7f6347000700 1 -- 172.21.15.16:0/927416479 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f634807c030 con 0x5601bf314510 2024-04-05T23:33:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.516+0000 7f6347000700 1 -- 172.21.15.16:0/927416479 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63480cac40 con 0x5601bf314510 2024-04-05T23:33:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.516+0000 7f636fa8cb80 1 -- 172.21.15.16:0/927416479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 msgr2=0x5601bf3148f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.516+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/927416479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf3148f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f63480968a0 tx=0x7f6348003680 comp rx=0 tx=0).stop 2024-04-05T23:33:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.517+0000 7f636fa8cb80 1 -- 172.21.15.16:0/927416479 shutdown_connections 2024-04-05T23:33:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.517+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/927416479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf3148f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.517+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/927416479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf315a60 0x5601bf31d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.517+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/927416479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf314e70 0x5601bf325cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.517+0000 7f636fa8cb80 1 -- 172.21.15.16:0/927416479 >> 172.21.15.16:0/927416479 conn(0x5601bf2881d0 msgr2=0x5601bf30f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.517+0000 7f636fa8cb80 1 -- 172.21.15.16:0/927416479 shutdown_connections 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.517+0000 7f636fa8cb80 1 -- 172.21.15.16:0/927416479 wait complete. 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.517+0000 7f636fa8cb80 1 -- start start 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.517+0000 7f636fa8cb80 1 Processor -- start 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf321790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf321cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f636fa8cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf315a60 0x5601bf324c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf325190 con 0x5601bf314510 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf325300 con 0x5601bf314e70 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f636fa8cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bf3254c0 con 0x5601bf315a60 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f636fa8cb80 10 client.0 _ll_register_callbacks cb 0x7f63480997e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6359600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf321790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6358c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf321cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f635a000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf315a60 0x5601bf324c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6358c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf321cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:55688/0 (socket says 172.21.15.16:55688) 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6359600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf321790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55950/0 (socket says 172.21.15.16:55950) 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6358c00700 1 -- 172.21.15.16:0/1815276000 learned_addr learned my addr 172.21.15.16:0/1815276000 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6358c00700 1 -- 172.21.15.16:0/1815276000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf315a60 msgr2=0x5601bf324c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6358c00700 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf315a60 0x5601bf324c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6358c00700 1 -- 172.21.15.16:0/1815276000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 msgr2=0x5601bf321790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6358c00700 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf321790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.518+0000 7f6358c00700 1 -- 172.21.15.16:0/1815276000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c003bf0 con 0x5601bf314e70 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.519+0000 7f6358c00700 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf321cd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f634000b900 tx=0x7f6340009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a14080549e57e7e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.519+0000 7f633aa00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.519+0000 7f633aa00700 1 -- 172.21.15.16:0/1815276000 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340039070 con 0x5601bf314e70 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.519+0000 7f636fa8cb80 1 -- 172.21.15.16:0/1815276000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601bf31d0a0 con 0x5601bf314e70 2024-04-05T23:33:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.519+0000 7f633aa00700 1 -- 172.21.15.16:0/1815276000 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6340034520 con 0x5601bf314e70 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.519+0000 7f633aa00700 1 -- 172.21.15.16:0/1815276000 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6340034d80 con 0x5601bf314e70 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.519+0000 7f636fa8cb80 10 client.25007 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.519+0000 7f636fa8cb80 1 -- 172.21.15.16:0/1815276000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5601bf22ae50 con 0x5601bf314e70 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.520+0000 7f633aa00700 1 -- 172.21.15.16:0/1815276000 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5601bf31d0a0 con 0x5601bf314e70 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.520+0000 7f633aa00700 1 -- 172.21.15.16:0/1815276000 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f6340025470 con 0x5601bf314e70 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.520+0000 7f636fa8cb80 10 client.25007 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.520+0000 7f636fa8cb80 10 client.25007 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.520+0000 7f636fa8cb80 20 client.25007 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.520+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5601bf382270 0x5601bf37ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.520+0000 7f636fa8cb80 4 client.25007 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.520+0000 7f636fa8cb80 1 -- 172.21.15.16:0/1815276000 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f634c00f040 con 0x5601bf382270 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.520+0000 7f6359600700 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5601bf382270 0x5601bf37ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f6359600700 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5601bf382270 0x5601bf37ebe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b7d543d9c560f4c6 server_cookie=caba795285ffa9c9 in_seq=0 out_seq=0 2024-04-05T23:33:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f633aa00700 10 client.25007 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:33:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f633aa00700 1 -- 172.21.15.16:0/1815276000 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7f634c00f040 con 0x5601bf382270 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f633aa00700 10 client.25007 handle_command_reply: tid=0 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 client.25007 shutdown 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 -- 172.21.15.16:0/1815276000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 msgr2=0x5601bf321cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf321cd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f634000b900 tx=0x7f6340009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 -- 172.21.15.16:0/1815276000 shutdown_connections 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5601bf382270 0x5601bf37ebe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601bf314510 0x5601bf321790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601bf315a60 0x5601bf324c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 --2- 172.21.15.16:0/1815276000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5601bf314e70 0x5601bf321cd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f633aa00700 0 client.25007 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 -- 172.21.15.16:0/1815276000 >> 172.21.15.16:0/1815276000 conn(0x5601bf2881d0 msgr2=0x5601bf3325e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 -- 172.21.15.16:0/1815276000 shutdown_connections 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 1 -- 172.21.15.16:0/1815276000 wait complete. 2024-04-05T23:33:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.521+0000 7f636fa8cb80 20 client.25007 trim_cache size 0 max 16384 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24526, 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 52.519222669999998, 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 74.323696687999998 2024-04-05T23:33:01.520 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 -- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f633c0a1230 msgr2=0x7f633c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f633c0a1230 0x7f633c0a36f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6340000c00 tx=0x7f6340005fb0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 -- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354100e60 msgr2=0x7f6354074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354100e60 0x7f6354074c40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f634800e970 tx=0x7f634800ec80 comp rx=0 tx=0).stop 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 -- 172.21.15.16:0/1172491139 shutdown_connections 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f633c0a1230 0x7f633c0a36f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6354100e60 0x7f6354074c40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6354102a00 0x7f6354073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 --2- 172.21.15.16:0/1172491139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6354102040 0x7f63540732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 -- 172.21.15.16:0/1172491139 >> 172.21.15.16:0/1172491139 conn(0x7f63540fc440 msgr2=0x7f6354105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 -- 172.21.15.16:0/1172491139 shutdown_connections 2024-04-05T23:33:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.522+0000 7f635c000700 1 -- 172.21.15.16:0/1172491139 wait complete. 2024-04-05T23:33:01.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:01 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3590248868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:01.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-05T23:33:01.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:01 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3590248868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:01.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.739+0000 7ff65be00700 1 -- 172.21.15.16:0/4219564904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654102040 msgr2=0x7ff6541024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.739+0000 7ff65be00700 1 --2- 172.21.15.16:0/4219564904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654102040 0x7ff6541024c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff640009990 tx=0x7ff640009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.739+0000 7ff65be00700 1 -- 172.21.15.16:0/4219564904 shutdown_connections 2024-04-05T23:33:01.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.739+0000 7ff65be00700 1 --2- 172.21.15.16:0/4219564904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff654102a00 0x7ff654109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.739+0000 7ff65be00700 1 --2- 172.21.15.16:0/4219564904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff654100e60 0x7ff654101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.739+0000 7ff65be00700 1 --2- 172.21.15.16:0/4219564904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654102040 0x7ff6541024c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.739+0000 7ff65be00700 1 -- 172.21.15.16:0/4219564904 >> 172.21.15.16:0/4219564904 conn(0x7ff6540fc440 msgr2=0x7ff6540fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.739+0000 7ff65be00700 1 -- 172.21.15.16:0/4219564904 shutdown_connections 2024-04-05T23:33:01.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.739+0000 7ff65be00700 1 -- 172.21.15.16:0/4219564904 wait complete. 2024-04-05T23:33:01.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.740+0000 7ff65be00700 1 Processor -- start 2024-04-05T23:33:01.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.740+0000 7ff65be00700 1 -- start start 2024-04-05T23:33:01.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.740+0000 7ff65be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff654100e60 0x7ff654199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.740+0000 7ff65be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff654102040 0x7ff6541995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.740+0000 7ff65be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654102a00 0x7ff6541a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.740+0000 7ff65be00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6541a0b90 con 0x7ff654102a00 2024-04-05T23:33:01.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.740+0000 7ff65be00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6541a0cd0 con 0x7ff654100e60 2024-04-05T23:33:01.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.740+0000 7ff65be00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6541a0e60 con 0x7ff654102040 2024-04-05T23:33:01.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654102a00 0x7ff6541a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff654100e60 0x7ff654199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff654100e60 0x7ff654199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:55708/0 (socket says 172.21.15.16:55708) 2024-04-05T23:33:01.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659400700 1 -- 172.21.15.16:0/1546379984 learned_addr learned my addr 172.21.15.16:0/1546379984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff658a00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff654102040 0x7ff6541995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659400700 1 -- 172.21.15.16:0/1546379984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff654102040 msgr2=0x7ff6541995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659400700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff654102040 0x7ff6541995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659400700 1 -- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654102a00 msgr2=0x7ff6541a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659400700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654102a00 0x7ff6541a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659400700 1 -- 172.21.15.16:0/1546379984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff648009520 con 0x7ff654100e60 2024-04-05T23:33:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659e00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654102a00 0x7ff6541a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff659400700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff654100e60 0x7ff654199060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff64800ea50 tx=0x7ff64800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff652000700 1 -- 172.21.15.16:0/1546379984 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64800cbd0 con 0x7ff654100e60 2024-04-05T23:33:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff652000700 1 -- 172.21.15.16:0/1546379984 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff64800cd30 con 0x7ff654100e60 2024-04-05T23:33:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.741+0000 7ff652000700 1 -- 172.21.15.16:0/1546379984 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff648018680 con 0x7ff654100e60 2024-04-05T23:33:01.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.742+0000 7ff65be00700 1 -- 172.21.15.16:0/1546379984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6400095f0 con 0x7ff654100e60 2024-04-05T23:33:01.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.742+0000 7ff65be00700 1 -- 172.21.15.16:0/1546379984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6541a1470 con 0x7ff654100e60 2024-04-05T23:33:01.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.746+0000 7ff652000700 1 -- 172.21.15.16:0/1546379984 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7ff648010430 con 0x7ff654100e60 2024-04-05T23:33:01.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.747+0000 7ff66f884b80 1 Processor -- start 2024-04-05T23:33:01.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.747+0000 7ff66f884b80 1 -- start start 2024-04-05T23:33:01.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.747+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae22930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.747+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae234d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.747+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae23ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.747+0000 7ff66f884b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ad88b90 con 0x559c6ae22e70 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.747+0000 7ff66f884b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ad79940 con 0x559c6ae23a60 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.747+0000 7ff66f884b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ad77300 con 0x559c6ae22510 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.747+0000 7ff652000700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff63c0a11b0 0x7ff63c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff652000700 1 -- 172.21.15.16:0/1546379984 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff648014070 con 0x7ff654100e60 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff659e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae22930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff659400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff658a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae23ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff659e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae22930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42570/0 (socket says 172.21.15.16:42570) 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff658a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae23ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:55720/0 (socket says 172.21.15.16:55720) 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff658a00700 1 -- 172.21.15.16:0/2397788127 learned_addr learned my addr 172.21.15.16:0/2397788127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff659400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae234d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55988/0 (socket says 172.21.15.16:55988) 2024-04-05T23:33:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff659e00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff63c0a11b0 0x7ff63c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff658a00700 1 -- 172.21.15.16:0/2397788127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 msgr2=0x559c6ae22930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff658a00700 1 --2- 172.21.15.16:0/2397788127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae22930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff658a00700 1 -- 172.21.15.16:0/2397788127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 msgr2=0x559c6ae234d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff658a00700 1 --2- 172.21.15.16:0/2397788127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae234d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff658a00700 1 -- 172.21.15.16:0/2397788127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c6abf4360 con 0x559c6ae23a60 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.748+0000 7ff659400700 1 --2- 172.21.15.16:0/2397788127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae234d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.749+0000 7ff658a00700 1 --2- 172.21.15.16:0/2397788127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae23ee0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff640009990 tx=0x7ff640005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3f2f9f11a25fc8f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.749+0000 7ff659e00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff63c0a11b0 0x7ff63c0a3670 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009550 tx=0x7ff64c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.749+0000 7ff646600700 1 -- 172.21.15.16:0/2397788127 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64001d070 con 0x559c6ae23a60 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.749+0000 7ff646600700 1 -- 172.21.15.16:0/2397788127 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff64001e040 con 0x559c6ae23a60 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.749+0000 7ff646600700 1 -- 172.21.15.16:0/2397788127 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6400049c0 con 0x559c6ae23a60 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.749+0000 7ff66f884b80 1 -- 172.21.15.16:0/2397788127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 msgr2=0x559c6ae23ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.749+0000 7ff66f884b80 1 --2- 172.21.15.16:0/2397788127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae23ee0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff640009990 tx=0x7ff640005e80 comp rx=0 tx=0).stop 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.750+0000 7ff66f884b80 1 -- 172.21.15.16:0/2397788127 shutdown_connections 2024-04-05T23:33:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.750+0000 7ff66f884b80 1 --2- 172.21.15.16:0/2397788127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae23ee0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.750+0000 7ff66f884b80 1 --2- 172.21.15.16:0/2397788127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae22930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.750+0000 7ff66f884b80 1 --2- 172.21.15.16:0/2397788127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae234d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.750+0000 7ff66f884b80 1 -- 172.21.15.16:0/2397788127 >> 172.21.15.16:0/2397788127 conn(0x559c6ad961d0 msgr2=0x559c6ae1d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.750+0000 7ff66f884b80 1 -- 172.21.15.16:0/2397788127 shutdown_connections 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.750+0000 7ff66f884b80 1 -- 172.21.15.16:0/2397788127 wait complete. 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.750+0000 7ff66f884b80 1 -- start start 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.750+0000 7ff66f884b80 1 Processor -- start 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae2c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae2c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae335c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff66f884b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ad72bf0 con 0x559c6ae22e70 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff66f884b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ad0ed30 con 0x559c6ae23a60 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff66f884b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ad0e9d0 con 0x559c6ae22510 2024-04-05T23:33:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff66f884b80 10 client.0 _ll_register_callbacks cb 0x559c6ab7a350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff658a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae2c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff658a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae2c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56004/0 (socket says 172.21.15.16:56004) 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff658a00700 1 -- 172.21.15.16:0/1982514865 learned_addr learned my addr 172.21.15.16:0/1982514865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff659e00700 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff659400700 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae2c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff658a00700 1 -- 172.21.15.16:0/1982514865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 msgr2=0x559c6ae2c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff658a00700 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae2c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff658a00700 1 -- 172.21.15.16:0/1982514865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 msgr2=0x559c6ae335c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff658a00700 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae335c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff658a00700 1 -- 172.21.15.16:0/1982514865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6541a1470 con 0x559c6ae22e70 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.751+0000 7ff659400700 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae2c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.752+0000 7ff658a00700 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae2c5d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff640009ce0 tx=0x7ff64000f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0db447f5372d48d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.752+0000 7ff63aa00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.752+0000 7ff63aa00700 1 -- 172.21.15.16:0/1982514865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64001d070 con 0x559c6ae22e70 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.752+0000 7ff63aa00700 1 -- 172.21.15.16:0/1982514865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff64000fd30 con 0x559c6ae22e70 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.752+0000 7ff66f884b80 1 -- 172.21.15.16:0/1982514865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c6ae15440 con 0x559c6ae22e70 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.752+0000 7ff63aa00700 1 -- 172.21.15.16:0/1982514865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff640005030 con 0x559c6ae22e70 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.752+0000 7ff66f884b80 10 client.14994 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.752+0000 7ff66f884b80 1 -- 172.21.15.16:0/1982514865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7ff654109c50 con 0x559c6ae22e70 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.753+0000 7ff63aa00700 1 -- 172.21.15.16:0/1982514865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559c6ae15440 con 0x559c6ae22e70 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.753+0000 7ff63aa00700 1 -- 172.21.15.16:0/1982514865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff640004c70 con 0x559c6ae22e70 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.753+0000 7ff66f884b80 10 client.14994 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.753+0000 7ff66f884b80 10 client.14994 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.753+0000 7ff66f884b80 20 client.14994 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.753+0000 7ff66f884b80 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x559c6ae36d20 0x559c6ae3c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.753+0000 7ff66f884b80 4 client.14994 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.753+0000 7ff66f884b80 1 -- 172.21.15.16:0/1982514865 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c6ad6b6b0 con 0x559c6ae36d20 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.753+0000 7ff659400700 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x559c6ae36d20 0x559c6ae3c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.754+0000 7ff659400700 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x559c6ae36d20 0x559c6ae3c040 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96e05e2eed0face4 server_cookie=44f3f270c394f358 in_seq=0 out_seq=0 2024-04-05T23:33:01.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.754+0000 7ff63aa00700 10 client.14994 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.754+0000 7ff63aa00700 1 -- 172.21.15.16:0/1982514865 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559c6ad6b6b0 con 0x559c6ae36d20 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.754+0000 7ff63aa00700 10 client.14994 handle_command_reply: tid=0 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 client.14994 shutdown 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 -- 172.21.15.16:0/1982514865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 msgr2=0x559c6ae2c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae2c5d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff640009ce0 tx=0x7ff64000f6f0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 -- 172.21.15.16:0/1982514865 shutdown_connections 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x559c6ae36d20 0x559c6ae3c040 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae2c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff63aa00700 0 client.14994 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 --2- 172.21.15.16:0/1982514865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22e70 0x559c6ae2c5d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 -- 172.21.15.16:0/1982514865 >> 172.21.15.16:0/1982514865 conn(0x559c6ad961d0 msgr2=0x559c6ae17410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 -- 172.21.15.16:0/1982514865 shutdown_connections 2024-04-05T23:33:01.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 1 -- 172.21.15.16:0/1982514865 wait complete. 2024-04-05T23:33:01.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.755+0000 7ff66f884b80 20 client.14994 trim_cache size 0 max 16384 2024-04-05T23:33:01.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.773+0000 7ff66f884b80 1 Processor -- start 2024-04-05T23:33:01.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.773+0000 7ff66f884b80 1 -- start start 2024-04-05T23:33:01.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae228f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae22e70 0x559c6ae2b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae23a60 0x559c6ae33550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff66f884b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6abf6650 con 0x559c6ae23a60 2024-04-05T23:33:01.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff66f884b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ad6b6b0 con 0x559c6ae22e70 2024-04-05T23:33:01.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff66f884b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ae1db40 con 0x559c6ae22510 2024-04-05T23:33:01.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae23a60 0x559c6ae33550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae23a60 0x559c6ae33550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56010/0 (socket says 172.21.15.16:56010) 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659e00700 1 -- 172.21.15.16:0/4274969871 learned_addr learned my addr 172.21.15.16:0/4274969871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659400700 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae228f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff658a00700 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae22e70 0x559c6ae2b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659e00700 1 -- 172.21.15.16:0/4274969871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 msgr2=0x559c6ae228f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659e00700 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae228f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659e00700 1 -- 172.21.15.16:0/4274969871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae22e70 msgr2=0x559c6ae2b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659e00700 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae22e70 0x559c6ae2b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659e00700 1 -- 172.21.15.16:0/4274969871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c6ae2cfe0 con 0x559c6ae23a60 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff658a00700 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae22e70 0x559c6ae2b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.774+0000 7ff659e00700 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae23a60 0x559c6ae33550 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00eb60 tx=0x7ff64c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd627391df9c91c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff647000700 1 -- 172.21.15.16:0/4274969871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c02a070 con 0x559c6ae23a60 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff647000700 1 -- 172.21.15.16:0/4274969871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff64c00f040 con 0x559c6ae23a60 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff647000700 1 -- 172.21.15.16:0/4274969871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0044f0 con 0x559c6ae23a60 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff66f884b80 1 -- 172.21.15.16:0/4274969871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae23a60 msgr2=0x559c6ae33550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff66f884b80 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae23a60 0x559c6ae33550 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00eb60 tx=0x7ff64c009920 comp rx=0 tx=0).stop 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff66f884b80 1 -- 172.21.15.16:0/4274969871 shutdown_connections 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff66f884b80 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae23a60 0x559c6ae33550 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff66f884b80 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22510 0x559c6ae228f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff66f884b80 1 --2- 172.21.15.16:0/4274969871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae22e70 0x559c6ae2b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff66f884b80 1 -- 172.21.15.16:0/4274969871 >> 172.21.15.16:0/4274969871 conn(0x559c6ad961d0 msgr2=0x559c6ae1d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.775+0000 7ff66f884b80 1 -- 172.21.15.16:0/4274969871 shutdown_connections 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.776+0000 7ff66f884b80 1 -- 172.21.15.16:0/4274969871 wait complete. 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.776+0000 7ff66f884b80 1 -- start start 2024-04-05T23:33:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.776+0000 7ff66f884b80 1 Processor -- start 2024-04-05T23:33:01.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.776+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22510 0x559c6ae2dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.776+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22e70 0x559c6ae2e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.776+0000 7ff66f884b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae32ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.776+0000 7ff66f884b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ae33530 con 0x559c6ae22510 2024-04-05T23:33:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff66f884b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ae336a0 con 0x559c6ae23a60 2024-04-05T23:33:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff66f884b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ae33860 con 0x559c6ae22e70 2024-04-05T23:33:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff66f884b80 10 client.0 _ll_register_callbacks cb 0x7ff648096650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22510 0x559c6ae2dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22510 0x559c6ae2dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56026/0 (socket says 172.21.15.16:56026) 2024-04-05T23:33:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659400700 1 -- 172.21.15.16:0/943510020 learned_addr learned my addr 172.21.15.16:0/943510020 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff658a00700 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22e70 0x559c6ae2e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659e00700 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae32ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659400700 1 -- 172.21.15.16:0/943510020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22e70 msgr2=0x559c6ae2e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659400700 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22e70 0x559c6ae2e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659400700 1 -- 172.21.15.16:0/943510020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 msgr2=0x559c6ae32ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659400700 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae32ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659400700 1 -- 172.21.15.16:0/943510020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff654109c50 con 0x559c6ae22510 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff659400700 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22510 0x559c6ae2dd00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff648096720 tx=0x7ff648085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d970791d68b9bb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff63aa00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.777+0000 7ff63aa00700 1 -- 172.21.15.16:0/943510020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6480b6070 con 0x559c6ae22510 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff63aa00700 1 -- 172.21.15.16:0/943510020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff6480b0030 con 0x559c6ae22510 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff66f884b80 1 -- 172.21.15.16:0/943510020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff64001e040 con 0x559c6ae22510 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff63aa00700 1 -- 172.21.15.16:0/943510020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff64801e4d0 con 0x559c6ae22510 2024-04-05T23:33:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff66f884b80 10 client.15000 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff66f884b80 1 -- 172.21.15.16:0/943510020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7ff64c008de0 con 0x559c6ae22510 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff63aa00700 1 -- 172.21.15.16:0/943510020 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff64001e040 con 0x559c6ae22510 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff63aa00700 1 -- 172.21.15.16:0/943510020 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff6480a2210 con 0x559c6ae22510 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff66f884b80 10 client.15000 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff66f884b80 10 client.15000 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff66f884b80 20 client.15000 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff66f884b80 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x559c6ae98970 0x559c6ae95300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff66f884b80 4 client.15000 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff66f884b80 1 -- 172.21.15.16:0/943510020 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559c6abc1ca0 con 0x559c6ae98970 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.778+0000 7ff658a00700 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x559c6ae98970 0x559c6ae95300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff658a00700 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x559c6ae98970 0x559c6ae95300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b71f6141fdeb1c4 server_cookie=da0d8c5bf6c0938 in_seq=0 out_seq=0 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff63aa00700 10 client.15000 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff63aa00700 1 -- 172.21.15.16:0/943510020 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x559c6abc1ca0 con 0x559c6ae98970 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff63aa00700 10 client.15000 handle_command_reply: tid=0 2024-04-05T23:33:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff66f884b80 1 client.15000 shutdown 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff66f884b80 1 -- 172.21.15.16:0/943510020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22510 msgr2=0x559c6ae2dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff66f884b80 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22510 0x559c6ae2dd00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff648096720 tx=0x7ff648085040 comp rx=0 tx=0).stop 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff66f884b80 1 -- 172.21.15.16:0/943510020 shutdown_connections 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff66f884b80 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c6ae22510 0x559c6ae2dd00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff66f884b80 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x559c6ae98970 0x559c6ae95300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff66f884b80 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c6ae23a60 0x559c6ae32ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff66f884b80 1 --2- 172.21.15.16:0/943510020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c6ae22e70 0x559c6ae2e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff63aa00700 0 client.15000 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.779+0000 7ff66f884b80 1 -- 172.21.15.16:0/943510020 >> 172.21.15.16:0/943510020 conn(0x559c6ad961d0 msgr2=0x559c6ae2be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff66f884b80 1 -- 172.21.15.16:0/943510020 shutdown_connections 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff66f884b80 1 -- 172.21.15.16:0/943510020 wait complete. 2024-04-05T23:33:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff66f884b80 20 client.15000 trim_cache size 0 max 16384 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24550, 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 53.782399097000003, 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 70.907341243999994 2024-04-05T23:33:01.779 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 -- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff63c0a11b0 msgr2=0x7ff63c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff63c0a11b0 0x7ff63c0a3670 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009550 tx=0x7ff64c00b040 comp rx=0 tx=0).stop 2024-04-05T23:33:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 -- 172.21.15.16:0/1546379984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff654100e60 msgr2=0x7ff654199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff654100e60 0x7ff654199060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff64800ea50 tx=0x7ff64800ed60 comp rx=0 tx=0).stop 2024-04-05T23:33:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 -- 172.21.15.16:0/1546379984 shutdown_connections 2024-04-05T23:33:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff63c0a11b0 0x7ff63c0a3670 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654102a00 0x7ff6541a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff654100e60 0x7ff654199060 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 --2- 172.21.15.16:0/1546379984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff654102040 0x7ff6541995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.780+0000 7ff65be00700 1 -- 172.21.15.16:0/1546379984 >> 172.21.15.16:0/1546379984 conn(0x7ff6540fc440 msgr2=0x7ff6541078f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.781+0000 7ff65be00700 1 -- 172.21.15.16:0/1546379984 shutdown_connections 2024-04-05T23:33:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.781+0000 7ff65be00700 1 -- 172.21.15.16:0/1546379984 wait complete. 2024-04-05T23:33:01.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-05T23:33:01.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:01 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3590248868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.997+0000 7ff62d200700 1 -- 172.21.15.16:0/4023355723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100e60 msgr2=0x7ff628101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.997+0000 7ff62d200700 1 --2- 172.21.15.16:0/4023355723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100e60 0x7ff628101280 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff618009990 tx=0x7ff618009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.997+0000 7ff62d200700 1 -- 172.21.15.16:0/4023355723 shutdown_connections 2024-04-05T23:33:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.997+0000 7ff62d200700 1 --2- 172.21.15.16:0/4023355723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102a00 0x7ff628109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.997+0000 7ff62d200700 1 --2- 172.21.15.16:0/4023355723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628100e60 0x7ff628101280 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.997+0000 7ff62d200700 1 --2- 172.21.15.16:0/4023355723 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628102040 0x7ff6281024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.997+0000 7ff62d200700 1 -- 172.21.15.16:0/4023355723 >> 172.21.15.16:0/4023355723 conn(0x7ff6280fc440 msgr2=0x7ff6280fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.997+0000 7ff62d200700 1 -- 172.21.15.16:0/4023355723 shutdown_connections 2024-04-05T23:33:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.997+0000 7ff62d200700 1 -- 172.21.15.16:0/4023355723 wait complete. 2024-04-05T23:33:01.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.998+0000 7ff62d200700 1 Processor -- start 2024-04-05T23:33:01.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.998+0000 7ff62d200700 1 -- start start 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.998+0000 7ff62d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628100e60 0x7ff628196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.998+0000 7ff62d200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102040 0x7ff6281973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.998+0000 7ff62d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628102a00 0x7ff62819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.998+0000 7ff62d200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62819e9d0 con 0x7ff628102040 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.998+0000 7ff62d200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62819eb10 con 0x7ff628102a00 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.998+0000 7ff62d200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62819eca0 con 0x7ff628100e60 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102040 0x7ff6281973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102040 0x7ff6281973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56046/0 (socket says 172.21.15.16:56046) 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625400700 1 -- 172.21.15.16:0/2921060019 learned_addr learned my addr 172.21.15.16:0/2921060019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625e00700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628100e60 0x7ff628196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff626800700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628102a00 0x7ff62819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625400700 1 -- 172.21.15.16:0/2921060019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628100e60 msgr2=0x7ff628196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625400700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628100e60 0x7ff628196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625400700 1 -- 172.21.15.16:0/2921060019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628102a00 msgr2=0x7ff62819e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:01.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625400700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628102a00 0x7ff62819e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:01.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625400700 1 -- 172.21.15.16:0/2921060019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6180095f0 con 0x7ff628102040 2024-04-05T23:33:01.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff625400700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102040 0x7ff6281973e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff61400d980 tx=0x7ff61400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:01.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:01.999+0000 7ff61ea00700 1 -- 172.21.15.16:0/2921060019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614004020 con 0x7ff628102040 2024-04-05T23:33:01.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.000+0000 7ff61ea00700 1 -- 172.21.15.16:0/2921060019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff61400f690 con 0x7ff628102040 2024-04-05T23:33:01.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.000+0000 7ff62d200700 1 -- 172.21.15.16:0/2921060019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62819ef80 con 0x7ff628102040 2024-04-05T23:33:01.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.000+0000 7ff61ea00700 1 -- 172.21.15.16:0/2921060019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614003a60 con 0x7ff628102040 2024-04-05T23:33:01.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.000+0000 7ff62d200700 1 -- 172.21.15.16:0/2921060019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62819f450 con 0x7ff628102040 2024-04-05T23:33:01.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.003+0000 7ff61ea00700 1 -- 172.21.15.16:0/2921060019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7ff61400fb00 con 0x7ff628102040 2024-04-05T23:33:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.004+0000 7ff61ea00700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff60c0a1230 0x7ff60c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.004+0000 7ff640d05b80 1 Processor -- start 2024-04-05T23:33:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.004+0000 7ff61ea00700 1 -- 172.21.15.16:0/2921060019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff6140cc5d0 con 0x7ff628102040 2024-04-05T23:33:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.004+0000 7ff625e00700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff60c0a1230 0x7ff60c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff640d05b80 1 -- start start 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 0x55b57d777d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7788e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7792f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff640d05b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d6d9b90 con 0x55b57d778e70 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff640d05b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d6ca940 con 0x55b57d778280 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff640d05b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d6c8300 con 0x55b57d777920 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff625400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff626800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7792f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff625400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7788e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:34894/0 (socket says 172.21.15.16:34894) 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff625400700 1 -- 172.21.15.16:0/411601144 learned_addr learned my addr 172.21.15.16:0/411601144 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:01.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff625e00700 1 --2- 172.21.15.16:0/411601144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 0x55b57d777d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff625400700 1 -- 172.21.15.16:0/411601144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 msgr2=0x55b57d777d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff625400700 1 --2- 172.21.15.16:0/411601144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 0x55b57d777d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff625400700 1 -- 172.21.15.16:0/411601144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 msgr2=0x55b57d7792f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff625400700 1 --2- 172.21.15.16:0/411601144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7792f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.005+0000 7ff625400700 1 -- 172.21.15.16:0/411601144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b57d545360 con 0x55b57d778280 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.006+0000 7ff625e00700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff60c0a1230 0x7ff60c0a36f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff618000c00 tx=0x7ff618005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.006+0000 7ff625400700 1 --2- 172.21.15.16:0/411601144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7788e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff614095d70 tx=0x7ff6140994e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15c3815b55b58eed server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.006+0000 7ff613400700 1 -- 172.21.15.16:0/411601144 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614099870 con 0x55b57d778280 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.006+0000 7ff613400700 1 -- 172.21.15.16:0/411601144 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff61409d6a0 con 0x55b57d778280 2024-04-05T23:33:02.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.006+0000 7ff613400700 1 -- 172.21.15.16:0/411601144 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6140b92c0 con 0x55b57d778280 2024-04-05T23:33:02.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 -- 172.21.15.16:0/411601144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 msgr2=0x55b57d7788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:02.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 --2- 172.21.15.16:0/411601144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7788e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff614095d70 tx=0x7ff6140994e0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 -- 172.21.15.16:0/411601144 shutdown_connections 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 --2- 172.21.15.16:0/411601144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7792f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 --2- 172.21.15.16:0/411601144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 0x55b57d777d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 --2- 172.21.15.16:0/411601144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7788e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 -- 172.21.15.16:0/411601144 >> 172.21.15.16:0/411601144 conn(0x55b57d6e71d0 msgr2=0x55b57d76e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 -- 172.21.15.16:0/411601144 shutdown_connections 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 -- 172.21.15.16:0/411601144 wait complete. 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.007+0000 7ff640d05b80 1 -- start start 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff640d05b80 1 Processor -- start 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 0x55b57d781500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778280 0x55b57d781a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778e70 0x55b57d788a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff640d05b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d6c3bf0 con 0x55b57d778280 2024-04-05T23:33:02.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff640d05b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d65fd30 con 0x55b57d777920 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff640d05b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d65f9d0 con 0x55b57d778e70 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff640d05b80 10 client.0 _ll_register_callbacks cb 0x55b57d4cb350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff625400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778280 0x55b57d781a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff625e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 0x55b57d781500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff625400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778280 0x55b57d781a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48960/0 (socket says 172.21.15.16:48960) 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff625400700 1 -- 172.21.15.16:0/728069928 learned_addr learned my addr 172.21.15.16:0/728069928 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.008+0000 7ff626800700 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778e70 0x55b57d788a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff625400700 1 -- 172.21.15.16:0/728069928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778e70 msgr2=0x55b57d788a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff625400700 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778e70 0x55b57d788a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff625400700 1 -- 172.21.15.16:0/728069928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 msgr2=0x55b57d781500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff625400700 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 0x55b57d781500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff625400700 1 -- 172.21.15.16:0/728069928 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62819f450 con 0x55b57d778280 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff625400700 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778280 0x55b57d781a40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff61400be80 tx=0x7ff614004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d1568695cb391b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:02.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff607e00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff607e00700 1 -- 172.21.15.16:0/728069928 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614099870 con 0x55b57d778280 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff607e00700 1 -- 172.21.15.16:0/728069928 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff6140ca180 con 0x55b57d778280 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff640d05b80 1 -- 172.21.15.16:0/728069928 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b57d766440 con 0x55b57d778280 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff607e00700 1 -- 172.21.15.16:0/728069928 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff61407e030 con 0x55b57d778280 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff640d05b80 10 client.15012 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff640d05b80 1 -- 172.21.15.16:0/728069928 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b57d4f1b10 con 0x55b57d778280 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.009+0000 7ff607e00700 1 -- 172.21.15.16:0/728069928 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b57d766440 con 0x55b57d778280 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.010+0000 7ff607e00700 1 -- 172.21.15.16:0/728069928 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff6140a1860 con 0x55b57d778280 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.010+0000 7ff640d05b80 10 client.15012 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.010+0000 7ff640d05b80 10 client.15012 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.010+0000 7ff640d05b80 20 client.15012 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.010+0000 7ff640d05b80 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b57d78c2b0 0x55b57d791590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.010+0000 7ff640d05b80 4 client.15012 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:02.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.010+0000 7ff640d05b80 1 -- 172.21.15.16:0/728069928 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b57d6bc6b0 con 0x55b57d78c2b0 2024-04-05T23:33:02.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.010+0000 7ff625e00700 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b57d78c2b0 0x55b57d791590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.011+0000 7ff625e00700 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b57d78c2b0 0x55b57d791590 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb76b81f508cd99f server_cookie=a761f1c8810c3657 in_seq=0 out_seq=0 2024-04-05T23:33:02.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.011+0000 7ff607e00700 10 client.15012 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff607e00700 1 -- 172.21.15.16:0/728069928 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b57d6bc6b0 con 0x55b57d78c2b0 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff607e00700 10 client.15012 handle_command_reply: tid=0 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 client.15012 shutdown 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 -- 172.21.15.16:0/728069928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778280 msgr2=0x55b57d781a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778280 0x55b57d781a40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff61400be80 tx=0x7ff614004750 comp rx=0 tx=0).stop 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 -- 172.21.15.16:0/728069928 shutdown_connections 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b57d78c2b0 0x55b57d791590 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778e70 0x55b57d788a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff607e00700 0 client.15012 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 0x55b57d781500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 --2- 172.21.15.16:0/728069928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778280 0x55b57d781a40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 -- 172.21.15.16:0/728069928 >> 172.21.15.16:0/728069928 conn(0x55b57d6e71d0 msgr2=0x55b57d768410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:02.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 -- 172.21.15.16:0/728069928 shutdown_connections 2024-04-05T23:33:02.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 1 -- 172.21.15.16:0/728069928 wait complete. 2024-04-05T23:33:02.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.012+0000 7ff640d05b80 20 client.15012 trim_cache size 0 max 16384 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.030+0000 7ff640d05b80 1 Processor -- start 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff640d05b80 1 -- start start 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 0x55b57d780a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7889a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff640d05b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d547650 con 0x55b57d778e70 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff640d05b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d6bc6e0 con 0x55b57d778280 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff640d05b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d6bc990 con 0x55b57d777920 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff626800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff625e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 0x55b57d780a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff626800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7955b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48964/0 (socket says 172.21.15.16:48964) 2024-04-05T23:33:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff626800700 1 -- 172.21.15.16:0/3707645011 learned_addr learned my addr 172.21.15.16:0/3707645011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.031+0000 7ff625400700 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff626800700 1 -- 172.21.15.16:0/3707645011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 msgr2=0x55b57d780a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff626800700 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 0x55b57d780a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff626800700 1 -- 172.21.15.16:0/3707645011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 msgr2=0x55b57d7889a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff626800700 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7889a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff626800700 1 -- 172.21.15.16:0/3707645011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b57d689e50 con 0x55b57d778e70 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff625400700 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7889a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff625e00700 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 0x55b57d780a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff626800700 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7955b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff6200118f0 tx=0x7ff620009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a64bbbd8d3296c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff613e00700 1 -- 172.21.15.16:0/3707645011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000e040 con 0x55b57d778e70 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff613e00700 1 -- 172.21.15.16:0/3707645011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff62000f040 con 0x55b57d778e70 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.032+0000 7ff613e00700 1 -- 172.21.15.16:0/3707645011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200079b0 con 0x55b57d778e70 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 -- 172.21.15.16:0/3707645011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 msgr2=0x55b57d7955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7955b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff6200118f0 tx=0x7ff620009920 comp rx=0 tx=0).stop 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 -- 172.21.15.16:0/3707645011 shutdown_connections 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d7955b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d777920 0x55b57d780a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 --2- 172.21.15.16:0/3707645011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d778280 0x55b57d7889a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 -- 172.21.15.16:0/3707645011 >> 172.21.15.16:0/3707645011 conn(0x55b57d6e71d0 msgr2=0x55b57d76dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:02.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 -- 172.21.15.16:0/3707645011 shutdown_connections 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 -- 172.21.15.16:0/3707645011 wait complete. 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 -- start start 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.033+0000 7ff640d05b80 1 Processor -- start 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 0x55b57d788380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778280 0x55b57d7888c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff640d05b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d784f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff640d05b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d785640 con 0x55b57d778e70 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff640d05b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d7857b0 con 0x55b57d777920 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff640d05b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57d785970 con 0x55b57d778280 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff640d05b80 10 client.0 _ll_register_callbacks cb 0x7ff620009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff626800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d784f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff626800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d784f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48976/0 (socket says 172.21.15.16:48976) 2024-04-05T23:33:02.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff626800700 1 -- 172.21.15.16:0/2263874359 learned_addr learned my addr 172.21.15.16:0/2263874359 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff625e00700 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 0x55b57d788380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff625400700 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778280 0x55b57d7888c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff626800700 1 -- 172.21.15.16:0/2263874359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778280 msgr2=0x55b57d7888c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff626800700 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778280 0x55b57d7888c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff626800700 1 -- 172.21.15.16:0/2263874359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 msgr2=0x55b57d788380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff626800700 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 0x55b57d788380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.034+0000 7ff626800700 1 -- 172.21.15.16:0/2263874359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff628109c50 con 0x55b57d778e70 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.035+0000 7ff626800700 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d784f60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff620006080 tx=0x7ff620017780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7687b1e9b347b91 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.035+0000 7ff607e00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.035+0000 7ff607e00700 1 -- 172.21.15.16:0/2263874359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000e030 con 0x55b57d778e70 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.035+0000 7ff640d05b80 1 -- 172.21.15.16:0/2263874359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff61409d6a0 con 0x55b57d778e70 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.035+0000 7ff607e00700 1 -- 172.21.15.16:0/2263874359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff6200092e0 con 0x55b57d778e70 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.035+0000 7ff607e00700 1 -- 172.21.15.16:0/2263874359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff620017b80 con 0x55b57d778e70 2024-04-05T23:33:02.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.035+0000 7ff640d05b80 10 client.15018 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.035+0000 7ff640d05b80 1 -- 172.21.15.16:0/2263874359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7ff6200115d0 con 0x55b57d778e70 2024-04-05T23:33:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.035+0000 7ff607e00700 1 -- 172.21.15.16:0/2263874359 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff620037080 con 0x55b57d778e70 2024-04-05T23:33:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.036+0000 7ff607e00700 1 -- 172.21.15.16:0/2263874359 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff6200208b0 con 0x55b57d778e70 2024-04-05T23:33:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.036+0000 7ff640d05b80 10 client.15018 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.036+0000 7ff640d05b80 10 client.15018 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-05T23:33:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.036+0000 7ff640d05b80 20 client.15018 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.036+0000 7ff640d05b80 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b57d7e5560 0x55b57d7e1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.036+0000 7ff640d05b80 4 client.15018 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.036+0000 7ff640d05b80 1 -- 172.21.15.16:0/2263874359 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b57d512ca0 con 0x55b57d7e5560 2024-04-05T23:33:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.036+0000 7ff625e00700 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b57d7e5560 0x55b57d7e1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff625e00700 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b57d7e5560 0x55b57d7e1eb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d97983130b360c0e server_cookie=382481c068f3c008 in_seq=0 out_seq=0 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff607e00700 10 client.15018 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff607e00700 1 -- 172.21.15.16:0/2263874359 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55b57d512ca0 con 0x55b57d7e5560 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff607e00700 10 client.15018 handle_command_reply: tid=0 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 client.15018 shutdown 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 -- 172.21.15.16:0/2263874359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 msgr2=0x55b57d784f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d784f60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff620006080 tx=0x7ff620017780 comp rx=0 tx=0).stop 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 -- 172.21.15.16:0/2263874359 shutdown_connections 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b57d7e5560 0x55b57d7e1eb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b57d778e70 0x55b57d784f60 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b57d777920 0x55b57d788380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 --2- 172.21.15.16:0/2263874359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b57d778280 0x55b57d7888c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 -- 172.21.15.16:0/2263874359 >> 172.21.15.16:0/2263874359 conn(0x55b57d6e71d0 msgr2=0x55b57d7809e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff607e00700 0 client.15018 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 -- 172.21.15.16:0/2263874359 shutdown_connections 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 1 -- 172.21.15.16:0/2263874359 wait complete. 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.037+0000 7ff640d05b80 20 client.15018 trim_cache size 0 max 16384 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24566, 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 57.522928810000003, 2024-04-05T23:33:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 64.437130714999995 2024-04-05T23:33:02.036 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 -- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff60c0a1230 msgr2=0x7ff60c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff60c0a1230 0x7ff60c0a36f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff618000c00 tx=0x7ff618005fb0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 -- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102040 msgr2=0x7ff6281973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102040 0x7ff6281973e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff61400d980 tx=0x7ff61400dc90 comp rx=0 tx=0).stop 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 -- 172.21.15.16:0/2921060019 shutdown_connections 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ff60c0a1230 0x7ff60c0a36f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff628102a00 0x7ff62819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff628100e60 0x7ff628196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 --2- 172.21.15.16:0/2921060019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff628102040 0x7ff6281973e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 -- 172.21.15.16:0/2921060019 >> 172.21.15.16:0/2921060019 conn(0x7ff6280fc440 msgr2=0x7ff6281054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 -- 172.21.15.16:0/2921060019 shutdown_connections 2024-04-05T23:33:02.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:02.038+0000 7ff62d200700 1 -- 172.21.15.16:0/2921060019 wait complete. 2024-04-05T23:33:02.049 INFO:tasks.cephadm:Setup complete, yielding 2024-04-05T23:33:02.050 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T23:33:02.062 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:33:02.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-05T23:33:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:33:02.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:02 smithi027 ceph-mon[78597]: pgmap v263: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-05T23:33:02.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:02 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:33:02.779 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:02 smithi016 ceph-mon[38248]: pgmap v263: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-05T23:33:02.779 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:02 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:33:02.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:02 smithi130 ceph-mon[79417]: pgmap v263: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-05T23:33:02.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:02 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:33:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.024+0000 7ffb6fa00700 1 -- 172.21.15.16:0/201046249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb68102a90 msgr2=0x7ffb681093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.024+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/201046249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb68102a90 0x7ffb681093b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000b0a0 tx=0x7ffb6000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.024+0000 7ffb6fa00700 1 -- 172.21.15.16:0/201046249 shutdown_connections 2024-04-05T23:33:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.024+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/201046249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb68100ef0 0x7ffb68101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.024+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/201046249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb68102a90 0x7ffb681093b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.024+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/201046249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb681020d0 0x7ffb68102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.024+0000 7ffb6fa00700 1 -- 172.21.15.16:0/201046249 >> 172.21.15.16:0/201046249 conn(0x7ffb680fc4e0 msgr2=0x7ffb680fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.024+0000 7ffb6fa00700 1 -- 172.21.15.16:0/201046249 shutdown_connections 2024-04-05T23:33:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.024+0000 7ffb6fa00700 1 -- 172.21.15.16:0/201046249 wait complete. 2024-04-05T23:33:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.025+0000 7ffb6fa00700 1 Processor -- start 2024-04-05T23:33:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.025+0000 7ffb6fa00700 1 -- start start 2024-04-05T23:33:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6fa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb68100ef0 0x7ffb68199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6fa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb681020d0 0x7ffb681996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6fa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb68102a90 0x7ffb681a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6fa00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681a0cb0 con 0x7ffb681020d0 2024-04-05T23:33:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6fa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681a0df0 con 0x7ffb68102a90 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6fa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681a0f80 con 0x7ffb68100ef0 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6d000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb68100ef0 0x7ffb68199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6d000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb68100ef0 0x7ffb68199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:45198/0 (socket says 172.21.15.16:45198) 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb67e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb681020d0 0x7ffb681996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6d000700 1 -- 172.21.15.16:0/2664033426 learned_addr learned my addr 172.21.15.16:0/2664033426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.026+0000 7ffb6da00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb68102a90 0x7ffb681a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb6da00700 1 -- 172.21.15.16:0/2664033426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb68100ef0 msgr2=0x7ffb68199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb6da00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb68100ef0 0x7ffb68199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb6da00700 1 -- 172.21.15.16:0/2664033426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb681020d0 msgr2=0x7ffb681996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb6da00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb681020d0 0x7ffb681996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb6da00700 1 -- 172.21.15.16:0/2664033426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb60009d00 con 0x7ffb68102a90 2024-04-05T23:33:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb6da00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb68102a90 0x7ffb681a06e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ffb60007b90 tx=0x7ffb60007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb65600700 1 -- 172.21.15.16:0/2664033426 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6000e040 con 0x7ffb68102a90 2024-04-05T23:33:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb681a1200 con 0x7ffb68102a90 2024-04-05T23:33:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb65600700 1 -- 172.21.15.16:0/2664033426 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb6000be40 con 0x7ffb68102a90 2024-04-05T23:33:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb65600700 1 -- 172.21.15.16:0/2664033426 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6001b460 con 0x7ffb68102a90 2024-04-05T23:33:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.027+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb681a1610 con 0x7ffb68102a90 2024-04-05T23:33:04.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.029+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6804ebb0 con 0x7ffb68102a90 2024-04-05T23:33:04.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.030+0000 7ffb65600700 1 -- 172.21.15.16:0/2664033426 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7ffb6001d470 con 0x7ffb68102a90 2024-04-05T23:33:04.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.032+0000 7ffb65600700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb500a1230 0x7ffb500a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:04.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.032+0000 7ffb65600700 1 -- 172.21.15.16:0/2664033426 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ffb600cdb60 con 0x7ffb68102a90 2024-04-05T23:33:04.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.032+0000 7ffb6d000700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb500a1230 0x7ffb500a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:04.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.033+0000 7ffb6d000700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb500a1230 0x7ffb500a36f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c005950 tx=0x7ffb5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:04.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.036+0000 7ffb65600700 1 -- 172.21.15.16:0/2664033426 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ffb600971d0 con 0x7ffb68102a90 2024-04-05T23:33:04.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.286+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb68074830 con 0x7ffb500a1230 2024-04-05T23:33:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.287+0000 7ffb65600700 1 -- 172.21.15.16:0/2664033426 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7ffb68074830 con 0x7ffb500a1230 2024-04-05T23:33:04.280 INFO:teuthology.orchestra.run.smithi016.stdout:Backend: cephadm 2024-04-05T23:33:04.280 INFO:teuthology.orchestra.run.smithi016.stdout:Available: Yes 2024-04-05T23:33:04.280 INFO:teuthology.orchestra.run.smithi016.stdout:Paused: No 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb500a1230 msgr2=0x7ffb500a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb500a1230 0x7ffb500a36f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c005950 tx=0x7ffb5c009260 comp rx=0 tx=0).stop 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb68102a90 msgr2=0x7ffb681a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb68102a90 0x7ffb681a06e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ffb60007b90 tx=0x7ffb60007930 comp rx=0 tx=0).stop 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 shutdown_connections 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7ffb500a1230 0x7ffb500a36f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb68102a90 0x7ffb681a06e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb68100ef0 0x7ffb68199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 --2- 172.21.15.16:0/2664033426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb681020d0 0x7ffb681996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 >> 172.21.15.16:0/2664033426 conn(0x7ffb680fc4e0 msgr2=0x7ffb68073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 shutdown_connections 2024-04-05T23:33:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:04.290+0000 7ffb6fa00700 1 -- 172.21.15.16:0/2664033426 wait complete. 2024-04-05T23:33:04.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:04 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:04 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:04 smithi016 ceph-mon[38248]: pgmap v264: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-05T23:33:04.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:04 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:04 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:04 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:04 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:04 smithi130 ceph-mon[79417]: pgmap v264: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-05T23:33:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:04 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:04 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-05T23:33:04.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:04 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:04 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:04 smithi027 ceph-mon[78597]: pgmap v264: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-05T23:33:04.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:04 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:04.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:04 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:05.069 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:33:05.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:05 smithi016 ceph-mon[38248]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:05.550 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:05.550 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:05 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:05.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:05 smithi130 ceph-mon[79417]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:05.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:05 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:05.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:05 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:05.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:05 smithi027 ceph-mon[78597]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:05.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:05 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:05.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:05 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.202+0000 7feb8c800700 1 -- 172.21.15.16:0/2920576713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb841020d0 msgr2=0x7feb84102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.202+0000 7feb8c800700 1 --2- 172.21.15.16:0/2920576713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb841020d0 0x7feb84102550 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feb74009990 tx=0x7feb74009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.202+0000 7feb8c800700 1 -- 172.21.15.16:0/2920576713 shutdown_connections 2024-04-05T23:33:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.202+0000 7feb8c800700 1 --2- 172.21.15.16:0/2920576713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb84102a90 0x7feb841093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.202+0000 7feb8c800700 1 --2- 172.21.15.16:0/2920576713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb84100ef0 0x7feb84101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.202+0000 7feb8c800700 1 --2- 172.21.15.16:0/2920576713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb841020d0 0x7feb84102550 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.202+0000 7feb8c800700 1 -- 172.21.15.16:0/2920576713 >> 172.21.15.16:0/2920576713 conn(0x7feb840fc4e0 msgr2=0x7feb840fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.202+0000 7feb8c800700 1 -- 172.21.15.16:0/2920576713 shutdown_connections 2024-04-05T23:33:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.202+0000 7feb8c800700 1 -- 172.21.15.16:0/2920576713 wait complete. 2024-04-05T23:33:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.203+0000 7feb8c800700 1 Processor -- start 2024-04-05T23:33:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.203+0000 7feb8c800700 1 -- start start 2024-04-05T23:33:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb8c800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb84100ef0 0x7feb84074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb8c800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb841020d0 0x7feb840732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb8c800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb84102a90 0x7feb84073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb8c800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb84073fb0 con 0x7feb84102a90 2024-04-05T23:33:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb8c800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb84074120 con 0x7feb841020d0 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb8c800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb841a0f20 con 0x7feb84100ef0 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb8a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb84102a90 0x7feb84073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb8a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb84102a90 0x7feb84073810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48996/0 (socket says 172.21.15.16:48996) 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb8a800700 1 -- 172.21.15.16:0/4101731452 learned_addr learned my addr 172.21.15.16:0/4101731452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb89400700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb841020d0 0x7feb840732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.204+0000 7feb89e00700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb84100ef0 0x7feb84074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb89400700 1 -- 172.21.15.16:0/4101731452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb84100ef0 msgr2=0x7feb84074c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb89400700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb84100ef0 0x7feb84074c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb89400700 1 -- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb84102a90 msgr2=0x7feb84073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb89400700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb84102a90 0x7feb84073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb89400700 1 -- 172.21.15.16:0/4101731452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb740095f0 con 0x7feb841020d0 2024-04-05T23:33:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb8a800700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb84102a90 0x7feb84073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb89400700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb841020d0 0x7feb840732d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb740098e0 tx=0x7feb7400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb7ea00700 1 -- 172.21.15.16:0/4101731452 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7401d070 con 0x7feb841020d0 2024-04-05T23:33:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb7ea00700 1 -- 172.21.15.16:0/4101731452 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb74003960 con 0x7feb841020d0 2024-04-05T23:33:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb841a1150 con 0x7feb841020d0 2024-04-05T23:33:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.205+0000 7feb7ea00700 1 -- 172.21.15.16:0/4101731452 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb74005560 con 0x7feb841020d0 2024-04-05T23:33:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.206+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb841a1560 con 0x7feb841020d0 2024-04-05T23:33:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.206+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb84108480 con 0x7feb841020d0 2024-04-05T23:33:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.208+0000 7feb7ea00700 1 -- 172.21.15.16:0/4101731452 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7feb7400fad0 con 0x7feb841020d0 2024-04-05T23:33:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.210+0000 7feb7ea00700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb6c0a11b0 0x7feb6c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.210+0000 7feb7ea00700 1 -- 172.21.15.16:0/4101731452 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feb740cde50 con 0x7feb841020d0 2024-04-05T23:33:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.210+0000 7feb89e00700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb6c0a11b0 0x7feb6c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.211+0000 7feb89e00700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb6c0a11b0 0x7feb6c0a3670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feb84100c20 tx=0x7feb7800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:06.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.212+0000 7feb7ea00700 1 -- 172.21.15.16:0/4101731452 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7feb740974c0 con 0x7feb841020d0 2024-04-05T23:33:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.434+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feb8402cdb0 con 0x7feb6c0a11b0 2024-04-05T23:33:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.449+0000 7feb7ea00700 1 -- 172.21.15.16:0/4101731452 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7feb8402cdb0 con 0x7feb6c0a11b0 2024-04-05T23:33:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T23:33:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:mds.a smithi016 running (70s) 1s ago 70s - - 18.2.2-1173-g8919605b 067bdb249792 97565fff1df1 2024-04-05T23:33:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:mds.b smithi027 running (69s) 3s ago 68s 32.7M - 18.2.2-1173-g8919605b 067bdb249792 00bd87182a78 2024-04-05T23:33:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:mds.c smithi130 running (79s) 3s ago 79s 20.6M - 18.2.2-1173-g8919605b 067bdb249792 8f86326b12d6 2024-04-05T23:33:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:mds.d smithi016 running (84s) 1s ago 84s - - 18.2.2-1173-g8919605b 067bdb249792 d979cec73102 2024-04-05T23:33:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:mds.e smithi027 running (83s) 3s ago 82s 17.9M - 18.2.2-1173-g8919605b 067bdb249792 a624edd6e0b5 2024-04-05T23:33:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:mds.f smithi130 running (81s) 3s ago 80s 19.7M - 18.2.2-1173-g8919605b 067bdb249792 c09010468ca6 2024-04-05T23:33:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:mds.g smithi016 running (73s) 1s ago 73s - - 18.2.2-1173-g8919605b 067bdb249792 b7e9cca0077b 2024-04-05T23:33:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:mds.h smithi027 running (77s) 3s ago 77s 19.8M - 18.2.2-1173-g8919605b 067bdb249792 5225dfab15ea 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mds.i smithi130 running (75s) 3s ago 75s 18.9M - 18.2.2-1173-g8919605b 067bdb249792 81b4772bd568 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mds.j smithi016 running (87s) 1s ago 87s - - 18.2.2-1173-g8919605b 067bdb249792 63f238f7a5c9 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mds.k smithi027 running (89s) 3s ago 89s 20.7M - 18.2.2-1173-g8919605b 067bdb249792 71dcf5522f8e 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mds.l smithi130 running (91s) 3s ago 91s 18.8M - 18.2.2-1173-g8919605b 067bdb249792 63d7f1c2f274 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi016 *:9283,8765 running (7m) 1s ago 7m - - 18.2.2-1173-g8919605b 067bdb249792 9e4e09a1a78c 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi027 *:8443,8765 running (5m) 3s ago 5m 457M - 18.2.2-1173-g8919605b 067bdb249792 8a6b96cd5548 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.z smithi130 *:8443,8765 running (5m) 3s ago 5m 461M - 18.2.2-1173-g8919605b 067bdb249792 843859955bac 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (7m) 1s ago 7m - 2048M 18.2.2-1173-g8919605b 067bdb249792 a904b918c113 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi027 running (5m) 3s ago 5m 64.2M 2048M 18.2.2-1173-g8919605b 067bdb249792 cfa71e27dce0 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi130 running (5m) 3s ago 5m 61.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 96e30fb978aa 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (5m) 1s ago 5m - 4318M 18.2.2-1173-g8919605b 067bdb249792 8bfa5c9ba765 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi027 running (4m) 3s ago 4m 80.4M 4278M 18.2.2-1173-g8919605b 067bdb249792 2f29d8fa109b 2024-04-05T23:33:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi130 running (4m) 3s ago 4m 77.1M 4278M 18.2.2-1173-g8919605b 067bdb249792 9b4cdfdeccc0 2024-04-05T23:33:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (4m) 1s ago 4m - 4318M 18.2.2-1173-g8919605b 067bdb249792 0103fe04d706 2024-04-05T23:33:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi027 running (4m) 3s ago 4m 60.8M 4278M 18.2.2-1173-g8919605b 067bdb249792 e9c61e86e42d 2024-04-05T23:33:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi130 running (3m) 3s ago 3m 63.9M 4278M 18.2.2-1173-g8919605b 067bdb249792 ab245a226dcc 2024-04-05T23:33:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi016 running (3m) 1s ago 3m - 4318M 18.2.2-1173-g8919605b 067bdb249792 6ad9c8c983b0 2024-04-05T23:33:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi027 running (3m) 3s ago 3m 73.6M 4278M 18.2.2-1173-g8919605b 067bdb249792 1c225896ac78 2024-04-05T23:33:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 smithi130 running (2m) 3s ago 2m 58.8M 4278M 18.2.2-1173-g8919605b 067bdb249792 d968a846d6a3 2024-04-05T23:33:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 smithi016 running (2m) 1s ago 2m - 4318M 18.2.2-1173-g8919605b 067bdb249792 144097bdaa0c 2024-04-05T23:33:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 smithi027 running (2m) 3s ago 2m 63.2M 4278M 18.2.2-1173-g8919605b 067bdb249792 a21c30c11298 2024-04-05T23:33:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 smithi130 running (107s) 3s ago 107s 53.2M 4278M 18.2.2-1173-g8919605b 067bdb249792 d51b1632171f 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb6c0a11b0 msgr2=0x7feb6c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb6c0a11b0 0x7feb6c0a3670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feb84100c20 tx=0x7feb7800b410 comp rx=0 tx=0).stop 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb841020d0 msgr2=0x7feb840732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb841020d0 0x7feb840732d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb740098e0 tx=0x7feb7400f690 comp rx=0 tx=0).stop 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 shutdown_connections 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7feb6c0a11b0 0x7feb6c0a3670 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb84102a90 0x7feb84073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb84100ef0 0x7feb84074c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 --2- 172.21.15.16:0/4101731452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb841020d0 0x7feb840732d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 >> 172.21.15.16:0/4101731452 conn(0x7feb840fc4e0 msgr2=0x7feb841055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 shutdown_connections 2024-04-05T23:33:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:06.451+0000 7feb8c800700 1 -- 172.21.15.16:0/4101731452 wait complete. 2024-04-05T23:33:06.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:06 smithi016 ceph-mon[38248]: pgmap v265: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-05T23:33:06.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:33:06.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:33:06.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:06 smithi016 ceph-mon[38248]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:06.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:06 smithi130 ceph-mon[79417]: pgmap v265: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-05T23:33:06.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:33:06.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:33:06.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:06 smithi130 ceph-mon[79417]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:06.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:06 smithi027 ceph-mon[78597]: pgmap v265: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-05T23:33:06.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:33:06.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:33:06.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:06 smithi027 ceph-mon[78597]: from='mgr.14152 172.21.15.16:0/1549690400' entity='mgr.x' 2024-04-05T23:33:06.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-05T23:33:07.183 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:33:07.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:07 smithi016 ceph-mon[38248]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:07.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:07 smithi130 ceph-mon[79417]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:07.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:07 smithi027 ceph-mon[78597]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:08.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.549+0000 7fabca000700 1 -- 172.21.15.16:0/960570634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 msgr2=0x7fabc4108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:08.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.549+0000 7fabca000700 1 --2- 172.21.15.16:0/960570634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 0x7fabc4108eb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00b0a0 tx=0x7fabbc00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.549+0000 7fabca000700 1 -- 172.21.15.16:0/960570634 shutdown_connections 2024-04-05T23:33:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.549+0000 7fabca000700 1 --2- 172.21.15.16:0/960570634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 0x7fabc4108eb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.549+0000 7fabca000700 1 --2- 172.21.15.16:0/960570634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabc4101840 0x7fabc4103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.549+0000 7fabca000700 1 --2- 172.21.15.16:0/960570634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4104170 0x7fabc4106560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.549+0000 7fabca000700 1 -- 172.21.15.16:0/960570634 >> 172.21.15.16:0/960570634 conn(0x7fabc40fb1f0 msgr2=0x7fabc40fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.549+0000 7fabca000700 1 -- 172.21.15.16:0/960570634 shutdown_connections 2024-04-05T23:33:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.549+0000 7fabca000700 1 -- 172.21.15.16:0/960570634 wait complete. 2024-04-05T23:33:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.550+0000 7fabca000700 1 Processor -- start 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.550+0000 7fabca000700 1 -- start start 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.550+0000 7fabca000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabc4101840 0x7fabc4198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.550+0000 7fabca000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4104170 0x7fabc4199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.550+0000 7fabca000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 0x7fabc41a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.550+0000 7fabca000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc41a0b00 con 0x7fabc4104170 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.550+0000 7fabca000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc41a0c40 con 0x7fabc4106aa0 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.550+0000 7fabca000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc41a0dd0 con 0x7fabc4101840 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc9600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 0x7fabc41a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc9600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 0x7fabc41a0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:34968/0 (socket says 172.21.15.16:34968) 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc9600700 1 -- 172.21.15.16:0/1324204415 learned_addr learned my addr 172.21.15.16:0/1324204415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc3e00700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4104170 0x7fabc4199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc8c00700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabc4101840 0x7fabc4198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc9600700 1 -- 172.21.15.16:0/1324204415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabc4101840 msgr2=0x7fabc4198fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc9600700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabc4101840 0x7fabc4198fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc9600700 1 -- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4104170 msgr2=0x7fabc4199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc9600700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4104170 0x7fabc4199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc9600700 1 -- 172.21.15.16:0/1324204415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabbc009d00 con 0x7fabc4106aa0 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.551+0000 7fabc3e00700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4104170 0x7fabc4199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.552+0000 7fabc9600700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 0x7fabc41a0530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0062a0 tx=0x7fabbc007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.552+0000 7fabc1600700 1 -- 172.21.15.16:0/1324204415 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc00e040 con 0x7fabc4106aa0 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.552+0000 7fabc1600700 1 -- 172.21.15.16:0/1324204415 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabbc003c00 con 0x7fabc4106aa0 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.552+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc41a1050 con 0x7fabc4106aa0 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.552+0000 7fabc1600700 1 -- 172.21.15.16:0/1324204415 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc0052a0 con 0x7fabc4106aa0 2024-04-05T23:33:08.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.552+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc41a14c0 con 0x7fabc4106aa0 2024-04-05T23:33:08.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.553+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc4108620 con 0x7fabc4106aa0 2024-04-05T23:33:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.555+0000 7fabc1600700 1 -- 172.21.15.16:0/1324204415 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fabbc01d470 con 0x7fabc4106aa0 2024-04-05T23:33:08.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.556+0000 7fabc1600700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fabac0a11b0 0x7fabac0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:08.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.556+0000 7fabc1600700 1 -- 172.21.15.16:0/1324204415 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fabbc0ce4f0 con 0x7fabc4106aa0 2024-04-05T23:33:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.557+0000 7fabc8c00700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fabac0a11b0 0x7fabac0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.557+0000 7fabc8c00700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fabac0a11b0 0x7fabac0a3670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fabb8009c60 tx=0x7fabb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.558+0000 7fabc1600700 1 -- 172.21.15.16:0/1324204415 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fabbc097b60 con 0x7fabc4106aa0 2024-04-05T23:33:08.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:08 smithi016 ceph-mon[38248]: pgmap v266: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-05T23:33:08.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:08 smithi130 ceph-mon[79417]: pgmap v266: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-05T23:33:08.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:08 smithi027 ceph-mon[78597]: pgmap v266: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-05T23:33:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.780+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fabc402cdb0 con 0x7fabac0a11b0 2024-04-05T23:33:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.785+0000 7fabc1600700 1 -- 172.21.15.16:0/1324204415 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fabc402cdb0 con 0x7fabac0a11b0 2024-04-05T23:33:08.777 INFO:teuthology.orchestra.run.smithi016.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-05T23:33:08.777 INFO:teuthology.orchestra.run.smithi016.stdout:mds.all 12/12 5s ago 96s smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi027=b;smithi027=e;smithi027=h;smithi027=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-05T23:33:08.777 INFO:teuthology.orchestra.run.smithi016.stdout:mgr 3/3 5s ago 5m smithi016=x;smithi027=y;smithi130=z;count:3 2024-04-05T23:33:08.778 INFO:teuthology.orchestra.run.smithi016.stdout:mon 3/3 5s ago 6m smithi016:172.21.15.16=a;smithi027:172.21.15.27=b;smithi130:172.21.15.130=c;count:3 2024-04-05T23:33:08.778 INFO:teuthology.orchestra.run.smithi016.stdout:osd 12 5s ago - 2024-04-05T23:33:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fabac0a11b0 msgr2=0x7fabac0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fabac0a11b0 0x7fabac0a3670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fabb8009c60 tx=0x7fabb8009260 comp rx=0 tx=0).stop 2024-04-05T23:33:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 msgr2=0x7fabc41a0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 0x7fabc41a0530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0062a0 tx=0x7fabbc007930 comp rx=0 tx=0).stop 2024-04-05T23:33:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 shutdown_connections 2024-04-05T23:33:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fabac0a11b0 0x7fabac0a3670 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc4106aa0 0x7fabc41a0530 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabc4101840 0x7fabc4198fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 --2- 172.21.15.16:0/1324204415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4104170 0x7fabc4199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 >> 172.21.15.16:0/1324204415 conn(0x7fabc40fb1f0 msgr2=0x7fabc4100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 shutdown_connections 2024-04-05T23:33:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:08.788+0000 7fabca000700 1 -- 172.21.15.16:0/1324204415 wait complete. 2024-04-05T23:33:09.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-05T23:33:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:33:10.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:10 smithi016 ceph-mon[38248]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:10.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:10 smithi016 ceph-mon[38248]: pgmap v267: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-05T23:33:10.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:10 smithi130 ceph-mon[79417]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:10.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:10 smithi130 ceph-mon[79417]: pgmap v267: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-05T23:33:10.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:10 smithi027 ceph-mon[78597]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:10.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:10 smithi027 ceph-mon[78597]: pgmap v267: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-05T23:33:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.753+0000 7f618e400700 1 -- 172.21.15.16:0/781740356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61881020d0 msgr2=0x7f6188102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.753+0000 7f618e400700 1 --2- 172.21.15.16:0/781740356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61881020d0 0x7f6188102550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6174009990 tx=0x7f6174009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.753+0000 7f618e400700 1 -- 172.21.15.16:0/781740356 shutdown_connections 2024-04-05T23:33:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.753+0000 7f618e400700 1 --2- 172.21.15.16:0/781740356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6188102a90 0x7f61881093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.753+0000 7f618e400700 1 --2- 172.21.15.16:0/781740356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6188100ef0 0x7f6188101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.753+0000 7f618e400700 1 --2- 172.21.15.16:0/781740356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61881020d0 0x7f6188102550 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.753+0000 7f618e400700 1 -- 172.21.15.16:0/781740356 >> 172.21.15.16:0/781740356 conn(0x7f61880fc4e0 msgr2=0x7f61880fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.753+0000 7f618e400700 1 -- 172.21.15.16:0/781740356 shutdown_connections 2024-04-05T23:33:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.754+0000 7f618e400700 1 -- 172.21.15.16:0/781740356 wait complete. 2024-04-05T23:33:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.754+0000 7f618e400700 1 Processor -- start 2024-04-05T23:33:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f618e400700 1 -- start start 2024-04-05T23:33:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f618e400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6188100ef0 0x7f6188074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f618e400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61881020d0 0x7f61880732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f618e400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6188102a90 0x7f6188073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f618e400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6188073fb0 con 0x7f6188100ef0 2024-04-05T23:33:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f618e400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6188074120 con 0x7f6188102a90 2024-04-05T23:33:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f618e400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61881a0eb0 con 0x7f61881020d0 2024-04-05T23:33:10.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f6187400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6188100ef0 0x7f6188074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:10.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f6187400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6188100ef0 0x7f6188074c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49022/0 (socket says 172.21.15.16:49022) 2024-04-05T23:33:10.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f6186a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61881020d0 0x7f61880732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:10.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f6187400700 1 -- 172.21.15.16:0/551004444 learned_addr learned my addr 172.21.15.16:0/551004444 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:10.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.755+0000 7f6187e00700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6188102a90 0x7f6188073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f6187400700 1 -- 172.21.15.16:0/551004444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61881020d0 msgr2=0x7f61880732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f6187400700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61881020d0 0x7f61880732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f6187400700 1 -- 172.21.15.16:0/551004444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6188102a90 msgr2=0x7f6188073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f6187400700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6188102a90 0x7f6188073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f6187400700 1 -- 172.21.15.16:0/551004444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61740095f0 con 0x7f6188100ef0 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f6187400700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6188100ef0 0x7f6188074c30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f617800e970 tx=0x7f617800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f617fe00700 1 -- 172.21.15.16:0/551004444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178009790 con 0x7f6188100ef0 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61881a1140 con 0x7f6188100ef0 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f617fe00700 1 -- 172.21.15.16:0/551004444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f617800c7e0 con 0x7f6188100ef0 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f617fe00700 1 -- 172.21.15.16:0/551004444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178003c70 con 0x7f6188100ef0 2024-04-05T23:33:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.756+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61881a1640 con 0x7f6188100ef0 2024-04-05T23:33:10.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.758+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61881083c0 con 0x7f6188100ef0 2024-04-05T23:33:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.760+0000 7f617fe00700 1 -- 172.21.15.16:0/551004444 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f617800cc50 con 0x7f6188100ef0 2024-04-05T23:33:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.761+0000 7f617fe00700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f616c0a11b0 0x7f616c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.761+0000 7f617fe00700 1 -- 172.21.15.16:0/551004444 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6178014070 con 0x7f6188100ef0 2024-04-05T23:33:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.761+0000 7f6186a00700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f616c0a11b0 0x7f616c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.762+0000 7f6186a00700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f616c0a11b0 0x7f616c0a3670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6174006010 tx=0x7f61740058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.765+0000 7f617fe00700 1 -- 172.21.15.16:0/551004444 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6178096290 con 0x7f6188100ef0 2024-04-05T23:33:10.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.985+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6188061320 con 0x7f616c0a11b0 2024-04-05T23:33:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.987+0000 7f617fe00700 1 -- 172.21.15.16:0/551004444 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f6188061320 con 0x7f616c0a11b0 2024-04-05T23:33:10.978 INFO:teuthology.orchestra.run.smithi016.stdout:HOST ADDR LABELS STATUS 2024-04-05T23:33:10.979 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 172.21.15.16 2024-04-05T23:33:10.979 INFO:teuthology.orchestra.run.smithi016.stdout:smithi027 172.21.15.27 2024-04-05T23:33:10.979 INFO:teuthology.orchestra.run.smithi016.stdout:smithi130 172.21.15.130 2024-04-05T23:33:10.979 INFO:teuthology.orchestra.run.smithi016.stdout:3 hosts in cluster 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f616c0a11b0 msgr2=0x7f616c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f616c0a11b0 0x7f616c0a3670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6174006010 tx=0x7f61740058e0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6188100ef0 msgr2=0x7f6188074c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6188100ef0 0x7f6188074c30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f617800e970 tx=0x7f617800ec80 comp rx=0 tx=0).stop 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 shutdown_connections 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f616c0a11b0 0x7f616c0a3670 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6188100ef0 0x7f6188074c30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6188102a90 0x7f6188073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 --2- 172.21.15.16:0/551004444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61881020d0 0x7f61880732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 >> 172.21.15.16:0/551004444 conn(0x7f61880fc4e0 msgr2=0x7f6188105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 shutdown_connections 2024-04-05T23:33:10.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:10.989+0000 7f618e400700 1 -- 172.21.15.16:0/551004444 wait complete. 2024-04-05T23:33:11.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-05T23:33:11.845 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:33:12.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:12 smithi016 ceph-mon[38248]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:12.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:12 smithi016 ceph-mon[38248]: pgmap v268: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-05T23:33:12.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:12 smithi130 ceph-mon[79417]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:12.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:12 smithi130 ceph-mon[79417]: pgmap v268: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-05T23:33:12.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:12 smithi027 ceph-mon[78597]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:12.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:12 smithi027 ceph-mon[78597]: pgmap v268: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-05T23:33:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.035+0000 7f2c03e00700 1 -- 172.21.15.16:0/744364260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc1020d0 msgr2=0x7f2bfc102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.035+0000 7f2c03e00700 1 --2- 172.21.15.16:0/744364260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc1020d0 0x7f2bfc102550 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009990 tx=0x7f2be8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.035+0000 7f2c03e00700 1 -- 172.21.15.16:0/744364260 shutdown_connections 2024-04-05T23:33:13.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.035+0000 7f2c03e00700 1 --2- 172.21.15.16:0/744364260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bfc102a90 0x7f2bfc1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.035+0000 7f2c03e00700 1 --2- 172.21.15.16:0/744364260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc100ef0 0x7f2bfc101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.035+0000 7f2c03e00700 1 --2- 172.21.15.16:0/744364260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc1020d0 0x7f2bfc102550 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.035+0000 7f2c03e00700 1 -- 172.21.15.16:0/744364260 >> 172.21.15.16:0/744364260 conn(0x7f2bfc0fc4e0 msgr2=0x7f2bfc0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:13.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.036+0000 7f2c03e00700 1 -- 172.21.15.16:0/744364260 shutdown_connections 2024-04-05T23:33:13.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.036+0000 7f2c03e00700 1 -- 172.21.15.16:0/744364260 wait complete. 2024-04-05T23:33:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c03e00700 1 Processor -- start 2024-04-05T23:33:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c03e00700 1 -- start start 2024-04-05T23:33:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c03e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc100ef0 0x7f2bfc196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c03e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bfc1020d0 0x7f2bfc1973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c03e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc102a90 0x7f2bfc19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c03e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc19e9c0 con 0x7f2bfc100ef0 2024-04-05T23:33:13.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c03e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc19eb00 con 0x7f2bfc102a90 2024-04-05T23:33:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c03e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc19ec90 con 0x7f2bfc1020d0 2024-04-05T23:33:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c01400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc100ef0 0x7f2bfc196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c01400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc100ef0 0x7f2bfc196e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42466/0 (socket says 172.21.15.16:42466) 2024-04-05T23:33:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c01400700 1 -- 172.21.15.16:0/4287573171 learned_addr learned my addr 172.21.15.16:0/4287573171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.037+0000 7f2c00a00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bfc1020d0 0x7f2bfc1973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2c01e00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc102a90 0x7f2bfc19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2c01400700 1 -- 172.21.15.16:0/4287573171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bfc1020d0 msgr2=0x7f2bfc1973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2c01400700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bfc1020d0 0x7f2bfc1973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2c01400700 1 -- 172.21.15.16:0/4287573171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc102a90 msgr2=0x7f2bfc19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2c01400700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc102a90 0x7f2bfc19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2c01400700 1 -- 172.21.15.16:0/4287573171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80095f0 con 0x7f2bfc100ef0 2024-04-05T23:33:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2c01400700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc100ef0 0x7f2bfc196e80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000e9a0 tx=0x7f2bf000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2bfa000700 1 -- 172.21.15.16:0/4287573171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0009790 con 0x7f2bfc100ef0 2024-04-05T23:33:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2bfa000700 1 -- 172.21.15.16:0/4287573171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2bf000c810 con 0x7f2bfc100ef0 2024-04-05T23:33:13.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bfc19ef70 con 0x7f2bfc100ef0 2024-04-05T23:33:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.038+0000 7f2bfa000700 1 -- 172.21.15.16:0/4287573171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0003c70 con 0x7f2bfc100ef0 2024-04-05T23:33:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.039+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc19f390 con 0x7f2bfc100ef0 2024-04-05T23:33:13.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.040+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bfc108430 con 0x7f2bfc100ef0 2024-04-05T23:33:13.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.042+0000 7f2bfa000700 1 -- 172.21.15.16:0/4287573171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f2bf000cc80 con 0x7f2bfc100ef0 2024-04-05T23:33:13.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.043+0000 7f2bfa000700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2be40a1230 0x7f2be40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.043+0000 7f2bfa000700 1 -- 172.21.15.16:0/4287573171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2bf0014070 con 0x7f2bfc100ef0 2024-04-05T23:33:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.043+0000 7f2c00a00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2be40a1230 0x7f2be40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:13.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.044+0000 7f2c00a00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2be40a1230 0x7f2be40a36f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2be800b2e0 tx=0x7f2be8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.047+0000 7f2bfa000700 1 -- 172.21.15.16:0/4287573171 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2bf00961c0 con 0x7f2bfc100ef0 2024-04-05T23:33:13.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.271+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 --> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bfc02ceb0 con 0x7f2be40a1230 2024-04-05T23:33:13.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.273+0000 7f2bfa000700 1 -- 172.21.15.16:0/4287573171 <== mgr.14152 v2:172.21.15.16:6800/1565223355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f2bfc02ceb0 con 0x7f2be40a1230 2024-04-05T23:33:13.265 INFO:teuthology.orchestra.run.smithi016.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-05T23:33:13.265 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-04-05T23:33:13.265 INFO:teuthology.orchestra.run.smithi016.stdout:smithi027 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001C400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-05T23:33:13.265 INFO:teuthology.orchestra.run.smithi016.stdout:smithi130 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN 372G No 109s ago Has a FileSystem, LVM detected 2024-04-05T23:33:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2be40a1230 msgr2=0x7f2be40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2be40a1230 0x7f2be40a36f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2be800b2e0 tx=0x7f2be8005fb0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc100ef0 msgr2=0x7f2bfc196e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc100ef0 0x7f2bfc196e80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000e9a0 tx=0x7f2bf000ecb0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 shutdown_connections 2024-04-05T23:33:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2be40a1230 0x7f2be40a36f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bfc100ef0 0x7f2bfc196e80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc102a90 0x7f2bfc19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 --2- 172.21.15.16:0/4287573171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bfc1020d0 0x7f2bfc1973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.275+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 >> 172.21.15.16:0/4287573171 conn(0x7f2bfc0fc4e0 msgr2=0x7f2bfc105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.276+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 shutdown_connections 2024-04-05T23:33:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:13.276+0000 7f2c03e00700 1 -- 172.21.15.16:0/4287573171 wait complete. 2024-04-05T23:33:13.656 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T23:33:13.667 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:33:13.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-05T23:33:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:33:14.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:14 smithi016 ceph-mon[38248]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:14.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:14 smithi016 ceph-mon[38248]: pgmap v269: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-05T23:33:14.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:14 smithi130 ceph-mon[79417]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:14.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:14 smithi130 ceph-mon[79417]: pgmap v269: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-05T23:33:14.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:14 smithi027 ceph-mon[78597]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:14.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:14 smithi027 ceph-mon[78597]: pgmap v269: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-05T23:33:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.236+0000 7fc3dc800700 1 -- 172.21.15.16:0/2377443156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d4102a90 msgr2=0x7fc3d41093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.236+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2377443156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d4102a90 0x7fc3d41093b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000b0a0 tx=0x7fc3d000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.236+0000 7fc3dc800700 1 -- 172.21.15.16:0/2377443156 shutdown_connections 2024-04-05T23:33:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.236+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2377443156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d4102a90 0x7fc3d41093b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.236+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2377443156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4100ef0 0x7fc3d4101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.236+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2377443156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d41020d0 0x7fc3d4102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.236+0000 7fc3dc800700 1 -- 172.21.15.16:0/2377443156 >> 172.21.15.16:0/2377443156 conn(0x7fc3d40fc4e0 msgr2=0x7fc3d40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.236+0000 7fc3dc800700 1 -- 172.21.15.16:0/2377443156 shutdown_connections 2024-04-05T23:33:15.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.236+0000 7fc3dc800700 1 -- 172.21.15.16:0/2377443156 wait complete. 2024-04-05T23:33:15.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.237+0000 7fc3dc800700 1 Processor -- start 2024-04-05T23:33:15.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3dc800700 1 -- start start 2024-04-05T23:33:15.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3dc800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4100ef0 0x7fc3d4199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3dc800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d41020d0 0x7fc3d41996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3dc800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d4102a90 0x7fc3d41a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3dc800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41a0cb0 con 0x7fc3d4102a90 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3dc800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41a0df0 con 0x7fc3d41020d0 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3dc800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41a0f80 con 0x7fc3d4100ef0 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3d9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4100ef0 0x7fc3d4199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3d9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4100ef0 0x7fc3d4199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55694/0 (socket says 172.21.15.16:55694) 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3d9e00700 1 -- 172.21.15.16:0/2864106298 learned_addr learned my addr 172.21.15.16:0/2864106298 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3da800700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d4102a90 0x7fc3d41a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.238+0000 7fc3d9400700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d41020d0 0x7fc3d41996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3d9e00700 1 -- 172.21.15.16:0/2864106298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d41020d0 msgr2=0x7fc3d41996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3d9e00700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d41020d0 0x7fc3d41996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3d9e00700 1 -- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d4102a90 msgr2=0x7fc3d41a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3d9e00700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d4102a90 0x7fc3d41a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3d9e00700 1 -- 172.21.15.16:0/2864106298 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c4009520 con 0x7fc3d4100ef0 2024-04-05T23:33:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3da800700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d4102a90 0x7fc3d41a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3d9e00700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4100ef0 0x7fc3d4199170 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800e9a0 tx=0x7fc3c800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3cea00700 1 -- 172.21.15.16:0/2864106298 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c8009790 con 0x7fc3d4100ef0 2024-04-05T23:33:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d0009d00 con 0x7fc3d4100ef0 2024-04-05T23:33:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3cea00700 1 -- 172.21.15.16:0/2864106298 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c800c810 con 0x7fc3d4100ef0 2024-04-05T23:33:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.239+0000 7fc3cea00700 1 -- 172.21.15.16:0/2864106298 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c8003c70 con 0x7fc3d4100ef0 2024-04-05T23:33:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.240+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d41a15d0 con 0x7fc3d4100ef0 2024-04-05T23:33:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.240+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d404ebb0 con 0x7fc3d4100ef0 2024-04-05T23:33:15.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.243+0000 7fc3cea00700 1 -- 172.21.15.16:0/2864106298 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fc3c800cc80 con 0x7fc3d4100ef0 2024-04-05T23:33:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.244+0000 7fc3cea00700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc3bc0a11b0 0x7fc3bc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.244+0000 7fc3cea00700 1 -- 172.21.15.16:0/2864106298 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc3c8014070 con 0x7fc3d4100ef0 2024-04-05T23:33:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.244+0000 7fc3d9400700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc3bc0a11b0 0x7fc3bc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.245+0000 7fc3d9400700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc3bc0a11b0 0x7fc3bc0a3670 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009df0 tx=0x7fc3c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.246+0000 7fc3cea00700 1 -- 172.21.15.16:0/2864106298 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc3c80962d0 con 0x7fc3d4100ef0 2024-04-05T23:33:15.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.503+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc3d410c9d0 con 0x7fc3d4100ef0 2024-04-05T23:33:15.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.504+0000 7fc3cea00700 1 -- 172.21.15.16:0/2864106298 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+2916 (secure 0 0 0) 0x7fc3c80998f0 con 0x7fc3d4100ef0 2024-04-05T23:33:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:e27 2024-04-05T23:33:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T23:33:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T23:33:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-05T23:33:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 27 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-05T23:32:04.511660+0000 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-05T23:32:11.436182+0000 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 300 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-05T23:33:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 5 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:in 0,1,2,3,4 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24566,1=24550,2=24526,3=24476,4=24472} 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data disabled 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-05T23:33:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 0 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b{0:24566} state up:active seq 3 addr [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.f{0:24520} state up:standby-replay seq 1 addr [v2:172.21.15.130:6834/3125278540,v1:172.21.15.130:6835/3125278540] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.i{1:24550} state up:active seq 6 addr [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.h{1:24518} state up:standby-replay seq 1 addr [v2:172.21.15.27:6836/541968377,v1:172.21.15.27:6837/541968377] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c{2:24526} state up:active seq 7 addr [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.e{2:24488} state up:standby-replay seq 1 addr [v2:172.21.15.27:6834/686908192,v1:172.21.15.27:6835/686908192] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d{3:24476} state up:active seq 8 addr [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.k{3:24452} state up:standby-replay seq 1 addr [v2:172.21.15.27:6832/1368666940,v1:172.21.15.27:6833/1368666940] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.l{4:24472} state up:active seq 10 addr [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a{4:14529} state up:standby-replay seq 1 addr [v2:172.21.15.16:6840/2579712996,v1:172.21.15.16:6841/2579712996] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-05T23:33:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.j{-1:14457} state up:standby seq 1 addr [v2:172.21.15.16:6834/1930674650,v1:172.21.15.16:6835/1930674650] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.g{-1:14517} state up:standby seq 1 addr [v2:172.21.15.16:6838/226142538,v1:172.21.15.16:6839/226142538] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T23:33:15.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.506+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc3bc0a11b0 msgr2=0x7fc3bc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:15.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc3bc0a11b0 0x7fc3bc0a3670 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009df0 tx=0x7fc3c4009260 comp rx=0 tx=0).stop 2024-04-05T23:33:15.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4100ef0 msgr2=0x7fc3d4199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:15.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4100ef0 0x7fc3d4199170 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800e9a0 tx=0x7fc3c800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 shutdown_connections 2024-04-05T23:33:15.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fc3bc0a11b0 0x7fc3bc0a3670 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d4102a90 0x7fc3d41a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4100ef0 0x7fc3d4199170 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 --2- 172.21.15.16:0/2864106298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d41020d0 0x7fc3d41996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:15.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 >> 172.21.15.16:0/2864106298 conn(0x7fc3d40fc4e0 msgr2=0x7fc3d4073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:15.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 shutdown_connections 2024-04-05T23:33:15.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:15.507+0000 7fc3dc800700 1 -- 172.21.15.16:0/2864106298 wait complete. 2024-04-05T23:33:15.502 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-04-05T23:33:15.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- bash -c 'ceph osd dump' 2024-04-05T23:33:16.277 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:33:16.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:16 smithi016 ceph-mon[38248]: pgmap v270: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-05T23:33:16.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:16 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2864106298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T23:33:16.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:16 smithi130 ceph-mon[79417]: pgmap v270: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-05T23:33:16.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:16 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2864106298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T23:33:16.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:16 smithi027 ceph-mon[78597]: pgmap v270: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-05T23:33:16.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:16 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2864106298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T23:33:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.442+0000 7f3d4f600700 1 -- 172.21.15.16:0/3747070212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d481020d0 msgr2=0x7f3d48102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.442+0000 7f3d4f600700 1 --2- 172.21.15.16:0/3747070212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d481020d0 0x7f3d48102550 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009990 tx=0x7f3d34009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.442+0000 7f3d4f600700 1 -- 172.21.15.16:0/3747070212 shutdown_connections 2024-04-05T23:33:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.442+0000 7f3d4f600700 1 --2- 172.21.15.16:0/3747070212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d48102a90 0x7f3d481093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.442+0000 7f3d4f600700 1 --2- 172.21.15.16:0/3747070212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d48100ef0 0x7f3d48101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.442+0000 7f3d4f600700 1 --2- 172.21.15.16:0/3747070212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d481020d0 0x7f3d48102550 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.442+0000 7f3d4f600700 1 -- 172.21.15.16:0/3747070212 >> 172.21.15.16:0/3747070212 conn(0x7f3d480fc4e0 msgr2=0x7f3d480fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.442+0000 7f3d4f600700 1 -- 172.21.15.16:0/3747070212 shutdown_connections 2024-04-05T23:33:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.443+0000 7f3d4f600700 1 -- 172.21.15.16:0/3747070212 wait complete. 2024-04-05T23:33:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.443+0000 7f3d4f600700 1 Processor -- start 2024-04-05T23:33:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4f600700 1 -- start start 2024-04-05T23:33:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4f600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d48100ef0 0x7f3d48074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4f600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d481020d0 0x7f3d480732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4f600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d48102a90 0x7f3d48073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4f600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d48073fb0 con 0x7f3d48100ef0 2024-04-05T23:33:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4f600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d48074120 con 0x7f3d48102a90 2024-04-05T23:33:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4f600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d481a0f20 con 0x7f3d481020d0 2024-04-05T23:33:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d48100ef0 0x7f3d48074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d48100ef0 0x7f3d48074c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42484/0 (socket says 172.21.15.16:42484) 2024-04-05T23:33:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4cc00700 1 -- 172.21.15.16:0/1670591806 learned_addr learned my addr 172.21.15.16:0/1670591806 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d4d600700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d48102a90 0x7f3d48073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.444+0000 7f3d47e00700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d481020d0 0x7f3d480732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d4cc00700 1 -- 172.21.15.16:0/1670591806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d481020d0 msgr2=0x7f3d480732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d4cc00700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d481020d0 0x7f3d480732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d4cc00700 1 -- 172.21.15.16:0/1670591806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d48102a90 msgr2=0x7f3d48073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d4cc00700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d48102a90 0x7f3d48073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d4cc00700 1 -- 172.21.15.16:0/1670591806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d340095f0 con 0x7f3d48100ef0 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d4cc00700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d48100ef0 0x7f3d48074c30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00e970 tx=0x7f3d3c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d45600700 1 -- 172.21.15.16:0/1670591806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c009790 con 0x7f3d48100ef0 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d481a11b0 con 0x7f3d48100ef0 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d45600700 1 -- 172.21.15.16:0/1670591806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d3c00c7e0 con 0x7f3d48100ef0 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d45600700 1 -- 172.21.15.16:0/1670591806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c003c10 con 0x7f3d48100ef0 2024-04-05T23:33:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.445+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d481a16b0 con 0x7f3d48100ef0 2024-04-05T23:33:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.446+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d48108480 con 0x7f3d48100ef0 2024-04-05T23:33:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.448+0000 7f3d45600700 1 -- 172.21.15.16:0/1670591806 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f3d3c00cc50 con 0x7f3d48100ef0 2024-04-05T23:33:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.450+0000 7f3d45600700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3d300a11b0 0x7f3d300a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.450+0000 7f3d45600700 1 -- 172.21.15.16:0/1670591806 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3d3c014070 con 0x7f3d48100ef0 2024-04-05T23:33:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.450+0000 7f3d47e00700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3d300a11b0 0x7f3d300a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.450+0000 7f3d47e00700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3d300a11b0 0x7f3d300a3670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400b2e0 tx=0x7f3d34005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.454+0000 7f3d45600700 1 -- 172.21.15.16:0/1670591806 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3d3c0961f0 con 0x7f3d48100ef0 2024-04-05T23:33:17.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.667+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f3d4810c7a0 con 0x7f3d48100ef0 2024-04-05T23:33:17.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.668+0000 7f3d45600700 1 -- 172.21.15.16:0/1670591806 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v81) v1 ==== 56+0+5273 (secure 0 0 0) 0x7f3d3c099810 con 0x7f3d48100ef0 2024-04-05T23:33:17.660 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 81 2024-04-05T23:33:17.660 INFO:teuthology.orchestra.run.smithi016.stdout:fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:33:17.660 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-05T23:25:35.231049+0000 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-05T23:32:38.369016+0000 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 26 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client luminous 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release reef 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-05T23:33:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 12 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 8 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6802/2345860669,v1:172.21.15.16:6803/2345860669] [v2:172.21.15.16:6804/2345860669,v1:172.21.15.16:6805/2345860669] exists,up 5fd073ed-664a-4b48-912a-3a2b1708c3a2 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 12 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6800/1426362237,v1:172.21.15.27:6801/1426362237] [v2:172.21.15.27:6802/1426362237,v1:172.21.15.27:6803/1426362237] exists,up 20981429-c950-4821-abbc-06a9d380c25d 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 17 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6800/3222862878,v1:172.21.15.130:6801/3222862878] [v2:172.21.15.130:6802/3222862878,v1:172.21.15.130:6803/3222862878] exists,up 794cd3d0-bb75-41a5-9ea9-e946ddea82b7 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 23 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6810/1156660397,v1:172.21.15.16:6811/1156660397] [v2:172.21.15.16:6812/1156660397,v1:172.21.15.16:6813/1156660397] exists,up 3ef270b8-eac6-4738-82f3-7aa26c536370 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 30 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6808/1591043365,v1:172.21.15.27:6809/1591043365] [v2:172.21.15.27:6810/1591043365,v1:172.21.15.27:6811/1591043365] exists,up e348d76d-044d-4a3a-8420-8e283130eb95 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 35 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6808/1595822481,v1:172.21.15.130:6809/1595822481] [v2:172.21.15.130:6810/1595822481,v1:172.21.15.130:6811/1595822481] exists,up 9d921f6c-abd3-47b6-a898-5f5bf0afd5d2 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 up in weight 1 up_from 42 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6818/2729842886,v1:172.21.15.16:6819/2729842886] [v2:172.21.15.16:6820/2729842886,v1:172.21.15.16:6821/2729842886] exists,up 6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 up in weight 1 up_from 48 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6816/3123372166,v1:172.21.15.27:6817/3123372166] [v2:172.21.15.27:6818/3123372166,v1:172.21.15.27:6819/3123372166] exists,up bc642843-7b4c-48ea-9c1c-74345e3da8df 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 up in weight 1 up_from 55 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6816/3206875138,v1:172.21.15.130:6817/3206875138] [v2:172.21.15.130:6818/3206875138,v1:172.21.15.130:6819/3206875138] exists,up 584a112e-7c86-49ea-b1b6-30733d0f1e31 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 up in weight 1 up_from 62 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6826/2518129600,v1:172.21.15.16:6827/2518129600] [v2:172.21.15.16:6828/2518129600,v1:172.21.15.16:6829/2518129600] exists,up 14bd3f6d-201c-45c2-b82a-aa358d1a0f28 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 up in weight 1 up_from 67 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6824/15853906,v1:172.21.15.27:6825/15853906] [v2:172.21.15.27:6826/15853906,v1:172.21.15.27:6827/15853906] exists,up 366e5c3c-48b4-4718-9128-021c930445ee 2024-04-05T23:33:17.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 up in weight 1 up_from 74 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6824/4254452589,v1:172.21.15.130:6825/4254452589] [v2:172.21.15.130:6826/4254452589,v1:172.21.15.130:6827/4254452589] exists,up 2f8660f8-f57e-4f24-9b2e-bab7a440fc2b 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.13 [2,11] 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.16 [2,4] 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.1c [8,11] 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.1 [4,9] 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.14 [4,6] 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.17 [4,8] 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.1b [5,9] 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.29 [3,9] 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/749035377 expires 2024-04-06T23:26:37.227308+0000 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/2429779560 expires 2024-04-06T23:26:37.227308+0000 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/2429779560 expires 2024-04-06T23:26:37.227308+0000 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3668421167 expires 2024-04-06T23:26:37.227308+0000 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/1106968003 expires 2024-04-06T23:26:11.600470+0000 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/1141331215 expires 2024-04-06T23:26:37.227308+0000 2024-04-05T23:33:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2307977786 expires 2024-04-06T23:26:11.600470+0000 2024-04-05T23:33:17.664 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/1696925921 expires 2024-04-06T23:26:11.600470+0000 2024-04-05T23:33:17.664 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/3856233973 expires 2024-04-06T23:26:11.600470+0000 2024-04-05T23:33:17.664 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/3856233973 expires 2024-04-06T23:26:11.600470+0000 2024-04-05T23:33:17.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3d300a11b0 msgr2=0x7f3d300a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3d300a11b0 0x7f3d300a3670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400b2e0 tx=0x7f3d34005fb0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d48100ef0 msgr2=0x7f3d48074c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d48100ef0 0x7f3d48074c30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00e970 tx=0x7f3d3c00ec80 comp rx=0 tx=0).stop 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 shutdown_connections 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3d300a11b0 0x7f3d300a3670 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d48100ef0 0x7f3d48074c30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d48102a90 0x7f3d48073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 --2- 172.21.15.16:0/1670591806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d481020d0 0x7f3d480732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 >> 172.21.15.16:0/1670591806 conn(0x7f3d480fc4e0 msgr2=0x7f3d481055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 shutdown_connections 2024-04-05T23:33:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:17.671+0000 7f3d4f600700 1 -- 172.21.15.16:0/1670591806 wait complete. 2024-04-05T23:33:18.075 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-05T23:33:18.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:33:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.298+0000 7f3f64e00700 1 -- 172.21.15.16:0/2442056657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f60074f20 msgr2=0x7f3f60073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.298+0000 7f3f64e00700 1 --2- 172.21.15.16:0/2442056657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f60074f20 0x7f3f60073370 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009990 tx=0x7f3f50009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.298+0000 7f3f64e00700 1 -- 172.21.15.16:0/2442056657 shutdown_connections 2024-04-05T23:33:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.298+0000 7f3f64e00700 1 --2- 172.21.15.16:0/2442056657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f60108c70 0x7f3f6010b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.298+0000 7f3f64e00700 1 --2- 172.21.15.16:0/2442056657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f60074f20 0x7f3f60073370 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.298+0000 7f3f64e00700 1 --2- 172.21.15.16:0/2442056657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f60073940 0x7f3f60073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.298+0000 7f3f64e00700 1 -- 172.21.15.16:0/2442056657 >> 172.21.15.16:0/2442056657 conn(0x7f3f600fc320 msgr2=0x7f3f600fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.298+0000 7f3f64e00700 1 -- 172.21.15.16:0/2442056657 shutdown_connections 2024-04-05T23:33:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.298+0000 7f3f64e00700 1 -- 172.21.15.16:0/2442056657 wait complete. 2024-04-05T23:33:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.299+0000 7f3f64e00700 1 Processor -- start 2024-04-05T23:33:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f64e00700 1 -- start start 2024-04-05T23:33:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f64e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f60073940 0x7f3f60198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f64e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f60074f20 0x7f3f601994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f64e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f60108c70 0x7f3f601a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f64e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601a0aa0 con 0x7f3f60108c70 2024-04-05T23:33:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f64e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601a0be0 con 0x7f3f60074f20 2024-04-05T23:33:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f64e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601a0d70 con 0x7f3f60073940 2024-04-05T23:33:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f5d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f60074f20 0x7f3f601994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f5de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f60073940 0x7f3f60198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.301+0000 7f3f5d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f60074f20 0x7f3f601994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42994/0 (socket says 172.21.15.16:42994) 2024-04-05T23:33:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.301+0000 7f3f5d400700 1 -- 172.21.15.16:0/3907029111 learned_addr learned my addr 172.21.15.16:0/3907029111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.300+0000 7f3f5e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f60108c70 0x7f3f601a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.301+0000 7f3f5de00700 1 -- 172.21.15.16:0/3907029111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f60074f20 msgr2=0x7f3f601994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.301+0000 7f3f5de00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f60074f20 0x7f3f601994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.301+0000 7f3f5de00700 1 -- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f60108c70 msgr2=0x7f3f601a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.301+0000 7f3f5de00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f60108c70 0x7f3f601a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.301+0000 7f3f5de00700 1 -- 172.21.15.16:0/3907029111 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f500095f0 con 0x7f3f60073940 2024-04-05T23:33:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.301+0000 7f3f5e800700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f60108c70 0x7f3f601a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.301+0000 7f3f5de00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f60073940 0x7f3f60198f70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3f50000c00 tx=0x7f3f50009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.302+0000 7f3f56a00700 1 -- 172.21.15.16:0/3907029111 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5001d070 con 0x7f3f60073940 2024-04-05T23:33:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.302+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f601a0ff0 con 0x7f3f60073940 2024-04-05T23:33:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.302+0000 7f3f56a00700 1 -- 172.21.15.16:0/3907029111 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f50021410 con 0x7f3f60073940 2024-04-05T23:33:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.302+0000 7f3f56a00700 1 -- 172.21.15.16:0/3907029111 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50017650 con 0x7f3f60073940 2024-04-05T23:33:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.302+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f601a13b0 con 0x7f3f60073940 2024-04-05T23:33:18.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.303+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f60107a30 con 0x7f3f60073940 2024-04-05T23:33:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.305+0000 7f3f56a00700 1 -- 172.21.15.16:0/3907029111 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f3f50021580 con 0x7f3f60073940 2024-04-05T23:33:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.306+0000 7f3f56a00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3f440a11b0 0x7f3f440a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.306+0000 7f3f56a00700 1 -- 172.21.15.16:0/3907029111 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3f500ce420 con 0x7f3f60073940 2024-04-05T23:33:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.306+0000 7f3f5d400700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3f440a11b0 0x7f3f440a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.307+0000 7f3f5d400700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3f440a11b0 0x7f3f440a3670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c005950 tx=0x7f3f4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.309+0000 7f3f56a00700 1 -- 172.21.15.16:0/3907029111 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3f50097a90 con 0x7f3f60073940 2024-04-05T23:33:18.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.530+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f6010e7f0 con 0x7f3f60073940 2024-04-05T23:33:18.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.532+0000 7f3f56a00700 1 -- 172.21.15.16:0/3907029111 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22519 (secure 0 0 0) 0x7f3f50005740 con 0x7f3f60073940 2024-04-05T23:33:18.524 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:18.524 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:38.369016+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:33:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3f440a11b0 msgr2=0x7f3f440a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3f440a11b0 0x7f3f440a3670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c005950 tx=0x7f3f4c009260 comp rx=0 tx=0).stop 2024-04-05T23:33:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f60073940 msgr2=0x7f3f60198f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f60073940 0x7f3f60198f70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3f50000c00 tx=0x7f3f50009580 comp rx=0 tx=0).stop 2024-04-05T23:33:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 shutdown_connections 2024-04-05T23:33:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f3f440a11b0 0x7f3f440a3670 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f60108c70 0x7f3f601a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f60073940 0x7f3f60198f70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 --2- 172.21.15.16:0/3907029111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f60074f20 0x7f3f601994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.534+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 >> 172.21.15.16:0/3907029111 conn(0x7f3f600fc320 msgr2=0x7f3f60104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.535+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 shutdown_connections 2024-04-05T23:33:18.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.535+0000 7f3f64e00700 1 -- 172.21.15.16:0/3907029111 wait complete. 2024-04-05T23:33:18.547 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-05T23:32:02.641673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-05T23:32:03.972521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:33:18.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:33:18.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:18 smithi016 ceph-mon[38248]: pgmap v271: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-05T23:33:18.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:18 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1670591806' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-05T23:33:18.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:18 smithi130 ceph-mon[79417]: pgmap v271: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-05T23:33:18.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:18 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1670591806' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-05T23:33:18.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:18 smithi027 ceph-mon[78597]: pgmap v271: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-05T23:33:18.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:18 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1670591806' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-05T23:33:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.758+0000 7efc78000700 1 -- 172.21.15.16:0/2990122224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc70108d80 msgr2=0x7efc7010b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.758+0000 7efc78000700 1 --2- 172.21.15.16:0/2990122224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc70108d80 0x7efc7010b210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efc6800b0a0 tx=0x7efc6800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.758+0000 7efc78000700 1 -- 172.21.15.16:0/2990122224 shutdown_connections 2024-04-05T23:33:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.758+0000 7efc78000700 1 --2- 172.21.15.16:0/2990122224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc70108d80 0x7efc7010b210 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.758+0000 7efc78000700 1 --2- 172.21.15.16:0/2990122224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc70074f20 0x7efc70073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.758+0000 7efc78000700 1 --2- 172.21.15.16:0/2990122224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc70073940 0x7efc70073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.758+0000 7efc78000700 1 -- 172.21.15.16:0/2990122224 >> 172.21.15.16:0/2990122224 conn(0x7efc700fc430 msgr2=0x7efc700fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.758+0000 7efc78000700 1 -- 172.21.15.16:0/2990122224 shutdown_connections 2024-04-05T23:33:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.759+0000 7efc78000700 1 -- 172.21.15.16:0/2990122224 wait complete. 2024-04-05T23:33:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.759+0000 7efc78000700 1 Processor -- start 2024-04-05T23:33:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc78000700 1 -- start start 2024-04-05T23:33:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc78000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc70073940 0x7efc70199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc78000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc70074f20 0x7efc701996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc78000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc70108d80 0x7efc701a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc78000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc701a0c90 con 0x7efc70108d80 2024-04-05T23:33:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc78000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc701a0dd0 con 0x7efc70073940 2024-04-05T23:33:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc78000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc701a0f60 con 0x7efc70074f20 2024-04-05T23:33:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc74c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc70074f20 0x7efc701996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc76000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc70108d80 0x7efc701a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc75600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc70073940 0x7efc70199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc74c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc70074f20 0x7efc701996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55746/0 (socket says 172.21.15.16:55746) 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc74c00700 1 -- 172.21.15.16:0/860680206 learned_addr learned my addr 172.21.15.16:0/860680206 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.760+0000 7efc76000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc70108d80 0x7efc701a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42522/0 (socket says 172.21.15.16:42522) 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc76000700 1 -- 172.21.15.16:0/860680206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc70074f20 msgr2=0x7efc701996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc76000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc70074f20 0x7efc701996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc76000700 1 -- 172.21.15.16:0/860680206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc70073940 msgr2=0x7efc70199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc76000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc70073940 0x7efc70199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc76000700 1 -- 172.21.15.16:0/860680206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c009520 con 0x7efc70108d80 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc75600700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc70073940 0x7efc70199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc76000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc70108d80 0x7efc701a06c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efc68007b90 tx=0x7efc68007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc6e000700 1 -- 172.21.15.16:0/860680206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6800e040 con 0x7efc70108d80 2024-04-05T23:33:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc6e000700 1 -- 172.21.15.16:0/860680206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc6800be40 con 0x7efc70108d80 2024-04-05T23:33:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc68009d00 con 0x7efc70108d80 2024-04-05T23:33:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc6e000700 1 -- 172.21.15.16:0/860680206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6801b460 con 0x7efc70108d80 2024-04-05T23:33:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.761+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc701a1470 con 0x7efc70108d80 2024-04-05T23:33:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.762+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc70107b40 con 0x7efc70108d80 2024-04-05T23:33:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.764+0000 7efc6e000700 1 -- 172.21.15.16:0/860680206 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7efc6801d470 con 0x7efc70108d80 2024-04-05T23:33:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.765+0000 7efc6e000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efc580a11b0 0x7efc580a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.766+0000 7efc6e000700 1 -- 172.21.15.16:0/860680206 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efc680cd9d0 con 0x7efc70108d80 2024-04-05T23:33:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.766+0000 7efc75600700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efc580a11b0 0x7efc580a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.766+0000 7efc75600700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efc580a11b0 0x7efc580a3670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efc64011490 tx=0x7efc64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:18.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.769+0000 7efc6e000700 1 -- 172.21.15.16:0/860680206 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efc68097040 con 0x7efc70108d80 2024-04-05T23:33:18.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.986+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc7010ea10 con 0x7efc70108d80 2024-04-05T23:33:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.987+0000 7efc6e000700 1 -- 172.21.15.16:0/860680206 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc6809a660 con 0x7efc70108d80 2024-04-05T23:33:18.979 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.989+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efc580a11b0 msgr2=0x7efc580a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.989+0000 7efc78000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efc580a11b0 0x7efc580a3670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efc64011490 tx=0x7efc64009260 comp rx=0 tx=0).stop 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.989+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc70108d80 msgr2=0x7efc701a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.989+0000 7efc78000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc70108d80 0x7efc701a06c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efc68007b90 tx=0x7efc68007930 comp rx=0 tx=0).stop 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.989+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 shutdown_connections 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.989+0000 7efc78000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efc580a11b0 0x7efc580a3670 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.989+0000 7efc78000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc70108d80 0x7efc701a06c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.990+0000 7efc78000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc70073940 0x7efc70199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.990+0000 7efc78000700 1 --2- 172.21.15.16:0/860680206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc70074f20 0x7efc701996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.990+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 >> 172.21.15.16:0/860680206 conn(0x7efc700fc430 msgr2=0x7efc70104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.990+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 shutdown_connections 2024-04-05T23:33:18.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:18.990+0000 7efc78000700 1 -- 172.21.15.16:0/860680206 wait complete. 2024-04-05T23:33:19.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-05T23:33:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.252+0000 7f697fa00700 1 -- 172.21.15.16:0/898887767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 msgr2=0x7f69781024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.252+0000 7f697fa00700 1 --2- 172.21.15.16:0/898887767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 0x7f69781024c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6964009990 tx=0x7f6964009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.252+0000 7f697fa00700 1 -- 172.21.15.16:0/898887767 shutdown_connections 2024-04-05T23:33:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.252+0000 7f697fa00700 1 --2- 172.21.15.16:0/898887767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6978102a00 0x7f6978109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.252+0000 7f697fa00700 1 --2- 172.21.15.16:0/898887767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978100e60 0x7f6978101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.252+0000 7f697fa00700 1 --2- 172.21.15.16:0/898887767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 0x7f69781024c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.252+0000 7f697fa00700 1 -- 172.21.15.16:0/898887767 >> 172.21.15.16:0/898887767 conn(0x7f69780fc440 msgr2=0x7f69780fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.252+0000 7f697fa00700 1 -- 172.21.15.16:0/898887767 shutdown_connections 2024-04-05T23:33:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.252+0000 7f697fa00700 1 -- 172.21.15.16:0/898887767 wait complete. 2024-04-05T23:33:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.253+0000 7f697fa00700 1 Processor -- start 2024-04-05T23:33:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.253+0000 7f697fa00700 1 -- start start 2024-04-05T23:33:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f697fa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978100e60 0x7f69781990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f697fa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 0x7f69781995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f697fa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6978102a00 0x7f69781a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f697fa00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69781a0bd0 con 0x7f6978102040 2024-04-05T23:33:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f697fa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69781a0d10 con 0x7f6978102a00 2024-04-05T23:33:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f697fa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69781a0ea0 con 0x7f6978100e60 2024-04-05T23:33:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f6977e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 0x7f69781995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f6977e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 0x7f69781995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42534/0 (socket says 172.21.15.16:42534) 2024-04-05T23:33:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f6977e00700 1 -- 172.21.15.16:0/526467939 learned_addr learned my addr 172.21.15.16:0/526467939 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f697da00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6978102a00 0x7f69781a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f697d000700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978100e60 0x7f69781990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f6977e00700 1 -- 172.21.15.16:0/526467939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978100e60 msgr2=0x7f69781990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f6977e00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978100e60 0x7f69781990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f6977e00700 1 -- 172.21.15.16:0/526467939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6978102a00 msgr2=0x7f69781a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f6977e00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6978102a00 0x7f69781a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.254+0000 7f6977e00700 1 -- 172.21.15.16:0/526467939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c009520 con 0x7f6978102040 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.255+0000 7f6977e00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 0x7f69781995e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6964000c00 tx=0x7f6964009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.255+0000 7f6975600700 1 -- 172.21.15.16:0/526467939 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696401d070 con 0x7f6978102040 2024-04-05T23:33:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.255+0000 7f6975600700 1 -- 172.21.15.16:0/526467939 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6964021410 con 0x7f6978102040 2024-04-05T23:33:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.255+0000 7f6975600700 1 -- 172.21.15.16:0/526467939 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964017650 con 0x7f6978102040 2024-04-05T23:33:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.255+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69640095f0 con 0x7f6978102040 2024-04-05T23:33:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.255+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69781a13b0 con 0x7f6978102040 2024-04-05T23:33:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.256+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6978067c30 con 0x7f6978102040 2024-04-05T23:33:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.258+0000 7f6975600700 1 -- 172.21.15.16:0/526467939 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f6964021580 con 0x7f6978102040 2024-04-05T23:33:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.259+0000 7f6975600700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f69600a1230 0x7f69600a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.259+0000 7f6975600700 1 -- 172.21.15.16:0/526467939 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f69640cdcc0 con 0x7f6978102040 2024-04-05T23:33:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.259+0000 7f697d000700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f69600a1230 0x7f69600a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.260+0000 7f697d000700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f69600a1230 0x7f69600a36f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6978101bb0 tx=0x7f696c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.263+0000 7f6975600700 1 -- 172.21.15.16:0/526467939 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f69640d4080 con 0x7f6978102040 2024-04-05T23:33:19.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.485+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f697810c940 con 0x7f6978102040 2024-04-05T23:33:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.485+0000 7f6975600700 1 -- 172.21.15.16:0/526467939 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6964097330 con 0x7f6978102040 2024-04-05T23:33:19.477 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-05T23:33:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f69600a1230 msgr2=0x7f69600a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f69600a1230 0x7f69600a36f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6978101bb0 tx=0x7f696c009260 comp rx=0 tx=0).stop 2024-04-05T23:33:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 msgr2=0x7f69781995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 0x7f69781995e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6964000c00 tx=0x7f6964009580 comp rx=0 tx=0).stop 2024-04-05T23:33:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 shutdown_connections 2024-04-05T23:33:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f69600a1230 0x7f69600a36f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6978102a00 0x7f69781a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978100e60 0x7f69781990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 --2- 172.21.15.16:0/526467939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978102040 0x7f69781995e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 >> 172.21.15.16:0/526467939 conn(0x7f69780fc440 msgr2=0x7f69781078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:19.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.487+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 shutdown_connections 2024-04-05T23:33:19.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.488+0000 7f697fa00700 1 -- 172.21.15.16:0/526467939 wait complete. 2024-04-05T23:33:19.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-05T23:33:19.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:19 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3907029111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:19.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:19 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/860680206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:19.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:19 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3907029111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:19.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:19 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/860680206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.707+0000 7fe91a800700 1 -- 172.21.15.16:0/607948837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102040 msgr2=0x7fe9141024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.707+0000 7fe91a800700 1 --2- 172.21.15.16:0/607948837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102040 0x7fe9141024c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe900009990 tx=0x7fe900009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.707+0000 7fe91a800700 1 -- 172.21.15.16:0/607948837 shutdown_connections 2024-04-05T23:33:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.707+0000 7fe91a800700 1 --2- 172.21.15.16:0/607948837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe914102a00 0x7fe914109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.707+0000 7fe91a800700 1 --2- 172.21.15.16:0/607948837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe914100e60 0x7fe914101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.707+0000 7fe91a800700 1 --2- 172.21.15.16:0/607948837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914102040 0x7fe9141024c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.707+0000 7fe91a800700 1 -- 172.21.15.16:0/607948837 >> 172.21.15.16:0/607948837 conn(0x7fe9140fc440 msgr2=0x7fe9140fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.708+0000 7fe91a800700 1 -- 172.21.15.16:0/607948837 shutdown_connections 2024-04-05T23:33:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.708+0000 7fe91a800700 1 -- 172.21.15.16:0/607948837 wait complete. 2024-04-05T23:33:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe91a800700 1 Processor -- start 2024-04-05T23:33:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe91a800700 1 -- start start 2024-04-05T23:33:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe91a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914100e60 0x7fe91419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe91a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe914102040 0x7fe91419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe91a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe914102a00 0x7fe9141a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe91a800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9141a4ff0 con 0x7fe914100e60 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe91a800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9141a5130 con 0x7fe914102040 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe91a800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9141a52c0 con 0x7fe914102a00 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe913400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914100e60 0x7fe91419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe913400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914100e60 0x7fe91419d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42558/0 (socket says 172.21.15.16:42558) 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe913400700 1 -- 172.21.15.16:0/4118207594 learned_addr learned my addr 172.21.15.16:0/4118207594 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe912a00700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe914102040 0x7fe91419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.709+0000 7fe913e00700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe914102a00 0x7fe9141a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe913400700 1 -- 172.21.15.16:0/4118207594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe914102a00 msgr2=0x7fe9141a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe913400700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe914102a00 0x7fe9141a4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe913400700 1 -- 172.21.15.16:0/4118207594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe914102040 msgr2=0x7fe91419da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe913400700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe914102040 0x7fe91419da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe913400700 1 -- 172.21.15.16:0/4118207594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9000095f0 con 0x7fe914100e60 2024-04-05T23:33:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe913400700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914100e60 0x7fe91419d4c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe90400d980 tx=0x7fe90400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe90be00700 1 -- 172.21.15.16:0/4118207594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90400f890 con 0x7fe914100e60 2024-04-05T23:33:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe90be00700 1 -- 172.21.15.16:0/4118207594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe904010460 con 0x7fe914100e60 2024-04-05T23:33:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9141a55a0 con 0x7fe914100e60 2024-04-05T23:33:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe90be00700 1 -- 172.21.15.16:0/4118207594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9040195c0 con 0x7fe914100e60 2024-04-05T23:33:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.710+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9141a5ab0 con 0x7fe914100e60 2024-04-05T23:33:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.711+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe914067c30 con 0x7fe914100e60 2024-04-05T23:33:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.713+0000 7fe90be00700 1 -- 172.21.15.16:0/4118207594 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fe9040105d0 con 0x7fe914100e60 2024-04-05T23:33:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.714+0000 7fe90be00700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe8f80a11b0 0x7fe8f80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.715+0000 7fe90be00700 1 -- 172.21.15.16:0/4118207594 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe9040cc870 con 0x7fe914100e60 2024-04-05T23:33:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.715+0000 7fe912a00700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe8f80a11b0 0x7fe8f80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.715+0000 7fe912a00700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe8f80a11b0 0x7fe8f80a3670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe90000b680 tx=0x7fe900005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.719+0000 7fe90be00700 1 -- 172.21.15.16:0/4118207594 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe904095e50 con 0x7fe914100e60 2024-04-05T23:33:19.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:19 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3907029111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:19.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:19 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/860680206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.933+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe9141a5e30 con 0x7fe914100e60 2024-04-05T23:33:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.933+0000 7fe90be00700 1 -- 172.21.15.16:0/4118207594 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe904099470 con 0x7fe914100e60 2024-04-05T23:33:19.926 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-05T23:33:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe8f80a11b0 msgr2=0x7fe8f80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe8f80a11b0 0x7fe8f80a3670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe90000b680 tx=0x7fe900005fd0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914100e60 msgr2=0x7fe91419d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914100e60 0x7fe91419d4c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe90400d980 tx=0x7fe90400dc90 comp rx=0 tx=0).stop 2024-04-05T23:33:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 shutdown_connections 2024-04-05T23:33:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fe8f80a11b0 0x7fe8f80a3670 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe914100e60 0x7fe91419d4c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe914102a00 0x7fe9141a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 --2- 172.21.15.16:0/4118207594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe914102040 0x7fe91419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:19.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 >> 172.21.15.16:0/4118207594 conn(0x7fe9140fc440 msgr2=0x7fe914073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:19.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 shutdown_connections 2024-04-05T23:33:19.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:19.936+0000 7fe91a800700 1 -- 172.21.15.16:0/4118207594 wait complete. 2024-04-05T23:33:19.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:33:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.154+0000 7fbd12200700 1 -- 172.21.15.16:0/556739953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 msgr2=0x7fbd0c103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.154+0000 7fbd12200700 1 --2- 172.21.15.16:0/556739953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 0x7fbd0c103c70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009990 tx=0x7fbcfc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.154+0000 7fbd12200700 1 -- 172.21.15.16:0/556739953 shutdown_connections 2024-04-05T23:33:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.154+0000 7fbd12200700 1 --2- 172.21.15.16:0/556739953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 0x7fbd0c103c70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.154+0000 7fbd12200700 1 --2- 172.21.15.16:0/556739953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c106ae0 0x7fbd0c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.154+0000 7fbd12200700 1 --2- 172.21.15.16:0/556739953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd0c1041b0 0x7fbd0c1065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.154+0000 7fbd12200700 1 -- 172.21.15.16:0/556739953 >> 172.21.15.16:0/556739953 conn(0x7fbd0c0fb230 msgr2=0x7fbd0c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.154+0000 7fbd12200700 1 -- 172.21.15.16:0/556739953 shutdown_connections 2024-04-05T23:33:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.154+0000 7fbd12200700 1 -- 172.21.15.16:0/556739953 wait complete. 2024-04-05T23:33:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.156+0000 7fbd12200700 1 Processor -- start 2024-04-05T23:33:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.156+0000 7fbd12200700 1 -- start start 2024-04-05T23:33:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.156+0000 7fbd12200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 0x7fbd0c198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.156+0000 7fbd12200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c1041b0 0x7fbd0c199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.156+0000 7fbd12200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd0c106ae0 0x7fbd0c1a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.156+0000 7fbd12200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c1a0a70 con 0x7fbd0c101880 2024-04-05T23:33:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.156+0000 7fbd12200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c1a0bb0 con 0x7fbd0c1041b0 2024-04-05T23:33:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.156+0000 7fbd12200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c1a0d40 con 0x7fbd0c106ae0 2024-04-05T23:33:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 0x7fbd0c198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 0x7fbd0c198f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42572/0 (socket says 172.21.15.16:42572) 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0b400700 1 -- 172.21.15.16:0/2140513995 learned_addr learned my addr 172.21.15.16:0/2140513995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0aa00700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c1041b0 0x7fbd0c199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0be00700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd0c106ae0 0x7fbd0c1a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0b400700 1 -- 172.21.15.16:0/2140513995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd0c106ae0 msgr2=0x7fbd0c1a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0b400700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd0c106ae0 0x7fbd0c1a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0b400700 1 -- 172.21.15.16:0/2140513995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c1041b0 msgr2=0x7fbd0c199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0b400700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c1041b0 0x7fbd0c199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0b400700 1 -- 172.21.15.16:0/2140513995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04009c40 con 0x7fbd0c101880 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0aa00700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c1041b0 0x7fbd0c199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.157+0000 7fbd0b400700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 0x7fbd0c198f40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc005910 tx=0x7fbcfc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:20.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.158+0000 7fbd03e00700 1 -- 172.21.15.16:0/2140513995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc01d070 con 0x7fbd0c101880 2024-04-05T23:33:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.158+0000 7fbd03e00700 1 -- 172.21.15.16:0/2140513995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcfc021410 con 0x7fbd0c101880 2024-04-05T23:33:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.158+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc0095f0 con 0x7fbd0c101880 2024-04-05T23:33:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.158+0000 7fbd03e00700 1 -- 172.21.15.16:0/2140513995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc017650 con 0x7fbd0c101880 2024-04-05T23:33:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.158+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c1a11f0 con 0x7fbd0c101880 2024-04-05T23:33:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.159+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0c108680 con 0x7fbd0c101880 2024-04-05T23:33:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.161+0000 7fbd03e00700 1 -- 172.21.15.16:0/2140513995 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fbcfc021580 con 0x7fbd0c101880 2024-04-05T23:33:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.163+0000 7fbd03e00700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbcf00a1230 0x7fbcf00a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.163+0000 7fbd03e00700 1 -- 172.21.15.16:0/2140513995 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbcfc0cdf10 con 0x7fbd0c101880 2024-04-05T23:33:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.163+0000 7fbd0aa00700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbcf00a1230 0x7fbcf00a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.163+0000 7fbd0aa00700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbcf00a1230 0x7fbcf00a36f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009cb0 tx=0x7fbcf8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.167+0000 7fbd03e00700 1 -- 172.21.15.16:0/2140513995 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbcfc097580 con 0x7fbd0c101880 2024-04-05T23:33:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.429+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd0c067b50 con 0x7fbd0c101880 2024-04-05T23:33:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.430+0000 7fbd03e00700 1 -- 172.21.15.16:0/2140513995 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10902 (secure 0 0 0) 0x7fbcfc09aba0 con 0x7fbd0c101880 2024-04-05T23:33:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:20.422 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:32:11.436182+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:33:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbcf00a1230 msgr2=0x7fbcf00a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbcf00a1230 0x7fbcf00a36f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009cb0 tx=0x7fbcf8009260 comp rx=0 tx=0).stop 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 msgr2=0x7fbd0c198f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 0x7fbd0c198f40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc005910 tx=0x7fbcfc009580 comp rx=0 tx=0).stop 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 shutdown_connections 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbcf00a1230 0x7fbcf00a36f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd0c101880 0x7fbd0c198f40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd0c106ae0 0x7fbd0c1a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 --2- 172.21.15.16:0/2140513995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c1041b0 0x7fbd0c199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 >> 172.21.15.16:0/2140513995 conn(0x7fbd0c0fb230 msgr2=0x7fbd0c100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 shutdown_connections 2024-04-05T23:33:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.433+0000 7fbd12200700 1 -- 172.21.15.16:0/2140513995 wait complete. 2024-04-05T23:33:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-04-05T23:33:20.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:33:20.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:20 smithi016 ceph-mon[38248]: pgmap v272: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-05T23:33:20.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:20 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/526467939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:20.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:20 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4118207594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:20.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:20 smithi130 ceph-mon[79417]: pgmap v272: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-05T23:33:20.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:20 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/526467939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:20.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:20 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4118207594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:20.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.651+0000 7fbe3e200700 1 -- 172.21.15.16:0/2499881957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe38104150 msgr2=0x7fbe38106540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.651+0000 7fbe3e200700 1 --2- 172.21.15.16:0/2499881957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe38104150 0x7fbe38106540 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbe240098f0 tx=0x7fbe24009c00 comp rx=0 tx=0).stop 2024-04-05T23:33:20.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.651+0000 7fbe3e200700 1 -- 172.21.15.16:0/2499881957 shutdown_connections 2024-04-05T23:33:20.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.651+0000 7fbe3e200700 1 --2- 172.21.15.16:0/2499881957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe38106a80 0x7fbe38108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.651+0000 7fbe3e200700 1 --2- 172.21.15.16:0/2499881957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe38101820 0x7fbe38103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.651+0000 7fbe3e200700 1 --2- 172.21.15.16:0/2499881957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe38104150 0x7fbe38106540 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.651+0000 7fbe3e200700 1 -- 172.21.15.16:0/2499881957 >> 172.21.15.16:0/2499881957 conn(0x7fbe380fb230 msgr2=0x7fbe380fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.651+0000 7fbe3e200700 1 -- 172.21.15.16:0/2499881957 shutdown_connections 2024-04-05T23:33:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.651+0000 7fbe3e200700 1 -- 172.21.15.16:0/2499881957 wait complete. 2024-04-05T23:33:20.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.652+0000 7fbe3e200700 1 Processor -- start 2024-04-05T23:33:20.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.652+0000 7fbe3e200700 1 -- start start 2024-04-05T23:33:20.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe3e200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe38101820 0x7fbe38198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:20.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe3e200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe38104150 0x7fbe38199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:20.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe3e200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe38106a80 0x7fbe381a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:20.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe3e200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe381a0b10 con 0x7fbe38106a80 2024-04-05T23:33:20.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe3e200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe381a0c50 con 0x7fbe38104150 2024-04-05T23:33:20.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe3e200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe381a0de0 con 0x7fbe38101820 2024-04-05T23:33:20.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe37e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe38106a80 0x7fbe381a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:20.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe37e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe38106a80 0x7fbe381a0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42586/0 (socket says 172.21.15.16:42586) 2024-04-05T23:33:20.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe37e00700 1 -- 172.21.15.16:0/647937124 learned_addr learned my addr 172.21.15.16:0/647937124 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:20.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe36a00700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe38104150 0x7fbe38199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:20.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe37400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe38101820 0x7fbe38198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:20.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe37e00700 1 -- 172.21.15.16:0/647937124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe38101820 msgr2=0x7fbe38198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe37e00700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe38101820 0x7fbe38198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe37e00700 1 -- 172.21.15.16:0/647937124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe38104150 msgr2=0x7fbe38199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe37e00700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe38104150 0x7fbe38199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe37e00700 1 -- 172.21.15.16:0/647937124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30009d00 con 0x7fbe38106a80 2024-04-05T23:33:20.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.653+0000 7fbe36a00700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe38104150 0x7fbe38199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:20.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.654+0000 7fbe37e00700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe38106a80 0x7fbe381a0540 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000e3c0 tx=0x7fbe3000e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:20.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.654+0000 7fbe2fe00700 1 -- 172.21.15.16:0/647937124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe300090d0 con 0x7fbe38106a80 2024-04-05T23:33:20.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.654+0000 7fbe2fe00700 1 -- 172.21.15.16:0/647937124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe3000f040 con 0x7fbe38106a80 2024-04-05T23:33:20.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.654+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe24009520 con 0x7fbe38106a80 2024-04-05T23:33:20.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.654+0000 7fbe2fe00700 1 -- 172.21.15.16:0/647937124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30007c20 con 0x7fbe38106a80 2024-04-05T23:33:20.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.654+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe381a1440 con 0x7fbe38106a80 2024-04-05T23:33:20.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.655+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe38108650 con 0x7fbe38106a80 2024-04-05T23:33:20.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.658+0000 7fbe2fe00700 1 -- 172.21.15.16:0/647937124 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fbe30003a00 con 0x7fbe38106a80 2024-04-05T23:33:20.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.659+0000 7fbe2fe00700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe1c0a11b0 0x7fbe1c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:20.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.660+0000 7fbe2fe00700 1 -- 172.21.15.16:0/647937124 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbe300cccc0 con 0x7fbe38106a80 2024-04-05T23:33:20.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.660+0000 7fbe37400700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe1c0a11b0 0x7fbe1c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:20.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.660+0000 7fbe37400700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe1c0a11b0 0x7fbe1c0a3670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009c60 tx=0x7fbe28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:20.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.662+0000 7fbe2fe00700 1 -- 172.21.15.16:0/647937124 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbe30096390 con 0x7fbe38106a80 2024-04-05T23:33:20.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:20 smithi027 ceph-mon[78597]: pgmap v272: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-05T23:33:20.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:20 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/526467939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:20.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:20 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4118207594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.882+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe3810c5e0 con 0x7fbe38106a80 2024-04-05T23:33:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.884+0000 7fbe2fe00700 1 -- 172.21.15.16:0/647937124 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22519 (secure 0 0 0) 0x7fbe300999b0 con 0x7fbe38106a80 2024-04-05T23:33:20.876 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:20.876 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:38.369016+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:33:20.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe1c0a11b0 msgr2=0x7fbe1c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe1c0a11b0 0x7fbe1c0a3670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009c60 tx=0x7fbe28009260 comp rx=0 tx=0).stop 2024-04-05T23:33:20.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe38106a80 msgr2=0x7fbe381a0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:20.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe38106a80 0x7fbe381a0540 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000e3c0 tx=0x7fbe3000e6d0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 shutdown_connections 2024-04-05T23:33:20.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbe1c0a11b0 0x7fbe1c0a3670 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe38106a80 0x7fbe381a0540 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe38101820 0x7fbe38198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 --2- 172.21.15.16:0/647937124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe38104150 0x7fbe38199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 >> 172.21.15.16:0/647937124 conn(0x7fbe380fb230 msgr2=0x7fbe38100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 shutdown_connections 2024-04-05T23:33:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:20.886+0000 7fbe3e200700 1 -- 172.21.15.16:0/647937124 wait complete. 2024-04-05T23:33:20.898 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-05T23:32:02.641673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-05T23:32:03.972521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:33:20.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:33:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.109+0000 7f7c0b200700 1 -- 172.21.15.16:0/2207380182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c04102040 msgr2=0x7f7c041024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.109+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2207380182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c04102040 0x7f7c041024c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009990 tx=0x7f7bec009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.109+0000 7f7c0b200700 1 -- 172.21.15.16:0/2207380182 shutdown_connections 2024-04-05T23:33:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.109+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2207380182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c04102a00 0x7f7c04109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.109+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2207380182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c04100e60 0x7f7c04101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.109+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2207380182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c04102040 0x7f7c041024c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.109+0000 7f7c0b200700 1 -- 172.21.15.16:0/2207380182 >> 172.21.15.16:0/2207380182 conn(0x7f7c040fc440 msgr2=0x7f7c040fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.110+0000 7f7c0b200700 1 -- 172.21.15.16:0/2207380182 shutdown_connections 2024-04-05T23:33:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.110+0000 7f7c0b200700 1 -- 172.21.15.16:0/2207380182 wait complete. 2024-04-05T23:33:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.111+0000 7f7c0b200700 1 Processor -- start 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.111+0000 7f7c0b200700 1 -- start start 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c0b200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c04100e60 0x7f7c04074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c0b200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c04102040 0x7f7c040732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c0b200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c04102a00 0x7f7c04073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c0b200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04073fb0 con 0x7f7c04102a00 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c0b200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04074120 con 0x7f7c04102040 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c0b200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c041a0f00 con 0x7f7c04100e60 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c09200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c04102a00 0x7f7c04073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c09200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c04102a00 0x7f7c04073810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42612/0 (socket says 172.21.15.16:42612) 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c09200700 1 -- 172.21.15.16:0/2348078284 learned_addr learned my addr 172.21.15.16:0/2348078284 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7bfbe00700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c04100e60 0x7f7c04074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c03e00700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c04102040 0x7f7c040732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c09200700 1 -- 172.21.15.16:0/2348078284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c04100e60 msgr2=0x7f7c04074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c09200700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c04100e60 0x7f7c04074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c09200700 1 -- 172.21.15.16:0/2348078284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c04102040 msgr2=0x7f7c040732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c09200700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c04102040 0x7f7c040732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c09200700 1 -- 172.21.15.16:0/2348078284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec0095f0 con 0x7f7c04102a00 2024-04-05T23:33:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.112+0000 7f7c03e00700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c04102040 0x7f7c040732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.113+0000 7f7c09200700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c04102a00 0x7f7c04073810 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00ef30 tx=0x7f7bfc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.113+0000 7f7c01600700 1 -- 172.21.15.16:0/2348078284 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc018070 con 0x7f7c04102a00 2024-04-05T23:33:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.113+0000 7f7c01600700 1 -- 172.21.15.16:0/2348078284 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bfc00f040 con 0x7f7c04102a00 2024-04-05T23:33:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.113+0000 7f7c01600700 1 -- 172.21.15.16:0/2348078284 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc00cca0 con 0x7f7c04102a00 2024-04-05T23:33:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.113+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c041a1130 con 0x7f7c04102a00 2024-04-05T23:33:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.113+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c041a15f0 con 0x7f7c04102a00 2024-04-05T23:33:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.114+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c04108350 con 0x7f7c04102a00 2024-04-05T23:33:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.116+0000 7f7c01600700 1 -- 172.21.15.16:0/2348078284 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f7bfc003a00 con 0x7f7c04102a00 2024-04-05T23:33:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.117+0000 7f7c01600700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7be80a11b0 0x7f7be80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.117+0000 7f7c01600700 1 -- 172.21.15.16:0/2348078284 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7bfc0cc5a0 con 0x7f7c04102a00 2024-04-05T23:33:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.117+0000 7f7bfbe00700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7be80a11b0 0x7f7be80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.118+0000 7f7bfbe00700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7be80a11b0 0x7f7be80a3670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7c04101bb0 tx=0x7f7bf4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.121+0000 7f7c01600700 1 -- 172.21.15.16:0/2348078284 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7bfc095c10 con 0x7f7c04102a00 2024-04-05T23:33:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.339+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c04067be0 con 0x7f7c04102a00 2024-04-05T23:33:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.339+0000 7f7c01600700 1 -- 172.21.15.16:0/2348078284 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7bfc099230 con 0x7f7c04102a00 2024-04-05T23:33:21.332 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:33:21.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7be80a11b0 msgr2=0x7f7be80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7be80a11b0 0x7f7be80a3670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7c04101bb0 tx=0x7f7bf4009260 comp rx=0 tx=0).stop 2024-04-05T23:33:21.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c04102a00 msgr2=0x7f7c04073810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c04102a00 0x7f7c04073810 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00ef30 tx=0x7f7bfc00c3a0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 shutdown_connections 2024-04-05T23:33:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7be80a11b0 0x7f7be80a3670 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c04102a00 0x7f7c04073810 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c04100e60 0x7f7c04074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 --2- 172.21.15.16:0/2348078284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c04102040 0x7f7c040732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 >> 172.21.15.16:0/2348078284 conn(0x7f7c040fc440 msgr2=0x7f7c04105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 shutdown_connections 2024-04-05T23:33:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.342+0000 7f7c0b200700 1 -- 172.21.15.16:0/2348078284 wait complete. 2024-04-05T23:33:21.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-05T23:33:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.566+0000 7f7bd5000700 1 -- 172.21.15.16:0/2112740436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 msgr2=0x7f7bd0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.566+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2112740436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 0x7f7bd0101280 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0009990 tx=0x7f7bc0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.566+0000 7f7bd5000700 1 -- 172.21.15.16:0/2112740436 shutdown_connections 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.566+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2112740436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 0x7f7bd0101280 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.566+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2112740436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd0102a00 0x7f7bd0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.566+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2112740436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd0102040 0x7f7bd01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.566+0000 7f7bd5000700 1 -- 172.21.15.16:0/2112740436 >> 172.21.15.16:0/2112740436 conn(0x7f7bd00fc440 msgr2=0x7f7bd00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.566+0000 7f7bd5000700 1 -- 172.21.15.16:0/2112740436 shutdown_connections 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.566+0000 7f7bd5000700 1 -- 172.21.15.16:0/2112740436 wait complete. 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bd5000700 1 Processor -- start 2024-04-05T23:33:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bd5000700 1 -- start start 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bd5000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 0x7f7bd01990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bd5000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd0102040 0x7f7bd01995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bd5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd0102a00 0x7f7bd01a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bd5000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd01a0bd0 con 0x7f7bd0100e60 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bd5000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd01a0d10 con 0x7f7bd0102a00 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bd5000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd01a0ea0 con 0x7f7bd0102040 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bc5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 0x7f7bd01990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bc5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 0x7f7bd01990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42636/0 (socket says 172.21.15.16:42636) 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.567+0000 7f7bc5e00700 1 -- 172.21.15.16:0/2844606333 learned_addr learned my addr 172.21.15.16:0/2844606333 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bce800700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd0102a00 0x7f7bd01a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bcde00700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd0102040 0x7f7bd01995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bc5e00700 1 -- 172.21.15.16:0/2844606333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd0102040 msgr2=0x7f7bd01995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bc5e00700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd0102040 0x7f7bd01995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bc5e00700 1 -- 172.21.15.16:0/2844606333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd0102a00 msgr2=0x7f7bd01a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bc5e00700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd0102a00 0x7f7bd01a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bc5e00700 1 -- 172.21.15.16:0/2844606333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc00095f0 con 0x7f7bd0100e60 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bc5e00700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 0x7f7bd01990a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0000c00 tx=0x7f7bc0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bc7400700 1 -- 172.21.15.16:0/2844606333 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc001d070 con 0x7f7bd0100e60 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bc7400700 1 -- 172.21.15.16:0/2844606333 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bc0021410 con 0x7f7bd0100e60 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd01a1120 con 0x7f7bd0100e60 2024-04-05T23:33:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.568+0000 7f7bc7400700 1 -- 172.21.15.16:0/2844606333 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0017650 con 0x7f7bd0100e60 2024-04-05T23:33:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.569+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd01a1540 con 0x7f7bd0100e60 2024-04-05T23:33:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.569+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd0067c30 con 0x7f7bd0100e60 2024-04-05T23:33:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.572+0000 7f7bc7400700 1 -- 172.21.15.16:0/2844606333 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f7bc0021580 con 0x7f7bd0100e60 2024-04-05T23:33:21.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.573+0000 7f7bc7400700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7bb40a11b0 0x7f7bb40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:21.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.573+0000 7f7bc7400700 1 -- 172.21.15.16:0/2844606333 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7bc00ce3e0 con 0x7f7bd0100e60 2024-04-05T23:33:21.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.573+0000 7f7bcde00700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7bb40a11b0 0x7f7bb40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:21.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.574+0000 7f7bcde00700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7bb40a11b0 0x7f7bb40a3670 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0100b80 tx=0x7f7bb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.575+0000 7f7bc7400700 1 -- 172.21.15.16:0/2844606333 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7bc0097ab0 con 0x7f7bd0100e60 2024-04-05T23:33:21.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:21 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2140513995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:21.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:21 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/647937124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:21.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.790+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7bd010c8b0 con 0x7f7bd0100e60 2024-04-05T23:33:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.791+0000 7f7bc7400700 1 -- 172.21.15.16:0/2844606333 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7bc0005740 con 0x7f7bd0100e60 2024-04-05T23:33:21.783 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-05T23:33:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7bb40a11b0 msgr2=0x7f7bb40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7bb40a11b0 0x7f7bb40a3670 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0100b80 tx=0x7f7bb8009260 comp rx=0 tx=0).stop 2024-04-05T23:33:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 msgr2=0x7f7bd01990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 0x7f7bd01990a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0000c00 tx=0x7f7bc0009580 comp rx=0 tx=0).stop 2024-04-05T23:33:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 shutdown_connections 2024-04-05T23:33:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7bb40a11b0 0x7f7bb40a3670 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd0100e60 0x7f7bd01990a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bd0102a00 0x7f7bd01a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 --2- 172.21.15.16:0/2844606333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd0102040 0x7f7bd01995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 >> 172.21.15.16:0/2844606333 conn(0x7f7bd00fc440 msgr2=0x7f7bd0073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 shutdown_connections 2024-04-05T23:33:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:21.793+0000 7f7bd5000700 1 -- 172.21.15.16:0/2844606333 wait complete. 2024-04-05T23:33:21.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:21 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2140513995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:21.802 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:21 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/647937124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:21.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-05T23:33:21.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:21 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2140513995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:21.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:21 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/647937124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.015+0000 7f0fda600700 1 -- 172.21.15.16:0/2787609071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4102040 msgr2=0x7f0fd41024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.015+0000 7f0fda600700 1 --2- 172.21.15.16:0/2787609071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4102040 0x7f0fd41024c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009990 tx=0x7f0fbc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.016+0000 7f0fda600700 1 -- 172.21.15.16:0/2787609071 shutdown_connections 2024-04-05T23:33:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.016+0000 7f0fda600700 1 --2- 172.21.15.16:0/2787609071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd4102a00 0x7f0fd4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.016+0000 7f0fda600700 1 --2- 172.21.15.16:0/2787609071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fd4100e60 0x7f0fd4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.016+0000 7f0fda600700 1 --2- 172.21.15.16:0/2787609071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4102040 0x7f0fd41024c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.016+0000 7f0fda600700 1 -- 172.21.15.16:0/2787609071 >> 172.21.15.16:0/2787609071 conn(0x7f0fd40fc440 msgr2=0x7f0fd40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.016+0000 7f0fda600700 1 -- 172.21.15.16:0/2787609071 shutdown_connections 2024-04-05T23:33:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.016+0000 7f0fda600700 1 -- 172.21.15.16:0/2787609071 wait complete. 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fda600700 1 Processor -- start 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fda600700 1 -- start start 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fda600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fd4100e60 0x7f0fd4199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fda600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4102040 0x7f0fd4199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fda600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd4102a00 0x7f0fd41a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fda600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41a0b70 con 0x7f0fd4102a00 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fda600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41a0cb0 con 0x7f0fd4102040 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fda600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41a0e40 con 0x7f0fd4100e60 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fd3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd4102a00 0x7f0fd41a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fd3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4102040 0x7f0fd4199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fd3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd4102a00 0x7f0fd41a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60406/0 (socket says 172.21.15.16:60406) 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.017+0000 7f0fd3e00700 1 -- 172.21.15.16:0/1531414236 learned_addr learned my addr 172.21.15.16:0/1531414236 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fcb400700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fd4100e60 0x7f0fd4199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd3e00700 1 -- 172.21.15.16:0/1531414236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fd4100e60 msgr2=0x7f0fd4199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd3e00700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fd4100e60 0x7f0fd4199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd3e00700 1 -- 172.21.15.16:0/1531414236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4102040 msgr2=0x7f0fd4199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd3e00700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4102040 0x7f0fd4199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd3e00700 1 -- 172.21.15.16:0/1531414236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc0095f0 con 0x7f0fd4102a00 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd3400700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4102040 0x7f0fd4199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd3e00700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd4102a00 0x7f0fd41a05a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00ef30 tx=0x7f0fcc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd0c00700 1 -- 172.21.15.16:0/1531414236 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc018070 con 0x7f0fd4102a00 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd41a10c0 con 0x7f0fd4102a00 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd0c00700 1 -- 172.21.15.16:0/1531414236 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fcc00f040 con 0x7f0fd4102a00 2024-04-05T23:33:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd41a1620 con 0x7f0fd4102a00 2024-04-05T23:33:22.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.018+0000 7f0fd0c00700 1 -- 172.21.15.16:0/1531414236 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc00cca0 con 0x7f0fd4102a00 2024-04-05T23:33:22.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.019+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd4067c30 con 0x7f0fd4102a00 2024-04-05T23:33:22.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.023+0000 7f0fd0c00700 1 -- 172.21.15.16:0/1531414236 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f0fcc003a00 con 0x7f0fd4102a00 2024-04-05T23:33:22.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.024+0000 7f0fd0c00700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0fb80a11b0 0x7f0fb80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.024+0000 7f0fd0c00700 1 -- 172.21.15.16:0/1531414236 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0fcc0ccf20 con 0x7f0fd4102a00 2024-04-05T23:33:22.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.025+0000 7f0fcb400700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0fb80a11b0 0x7f0fb80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.025+0000 7f0fcb400700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0fb80a11b0 0x7f0fb80a3670 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4101bb0 tx=0x7f0fc4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:22.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.027+0000 7f0fd0c00700 1 -- 172.21.15.16:0/1531414236 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0fcc096590 con 0x7f0fd4102a00 2024-04-05T23:33:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.243+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0fd41a1940 con 0x7f0fd4102a00 2024-04-05T23:33:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.243+0000 7f0fd0c00700 1 -- 172.21.15.16:0/1531414236 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0fcc094e60 con 0x7f0fd4102a00 2024-04-05T23:33:22.235 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-05T23:33:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.245+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0fb80a11b0 msgr2=0x7f0fb80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.245+0000 7f0fda600700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0fb80a11b0 0x7f0fb80a3670 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4101bb0 tx=0x7f0fc4009260 comp rx=0 tx=0).stop 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.245+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd4102a00 msgr2=0x7f0fd41a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.245+0000 7f0fda600700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd4102a00 0x7f0fd41a05a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00ef30 tx=0x7f0fcc00c3a0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.245+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 shutdown_connections 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.245+0000 7f0fda600700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f0fb80a11b0 0x7f0fb80a3670 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.245+0000 7f0fda600700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd4102a00 0x7f0fd41a05a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.245+0000 7f0fda600700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fd4100e60 0x7f0fd4199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.245+0000 7f0fda600700 1 --2- 172.21.15.16:0/1531414236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4102040 0x7f0fd4199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.246+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 >> 172.21.15.16:0/1531414236 conn(0x7f0fd40fc440 msgr2=0x7f0fd4073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.246+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 shutdown_connections 2024-04-05T23:33:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.246+0000 7f0fda600700 1 -- 172.21.15.16:0/1531414236 wait complete. 2024-04-05T23:33:22.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:33:22.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.463+0000 7f2ca3200700 1 -- 172.21.15.16:0/408766920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 msgr2=0x7f2c9c1011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.463+0000 7f2ca3200700 1 --2- 172.21.15.16:0/408766920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 0x7f2c9c1011d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009990 tx=0x7f2c8c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.463+0000 7f2ca3200700 1 -- 172.21.15.16:0/408766920 shutdown_connections 2024-04-05T23:33:22.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.463+0000 7f2ca3200700 1 --2- 172.21.15.16:0/408766920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c102940 0x7f2c9c1091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.463+0000 7f2ca3200700 1 --2- 172.21.15.16:0/408766920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 0x7f2c9c1011d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.463+0000 7f2ca3200700 1 --2- 172.21.15.16:0/408766920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c101fa0 0x7f2c9c102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.463+0000 7f2ca3200700 1 -- 172.21.15.16:0/408766920 >> 172.21.15.16:0/408766920 conn(0x7f2c9c0fc320 msgr2=0x7f2c9c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:22.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.463+0000 7f2ca3200700 1 -- 172.21.15.16:0/408766920 shutdown_connections 2024-04-05T23:33:22.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.463+0000 7f2ca3200700 1 -- 172.21.15.16:0/408766920 wait complete. 2024-04-05T23:33:22.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.464+0000 7f2ca3200700 1 Processor -- start 2024-04-05T23:33:22.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.464+0000 7f2ca3200700 1 -- start start 2024-04-05T23:33:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2ca3200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 0x7f2c9c074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2ca3200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c101fa0 0x7f2c9c0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2ca3200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c102940 0x7f2c9c073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2ca3200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c073fb0 con 0x7f2c9c101fa0 2024-04-05T23:33:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2ca3200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c074120 con 0x7f2c9c102940 2024-04-05T23:33:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2ca3200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c1a0dc0 con 0x7f2c9c100db0 2024-04-05T23:33:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2c9b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c101fa0 0x7f2c9c0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2c9be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 0x7f2c9c074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2c9be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 0x7f2c9c074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43382/0 (socket says 172.21.15.16:43382) 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2c9be00700 1 -- 172.21.15.16:0/3245373421 learned_addr learned my addr 172.21.15.16:0/3245373421 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.465+0000 7f2ca1200700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c102940 0x7f2c9c073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c9be00700 1 -- 172.21.15.16:0/3245373421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c102940 msgr2=0x7f2c9c073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c9be00700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c102940 0x7f2c9c073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c9be00700 1 -- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c101fa0 msgr2=0x7f2c9c0732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c9be00700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c101fa0 0x7f2c9c0732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c9be00700 1 -- 172.21.15.16:0/3245373421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c0095f0 con 0x7f2c9c100db0 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c9b400700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c101fa0 0x7f2c9c0732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c9be00700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 0x7f2c9c074c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0098e0 tx=0x7f2c8c005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c98c00700 1 -- 172.21.15.16:0/3245373421 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c01d070 con 0x7f2c9c100db0 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c98c00700 1 -- 172.21.15.16:0/3245373421 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c8c022470 con 0x7f2c9c100db0 2024-04-05T23:33:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9c1a0ff0 con 0x7f2c9c100db0 2024-04-05T23:33:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2c98c00700 1 -- 172.21.15.16:0/3245373421 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c017650 con 0x7f2c9c100db0 2024-04-05T23:33:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.466+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c1a1360 con 0x7f2c9c100db0 2024-04-05T23:33:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.467+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c108220 con 0x7f2c9c100db0 2024-04-05T23:33:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.469+0000 7f2c98c00700 1 -- 172.21.15.16:0/3245373421 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f2c8c0225e0 con 0x7f2c9c100db0 2024-04-05T23:33:22.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.471+0000 7f2c98c00700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2c800a11b0 0x7f2c800a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.471+0000 7f2c98c00700 1 -- 172.21.15.16:0/3245373421 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2c8c0cdf10 con 0x7f2c9c100db0 2024-04-05T23:33:22.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.471+0000 7f2c9b400700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2c800a11b0 0x7f2c800a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.471+0000 7f2c9b400700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2c800a11b0 0x7f2c800a3670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2c88011490 tx=0x7f2c88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:22.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.473+0000 7f2c98c00700 1 -- 172.21.15.16:0/3245373421 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2c8c097610 con 0x7f2c9c100db0 2024-04-05T23:33:22.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:22 smithi016 ceph-mon[38248]: pgmap v273: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-05T23:33:22.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:22 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2348078284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:22.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:22 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2844606333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:22.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:22 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1531414236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:22.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:22 smithi027 ceph-mon[78597]: pgmap v273: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-05T23:33:22.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:22 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2348078284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:22.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:22 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2844606333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:22.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:22 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1531414236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.739+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c9c067b50 con 0x7f2c9c100db0 2024-04-05T23:33:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.740+0000 7f2c98c00700 1 -- 172.21.15.16:0/3245373421 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10902 (secure 0 0 0) 0x7f2c8c09ac30 con 0x7f2c9c100db0 2024-04-05T23:33:22.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:22.732 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:32:11.436182+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:33:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2c800a11b0 msgr2=0x7f2c800a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2c800a11b0 0x7f2c800a3670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2c88011490 tx=0x7f2c88009260 comp rx=0 tx=0).stop 2024-04-05T23:33:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 msgr2=0x7f2c9c074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 0x7f2c9c074c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0098e0 tx=0x7f2c8c005720 comp rx=0 tx=0).stop 2024-04-05T23:33:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 shutdown_connections 2024-04-05T23:33:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f2c800a11b0 0x7f2c800a3670 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c102940 0x7f2c9c073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c9c100db0 0x7f2c9c074c40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 --2- 172.21.15.16:0/3245373421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c101fa0 0x7f2c9c0732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 >> 172.21.15.16:0/3245373421 conn(0x7f2c9c0fc320 msgr2=0x7f2c9c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 shutdown_connections 2024-04-05T23:33:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.742+0000 7f2ca3200700 1 -- 172.21.15.16:0/3245373421 wait complete. 2024-04-05T23:33:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-04-05T23:33:22.753 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:33:22.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:22 smithi130 ceph-mon[79417]: pgmap v273: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-05T23:33:22.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:22 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2348078284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:22.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:22 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2844606333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:22.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:22 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1531414236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:22.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.959+0000 7f7ce4e00700 1 -- 172.21.15.16:0/526715865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0073940 msgr2=0x7f7ce0073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.959+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/526715865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0073940 0x7f7ce0073dc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009990 tx=0x7f7ccc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.959+0000 7f7ce4e00700 1 -- 172.21.15.16:0/526715865 shutdown_connections 2024-04-05T23:33:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.959+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/526715865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce010d380 0x7f7ce010f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.959+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/526715865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce0074f20 0x7f7ce0073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.959+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/526715865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0073940 0x7f7ce0073dc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.959+0000 7f7ce4e00700 1 -- 172.21.15.16:0/526715865 >> 172.21.15.16:0/526715865 conn(0x7f7ce00fc320 msgr2=0x7f7ce00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.959+0000 7f7ce4e00700 1 -- 172.21.15.16:0/526715865 shutdown_connections 2024-04-05T23:33:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.959+0000 7f7ce4e00700 1 -- 172.21.15.16:0/526715865 wait complete. 2024-04-05T23:33:22.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.960+0000 7f7ce4e00700 1 Processor -- start 2024-04-05T23:33:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.960+0000 7f7ce4e00700 1 -- start start 2024-04-05T23:33:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.960+0000 7f7ce4e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce0073940 0x7f7ce019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.960+0000 7f7ce4e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0074f20 0x7f7ce019d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.960+0000 7f7ce4e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce010d380 0x7f7ce01a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.960+0000 7f7ce4e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce01a4ea0 con 0x7f7ce0074f20 2024-04-05T23:33:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.960+0000 7f7ce4e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce01a4fe0 con 0x7f7ce010d380 2024-04-05T23:33:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.960+0000 7f7ce4e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce01a5170 con 0x7f7ce0073940 2024-04-05T23:33:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdd400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0074f20 0x7f7ce019d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdd400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0074f20 0x7f7ce019d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60442/0 (socket says 172.21.15.16:60442) 2024-04-05T23:33:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdd400700 1 -- 172.21.15.16:0/2936361737 learned_addr learned my addr 172.21.15.16:0/2936361737 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdde00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce0073940 0x7f7ce019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cde800700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce010d380 0x7f7ce01a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdd400700 1 -- 172.21.15.16:0/2936361737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce0073940 msgr2=0x7f7ce019d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdd400700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce0073940 0x7f7ce019d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdd400700 1 -- 172.21.15.16:0/2936361737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce010d380 msgr2=0x7f7ce01a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdd400700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce010d380 0x7f7ce01a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdd400700 1 -- 172.21.15.16:0/2936361737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd8009c40 con 0x7f7ce0074f20 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdde00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce0073940 0x7f7ce019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cde800700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce010d380 0x7f7ce01a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.961+0000 7f7cdd400700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0074f20 0x7f7ce019d8b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000c00 tx=0x7f7ccc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.962+0000 7f7cd6a00700 1 -- 172.21.15.16:0/2936361737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc01d070 con 0x7f7ce0074f20 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.962+0000 7f7cd6a00700 1 -- 172.21.15.16:0/2936361737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ccc021410 con 0x7f7ce0074f20 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.962+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc0095f0 con 0x7f7ce0074f20 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.962+0000 7f7cd6a00700 1 -- 172.21.15.16:0/2936361737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc017650 con 0x7f7ce0074f20 2024-04-05T23:33:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.962+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce01a5680 con 0x7f7ce0074f20 2024-04-05T23:33:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.963+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce01035e0 con 0x7f7ce0074f20 2024-04-05T23:33:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.964+0000 7f7cd6a00700 1 -- 172.21.15.16:0/2936361737 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f7ccc021580 con 0x7f7ce0074f20 2024-04-05T23:33:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.966+0000 7f7cd6a00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7cc40a11b0 0x7f7cc40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.966+0000 7f7cd6a00700 1 -- 172.21.15.16:0/2936361737 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7ccc0cde80 con 0x7f7ce0074f20 2024-04-05T23:33:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.966+0000 7f7cdde00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7cc40a11b0 0x7f7cc40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.967+0000 7f7cdde00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7cc40a11b0 0x7f7cc40a3670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009cb0 tx=0x7f7cd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:22.970+0000 7f7cd6a00700 1 -- 172.21.15.16:0/2936361737 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7ccc0974f0 con 0x7f7ce0074f20 2024-04-05T23:33:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.187+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ce0112bf0 con 0x7f7ce0074f20 2024-04-05T23:33:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.188+0000 7f7cd6a00700 1 -- 172.21.15.16:0/2936361737 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22519 (secure 0 0 0) 0x7f7ccc09ab10 con 0x7f7ce0074f20 2024-04-05T23:33:23.181 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:23.181 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:32:38.369016+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:33:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7cc40a11b0 msgr2=0x7f7cc40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7cc40a11b0 0x7f7cc40a3670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009cb0 tx=0x7f7cd0009260 comp rx=0 tx=0).stop 2024-04-05T23:33:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0074f20 msgr2=0x7f7ce019d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0074f20 0x7f7ce019d8b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000c00 tx=0x7f7ccc009580 comp rx=0 tx=0).stop 2024-04-05T23:33:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 shutdown_connections 2024-04-05T23:33:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f7cc40a11b0 0x7f7cc40a3670 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce010d380 0x7f7ce01a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce0073940 0x7f7ce019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 --2- 172.21.15.16:0/2936361737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce0074f20 0x7f7ce019d8b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 >> 172.21.15.16:0/2936361737 conn(0x7f7ce00fc320 msgr2=0x7f7ce0100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 shutdown_connections 2024-04-05T23:33:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.191+0000 7f7ce4e00700 1 -- 172.21.15.16:0/2936361737 wait complete. 2024-04-05T23:33:23.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-05T23:32:04.511660+0000', 'modified': '2024-04-05T23:32:11.436182+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24566, 'mds_1': 24550, 'mds_2': 24526, 'mds_3': 24476, 'mds_4': 24472}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14529': {'gid': 14529, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6841/2579712996', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 2579712996}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 2579712996}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24452': {'gid': 24452, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6833/1368666940', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1368666940}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 1368666940}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24472': {'gid': 24472, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.130:6833/477825490', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 477825490}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 477825490}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24476': {'gid': 24476, 'name': 'd', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.16:6837/2898043660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 2898043660}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 2898043660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24488': {'gid': 24488, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6835/686908192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6834', 'nonce': 686908192}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 686908192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24518': {'gid': 24518, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.27:6837/541968377', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6836', 'nonce': 541968377}, {'type': 'v1', 'addr': '172.21.15.27:6837', 'nonce': 541968377}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6835/3125278540', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 3125278540}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 3125278540}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24526': {'gid': 24526, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.130:6837/3873050281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 3873050281}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 3873050281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.130:6839/2970827239', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 2970827239}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 2970827239}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24566': {'gid': 24566, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6839/491598025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6838', 'nonce': 491598025}, {'type': 'v1', 'addr': '172.21.15.27:6839', 'nonce': 491598025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-05T23:33:23.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-05T23:33:23.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-04-05T23:33:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.411+0000 7fedefc00700 1 -- 172.21.15.16:0/610545416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 msgr2=0x7fede8109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.411+0000 7fedefc00700 1 --2- 172.21.15.16:0/610545416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 0x7fede8109320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fede000b0a0 tx=0x7fede000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.411+0000 7fedefc00700 1 -- 172.21.15.16:0/610545416 shutdown_connections 2024-04-05T23:33:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.411+0000 7fedefc00700 1 --2- 172.21.15.16:0/610545416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 0x7fede8109320 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.411+0000 7fedefc00700 1 --2- 172.21.15.16:0/610545416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fede8100e60 0x7fede8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.411+0000 7fedefc00700 1 --2- 172.21.15.16:0/610545416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede8102040 0x7fede81024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.411+0000 7fedefc00700 1 -- 172.21.15.16:0/610545416 >> 172.21.15.16:0/610545416 conn(0x7fede80fc440 msgr2=0x7fede80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.411+0000 7fedefc00700 1 -- 172.21.15.16:0/610545416 shutdown_connections 2024-04-05T23:33:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.411+0000 7fedefc00700 1 -- 172.21.15.16:0/610545416 wait complete. 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.412+0000 7fedefc00700 1 Processor -- start 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.412+0000 7fedefc00700 1 -- start start 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fedefc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede8100e60 0x7fede8196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fedefc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fede8102040 0x7fede8197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fedefc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 0x7fede819e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fedefc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede819ea50 con 0x7fede8102a00 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fedefc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede819eb90 con 0x7fede8100e60 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fedefc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede819ed20 con 0x7fede8102040 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fededc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 0x7fede819e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fede7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fede8102040 0x7fede8197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fede7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fede8102040 0x7fede8197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43442/0 (socket says 172.21.15.16:43442) 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fededc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 0x7fede819e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60462/0 (socket says 172.21.15.16:60462) 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7fede7e00700 1 -- 172.21.15.16:0/462412021 learned_addr learned my addr 172.21.15.16:0/462412021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.413+0000 7feded200700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede8100e60 0x7fede8196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7feded200700 1 -- 172.21.15.16:0/462412021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fede8102040 msgr2=0x7fede8197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7feded200700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fede8102040 0x7fede8197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7feded200700 1 -- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 msgr2=0x7fede819e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7feded200700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 0x7fede819e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7feded200700 1 -- 172.21.15.16:0/462412021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede0009d00 con 0x7fede8100e60 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7fede7e00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fede8102040 0x7fede8197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7fededc00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 0x7fede819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7feded200700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede8100e60 0x7fede8196f20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feddc00e9c0 tx=0x7feddc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7fede5600700 1 -- 172.21.15.16:0/462412021 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc004220 con 0x7fede8100e60 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7fede5600700 1 -- 172.21.15.16:0/462412021 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feddc004380 con 0x7fede8100e60 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7fede5600700 1 -- 172.21.15.16:0/462412021 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc003be0 con 0x7fede8100e60 2024-04-05T23:33:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7fedefc00700 1 -- 172.21.15.16:0/462412021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede819efa0 con 0x7fede8100e60 2024-04-05T23:33:23.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.414+0000 7fedefc00700 1 -- 172.21.15.16:0/462412021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede819f3c0 con 0x7fede8100e60 2024-04-05T23:33:23.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.417+0000 7fede5600700 1 -- 172.21.15.16:0/462412021 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7feddc003d40 con 0x7fede8100e60 2024-04-05T23:33:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fede5600700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedd00a11b0 0x7fedd00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fee037fcb80 1 Processor -- start 2024-04-05T23:33:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fede5600700 1 -- 172.21.15.16:0/462412021 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feddc014070 con 0x7fede8100e60 2024-04-05T23:33:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fee037fcb80 1 -- start start 2024-04-05T23:33:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fede7e00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedd00a11b0 0x7fedd00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30de510 0x561ad30de930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dee70 0x561ad30df4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dfa60 0x561ad30dfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fee037fcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad3044b90 con 0x561ad30dee70 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fee037fcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad3035940 con 0x561ad30dfa60 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.419+0000 7fee037fcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad3033300 con 0x561ad30de510 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fededc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dfa60 0x561ad30dfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fede7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dee70 0x561ad30df4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fededc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dfa60 0x561ad30dfee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38200/0 (socket says 172.21.15.16:38200) 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fededc00700 1 -- 172.21.15.16:0/2757632455 learned_addr learned my addr 172.21.15.16:0/2757632455 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fede7e00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedd00a11b0 0x7fedd00a3670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fedd4005950 tx=0x7fedd4011950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7feded200700 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30de510 0x561ad30de930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fede7e00700 1 -- 172.21.15.16:0/2757632455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30de510 msgr2=0x561ad30de930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fede7e00700 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30de510 0x561ad30de930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fede7e00700 1 -- 172.21.15.16:0/2757632455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dfa60 msgr2=0x561ad30dfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fede7e00700 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dfa60 0x561ad30dfee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fede7e00700 1 -- 172.21.15.16:0/2757632455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ad2eb0360 con 0x561ad30dee70 2024-04-05T23:33:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7feded200700 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30de510 0x561ad30de930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:33:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fededc00700 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dfa60 0x561ad30dfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fede7e00700 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dee70 0x561ad30df4d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fedd4011d50 tx=0x7fedd401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf92941ca267951 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fedda000700 1 -- 172.21.15.16:0/2757632455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4004220 con 0x561ad30dee70 2024-04-05T23:33:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fedda000700 1 -- 172.21.15.16:0/2757632455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedd401de90 con 0x561ad30dee70 2024-04-05T23:33:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.420+0000 7fedda000700 1 -- 172.21.15.16:0/2757632455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4005480 con 0x561ad30dee70 2024-04-05T23:33:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.421+0000 7fee037fcb80 1 -- 172.21.15.16:0/2757632455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dee70 msgr2=0x561ad30df4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.421+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dee70 0x561ad30df4d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fedd4011d50 tx=0x7fedd401c000 comp rx=0 tx=0).stop 2024-04-05T23:33:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.421+0000 7fee037fcb80 1 -- 172.21.15.16:0/2757632455 shutdown_connections 2024-04-05T23:33:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.421+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dfa60 0x561ad30dfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.421+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30de510 0x561ad30de930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.421+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2757632455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dee70 0x561ad30df4d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.421+0000 7fee037fcb80 1 -- 172.21.15.16:0/2757632455 >> 172.21.15.16:0/2757632455 conn(0x561ad30521d0 msgr2=0x561ad30d9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.421+0000 7fee037fcb80 1 -- 172.21.15.16:0/2757632455 shutdown_connections 2024-04-05T23:33:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.421+0000 7fee037fcb80 1 -- 172.21.15.16:0/2757632455 wait complete. 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.422+0000 7fee037fcb80 1 -- start start 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.422+0000 7fee037fcb80 1 Processor -- start 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.422+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad30e8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.422+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 0x561ad30e86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.422+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad30ef700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.422+0000 7fee037fcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad302ebf0 con 0x561ad30dfa60 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.422+0000 7fee037fcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad2fcad30 con 0x561ad30de510 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.422+0000 7fee037fcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad2fca9d0 con 0x561ad30dee70 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fee037fcb80 10 client.0 _ll_register_callbacks cb 0x561ad2e36350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fededc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad30ef700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7feded200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad30e8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fededc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad30ef700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60478/0 (socket says 172.21.15.16:60478) 2024-04-05T23:33:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fededc00700 1 -- 172.21.15.16:0/2684754738 learned_addr learned my addr 172.21.15.16:0/2684754738 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fede7e00700 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 0x561ad30e86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fededc00700 1 -- 172.21.15.16:0/2684754738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 msgr2=0x561ad30e86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fededc00700 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 0x561ad30e86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fededc00700 1 -- 172.21.15.16:0/2684754738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 msgr2=0x561ad30e8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fededc00700 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad30e8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fededc00700 1 -- 172.21.15.16:0/2684754738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede8109c50 con 0x561ad30dfa60 2024-04-05T23:33:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7feded200700 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad30e8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fededc00700 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad30ef700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fede0007a60 tx=0x7fede0009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5062ae3dc1914 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.423+0000 7fedcea00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fedcea00700 1 -- 172.21.15.16:0/2684754738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede000e040 con 0x561ad30dfa60 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fedcea00700 1 -- 172.21.15.16:0/2684754738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fede0023070 con 0x561ad30dfa60 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fee037fcb80 1 -- 172.21.15.16:0/2684754738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ad30d1440 con 0x561ad30dfa60 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fedcea00700 1 -- 172.21.15.16:0/2684754738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fede0003a00 con 0x561ad30dfa60 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fee037fcb80 10 client.15138 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fee037fcb80 1 -- 172.21.15.16:0/2684754738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fedd401d490 con 0x561ad30dfa60 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fedcea00700 1 -- 172.21.15.16:0/2684754738 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561ad30d1440 con 0x561ad30dfa60 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fedcea00700 1 -- 172.21.15.16:0/2684754738 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fede000b0d0 con 0x561ad30dfa60 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fee037fcb80 10 client.15138 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fee037fcb80 10 client.15138 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.424+0000 7fee037fcb80 20 client.15138 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.425+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x561ad30f32b0 0x561ad30f8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.425+0000 7fee037fcb80 4 client.15138 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.425+0000 7fee037fcb80 1 -- 172.21.15.16:0/2684754738 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ad30276b0 con 0x561ad30f32b0 2024-04-05T23:33:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.425+0000 7feded200700 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x561ad30f32b0 0x561ad30f8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.425+0000 7feded200700 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x561ad30f32b0 0x561ad30f8550 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=67532647eaa19b0b server_cookie=2db93722fec3989 in_seq=0 out_seq=0 2024-04-05T23:33:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.425+0000 7fedcea00700 10 client.15138 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:33:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fedcea00700 1 -- 172.21.15.16:0/2684754738 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561ad30276b0 con 0x561ad30f32b0 2024-04-05T23:33:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fedcea00700 10 client.15138 handle_command_reply: tid=0 2024-04-05T23:33:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 client.15138 shutdown 2024-04-05T23:33:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 -- 172.21.15.16:0/2684754738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 msgr2=0x561ad30ef700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad30ef700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fede0007a60 tx=0x7fede0009c60 comp rx=0 tx=0).stop 2024-04-05T23:33:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 -- 172.21.15.16:0/2684754738 shutdown_connections 2024-04-05T23:33:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x561ad30f32b0 0x561ad30f8550 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad30ef700 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad30e8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fedcea00700 0 client.15138 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:33:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2684754738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 0x561ad30e86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 -- 172.21.15.16:0/2684754738 >> 172.21.15.16:0/2684754738 conn(0x561ad30521d0 msgr2=0x561ad30d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 -- 172.21.15.16:0/2684754738 shutdown_connections 2024-04-05T23:33:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.426+0000 7fee037fcb80 1 -- 172.21.15.16:0/2684754738 wait complete. 2024-04-05T23:33:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.427+0000 7fee037fcb80 20 client.15138 trim_cache size 0 max 16384 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fee037fcb80 1 Processor -- start 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fee037fcb80 1 -- start start 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30de510 0x561ad30fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dee70 0x561ad30de8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dfa60 0x561ad30e7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fee037fcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad2eb2650 con 0x561ad30de510 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fee037fcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad30e9080 con 0x561ad30dee70 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fee037fcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad30276b0 con 0x561ad30dfa60 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7feded200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30de510 0x561ad30fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fede7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dee70 0x561ad30de8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fede7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dee70 0x561ad30de8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38214/0 (socket says 172.21.15.16:38214) 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7feded200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30de510 0x561ad30fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60486/0 (socket says 172.21.15.16:60486) 2024-04-05T23:33:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fede7e00700 1 -- 172.21.15.16:0/1618919527 learned_addr learned my addr 172.21.15.16:0/1618919527 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.445+0000 7fededc00700 1 --2- 172.21.15.16:0/1618919527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dfa60 0x561ad30e7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7fede7e00700 1 -- 172.21.15.16:0/1618919527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dfa60 msgr2=0x561ad30e7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7fede7e00700 1 --2- 172.21.15.16:0/1618919527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dfa60 0x561ad30e7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7fede7e00700 1 -- 172.21.15.16:0/1618919527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30de510 msgr2=0x561ad30fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7fede7e00700 1 --2- 172.21.15.16:0/1618919527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30de510 0x561ad30fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7fede7e00700 1 -- 172.21.15.16:0/1618919527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ad2e5cb10 con 0x561ad30dee70 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7feded200700 1 --2- 172.21.15.16:0/1618919527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30de510 0x561ad30fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7fede7e00700 1 --2- 172.21.15.16:0/1618919527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dee70 0x561ad30de8f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009e40 tx=0x7fedd4005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc4ff4cf196664f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7feddaa00700 1 -- 172.21.15.16:0/1618919527 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd402c070 con 0x561ad30dee70 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7feddaa00700 1 -- 172.21.15.16:0/1618919527 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedd40044e0 con 0x561ad30dee70 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.446+0000 7feddaa00700 1 -- 172.21.15.16:0/1618919527 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd400f480 con 0x561ad30dee70 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 -- 172.21.15.16:0/1618919527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dee70 msgr2=0x561ad30de8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 --2- 172.21.15.16:0/1618919527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dee70 0x561ad30de8f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009e40 tx=0x7fedd4005dc0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 -- 172.21.15.16:0/1618919527 shutdown_connections 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 --2- 172.21.15.16:0/1618919527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30de510 0x561ad30fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 --2- 172.21.15.16:0/1618919527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dfa60 0x561ad30e7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 --2- 172.21.15.16:0/1618919527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30dee70 0x561ad30de8f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 -- 172.21.15.16:0/1618919527 >> 172.21.15.16:0/1618919527 conn(0x561ad30521d0 msgr2=0x561ad30fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 -- 172.21.15.16:0/1618919527 shutdown_connections 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 -- 172.21.15.16:0/1618919527 wait complete. 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 -- start start 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.447+0000 7fee037fcb80 1 Processor -- start 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad3143e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 0x561ad3144390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fee037fcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad314b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fee037fcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad314b8c0 con 0x561ad30dfa60 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fee037fcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad314ba00 con 0x561ad30de510 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fee037fcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ad314bb90 con 0x561ad30dee70 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fee037fcb80 10 client.0 _ll_register_callbacks cb 0x7fede00074e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fededc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad314b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7feded200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad3143e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7feded200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad3143e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38216/0 (socket says 172.21.15.16:38216) 2024-04-05T23:33:23.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7feded200700 1 -- 172.21.15.16:0/2781968763 learned_addr learned my addr 172.21.15.16:0/2781968763 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fede7e00700 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 0x561ad3144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fededc00700 1 -- 172.21.15.16:0/2781968763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 msgr2=0x561ad3144390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fededc00700 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 0x561ad3144390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.448+0000 7fededc00700 1 -- 172.21.15.16:0/2781968763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 msgr2=0x561ad3143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fededc00700 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad3143e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fededc00700 1 -- 172.21.15.16:0/2781968763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede819f3c0 con 0x561ad30dfa60 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fede7e00700 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 0x561ad3144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7feded200700 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad3143e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fededc00700 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad314b380 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fede0007b90 tx=0x7fede00095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2fb15d798f8dd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fedcea00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fedcea00700 1 -- 172.21.15.16:0/2781968763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede000e040 con 0x561ad30dfa60 2024-04-05T23:33:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fee037fcb80 1 -- 172.21.15.16:0/2781968763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedd401de90 con 0x561ad30dfa60 2024-04-05T23:33:23.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fedcea00700 1 -- 172.21.15.16:0/2781968763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fede0023030 con 0x561ad30dfa60 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fedcea00700 1 -- 172.21.15.16:0/2781968763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fede0003e10 con 0x561ad30dfa60 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fee037fcb80 10 client.15144 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.449+0000 7fee037fcb80 1 -- 172.21.15.16:0/2781968763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561ad2eb0360 con 0x561ad30dfa60 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7fedcea00700 1 -- 172.21.15.16:0/2781968763 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedd401de90 con 0x561ad30dfa60 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7fedcea00700 1 -- 172.21.15.16:0/2781968763 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fede0027770 con 0x561ad30dfa60 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7fee037fcb80 10 client.15144 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7fee037fcb80 10 client.15144 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7fee037fcb80 20 client.15144 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x561ad3150630 0x561ad314cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7fee037fcb80 4 client.15144 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7fee037fcb80 1 -- 172.21.15.16:0/2781968763 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561ad2e7dca0 con 0x561ad3150630 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7feded200700 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x561ad3150630 0x561ad314cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7feded200700 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x561ad3150630 0x561ad314cfc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=eca20a5552248214 server_cookie=8c333268b99a670f in_seq=0 out_seq=0 2024-04-05T23:33:23.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.450+0000 7fedcea00700 10 client.15144 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fedcea00700 1 -- 172.21.15.16:0/2781968763 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x561ad2e7dca0 con 0x561ad3150630 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fedcea00700 10 client.15144 handle_command_reply: tid=0 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 client.15144 shutdown 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 -- 172.21.15.16:0/2781968763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 msgr2=0x561ad314b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad314b380 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fede0007b90 tx=0x7fede00095f0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 -- 172.21.15.16:0/2781968763 shutdown_connections 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x561ad3150630 0x561ad314cfc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ad30dfa60 0x561ad314b380 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ad30de510 0x561ad3143e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 --2- 172.21.15.16:0/2781968763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ad30dee70 0x561ad3144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fedcea00700 0 client.15144 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 -- 172.21.15.16:0/2781968763 >> 172.21.15.16:0/2781968763 conn(0x561ad30521d0 msgr2=0x561ad30fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 -- 172.21.15.16:0/2781968763 shutdown_connections 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 1 -- 172.21.15.16:0/2781968763 wait complete. 2024-04-05T23:33:23.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.451+0000 7fee037fcb80 20 client.15144 trim_cache size 0 max 16384 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24472, 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 73.035025644000001, 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:23.448 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:23.449 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 108.36781033699999 2024-04-05T23:33:23.449 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 -- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedd00a11b0 msgr2=0x7fedd00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedd00a11b0 0x7fedd00a3670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fedd4005950 tx=0x7fedd4011950 comp rx=0 tx=0).stop 2024-04-05T23:33:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 -- 172.21.15.16:0/462412021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede8100e60 msgr2=0x7fede8196f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede8100e60 0x7fede8196f20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feddc00e9c0 tx=0x7feddc00ecd0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 -- 172.21.15.16:0/462412021 shutdown_connections 2024-04-05T23:33:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fedd00a11b0 0x7fedd00a3670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fede8102a00 0x7fede819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede8100e60 0x7fede8196f20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 --2- 172.21.15.16:0/462412021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fede8102040 0x7fede8197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 -- 172.21.15.16:0/462412021 >> 172.21.15.16:0/462412021 conn(0x7fede80fc440 msgr2=0x7fede81053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 -- 172.21.15.16:0/462412021 shutdown_connections 2024-04-05T23:33:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.452+0000 7fedefc00700 1 -- 172.21.15.16:0/462412021 wait complete. 2024-04-05T23:33:23.459 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:23 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3245373421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:23.459 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:23 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2936361737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:23.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-05T23:33:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.669+0000 7f5856e00700 1 -- 172.21.15.16:0/3666067648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5850100e60 msgr2=0x7f5850101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.669+0000 7f5856e00700 1 --2- 172.21.15.16:0/3666067648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5850100e60 0x7f5850101280 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5840009990 tx=0x7f5840009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.669+0000 7f5856e00700 1 -- 172.21.15.16:0/3666067648 shutdown_connections 2024-04-05T23:33:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.669+0000 7f5856e00700 1 --2- 172.21.15.16:0/3666067648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5850102a00 0x7f5850109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.669+0000 7f5856e00700 1 --2- 172.21.15.16:0/3666067648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5850100e60 0x7f5850101280 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.669+0000 7f5856e00700 1 --2- 172.21.15.16:0/3666067648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5850102040 0x7f58501024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.669+0000 7f5856e00700 1 -- 172.21.15.16:0/3666067648 >> 172.21.15.16:0/3666067648 conn(0x7f58500fc440 msgr2=0x7f58500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.669+0000 7f5856e00700 1 -- 172.21.15.16:0/3666067648 shutdown_connections 2024-04-05T23:33:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.669+0000 7f5856e00700 1 -- 172.21.15.16:0/3666067648 wait complete. 2024-04-05T23:33:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.670+0000 7f5856e00700 1 Processor -- start 2024-04-05T23:33:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f5856e00700 1 -- start start 2024-04-05T23:33:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f5856e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5850100e60 0x7f5850199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f5856e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5850102040 0x7f58501995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f5856e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5850102a00 0x7f58501a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f5856e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58501a0bc0 con 0x7f5850102040 2024-04-05T23:33:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f5856e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58501a0d00 con 0x7f5850100e60 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f5856e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58501a0e90 con 0x7f5850102a00 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f584f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5850102040 0x7f58501995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f584f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5850102040 0x7f58501995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60522/0 (socket says 172.21.15.16:60522) 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f584f400700 1 -- 172.21.15.16:0/619686039 learned_addr learned my addr 172.21.15.16:0/619686039 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f584fe00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5850100e60 0x7f5850199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f5854e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5850102a00 0x7f58501a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.671+0000 7f584f400700 1 -- 172.21.15.16:0/619686039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5850102a00 msgr2=0x7f58501a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f584f400700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5850102a00 0x7f58501a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f584f400700 1 -- 172.21.15.16:0/619686039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5850100e60 msgr2=0x7f5850199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f584f400700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5850100e60 0x7f5850199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f584f400700 1 -- 172.21.15.16:0/619686039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400095f0 con 0x7f5850102040 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f5854e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5850102a00 0x7f58501a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:23.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f584f400700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5850102040 0x7f58501995d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f583c00e970 tx=0x7f583c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f584cc00700 1 -- 172.21.15.16:0/619686039 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c009790 con 0x7f5850102040 2024-04-05T23:33:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f5856e00700 1 -- 172.21.15.16:0/619686039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58501a1170 con 0x7f5850102040 2024-04-05T23:33:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f584cc00700 1 -- 172.21.15.16:0/619686039 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f583c00eec0 con 0x7f5850102040 2024-04-05T23:33:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f584cc00700 1 -- 172.21.15.16:0/619686039 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c003cf0 con 0x7f5850102040 2024-04-05T23:33:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.672+0000 7f5856e00700 1 -- 172.21.15.16:0/619686039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58501a16d0 con 0x7f5850102040 2024-04-05T23:33:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.675+0000 7f584cc00700 1 -- 172.21.15.16:0/619686039 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f583c00ccd0 con 0x7f5850102040 2024-04-05T23:33:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.677+0000 7f586a92cb80 1 Processor -- start 2024-04-05T23:33:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.677+0000 7f586a92cb80 1 -- start start 2024-04-05T23:33:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.677+0000 7f584cc00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58340a1230 0x7f58340a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304510 0x55fe43304930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304e70 0x55fe433054d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe43305ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584cc00700 1 -- 172.21.15.16:0/619686039 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f583c014070 con 0x7f5850102040 2024-04-05T23:33:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f586a92cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4326ab90 con 0x55fe43305a60 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f586a92cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4325b940 con 0x55fe43304510 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f586a92cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe43259300 con 0x55fe43304e70 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f5854e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58340a1230 0x7f58340a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304510 0x55fe43304930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304510 0x55fe43304930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38254/0 (socket says 172.21.15.16:38254) 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304e70 0x55fe433054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584fe00700 1 -- 172.21.15.16:0/1601560528 learned_addr learned my addr 172.21.15.16:0/1601560528 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f5854e00700 1 --2- 172.21.15.16:0/1601560528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe43305ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584fe00700 1 -- 172.21.15.16:0/1601560528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304e70 msgr2=0x55fe433054d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584fe00700 1 --2- 172.21.15.16:0/1601560528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304e70 0x55fe433054d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584fe00700 1 -- 172.21.15.16:0/1601560528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 msgr2=0x55fe43305ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584fe00700 1 --2- 172.21.15.16:0/1601560528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe43305ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.678+0000 7f584fe00700 1 -- 172.21.15.16:0/1601560528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe430d6360 con 0x55fe43304510 2024-04-05T23:33:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.679+0000 7f5854e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58340a1230 0x7f58340a36f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5848009fd0 tx=0x7f5848007b00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.679+0000 7f584fe00700 1 --2- 172.21.15.16:0/1601560528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304510 0x55fe43304930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5850101bb0 tx=0x7f5840009570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89b3fdd9e8f047cf server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.679+0000 7f5845600700 1 -- 172.21.15.16:0/1601560528 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584001d070 con 0x55fe43304510 2024-04-05T23:33:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.679+0000 7f5845600700 1 -- 172.21.15.16:0/1601560528 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f584001e040 con 0x55fe43304510 2024-04-05T23:33:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.679+0000 7f5845600700 1 -- 172.21.15.16:0/1601560528 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840004860 con 0x55fe43304510 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 -- 172.21.15.16:0/1601560528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304510 msgr2=0x55fe43304930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 --2- 172.21.15.16:0/1601560528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304510 0x55fe43304930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5850101bb0 tx=0x7f5840009570 comp rx=0 tx=0).stop 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 -- 172.21.15.16:0/1601560528 shutdown_connections 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 --2- 172.21.15.16:0/1601560528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe43305ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 --2- 172.21.15.16:0/1601560528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304510 0x55fe43304930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 --2- 172.21.15.16:0/1601560528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304e70 0x55fe433054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 -- 172.21.15.16:0/1601560528 >> 172.21.15.16:0/1601560528 conn(0x55fe432781d0 msgr2=0x55fe432ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 -- 172.21.15.16:0/1601560528 shutdown_connections 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 -- 172.21.15.16:0/1601560528 wait complete. 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.680+0000 7f586a92cb80 1 -- start start 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f586a92cb80 1 Processor -- start 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe4330dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe4330e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43315560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f586a92cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe43254bf0 con 0x55fe43304e70 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f586a92cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe431f0d30 con 0x55fe43305a60 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f586a92cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe431f09d0 con 0x55fe43304510 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f586a92cb80 10 client.0 _ll_register_callbacks cb 0x55fe4305c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f584f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe4330e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f584fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe4330dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f584f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe4330e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60548/0 (socket says 172.21.15.16:60548) 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.681+0000 7f584f400700 1 -- 172.21.15.16:0/2091304693 learned_addr learned my addr 172.21.15.16:0/2091304693 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f5854e00700 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43315560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f584f400700 1 -- 172.21.15.16:0/2091304693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 msgr2=0x55fe4330dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f584f400700 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe4330dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f584f400700 1 -- 172.21.15.16:0/2091304693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 msgr2=0x55fe43315560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f584f400700 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43315560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f584f400700 1 -- 172.21.15.16:0/2091304693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5850109c50 con 0x55fe43304e70 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f5854e00700 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43315560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f584f400700 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe4330e520 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f583c0969c0 tx=0x7f583c09a740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4d3a371d20be72 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f583a000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f583a000700 1 -- 172.21.15.16:0/2091304693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c0a23d0 con 0x55fe43304e70 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f583a000700 1 -- 172.21.15.16:0/2091304693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f583c0ca860 con 0x55fe43304e70 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f586a92cb80 1 -- 172.21.15.16:0/2091304693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe432f7440 con 0x55fe43304e70 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f583a000700 1 -- 172.21.15.16:0/2091304693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f583c0b8250 con 0x55fe43304e70 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f586a92cb80 10 client.15156 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.682+0000 7f586a92cb80 1 -- 172.21.15.16:0/2091304693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe43082b10 con 0x55fe43304e70 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.683+0000 7f583a000700 1 -- 172.21.15.16:0/2091304693 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fe432f7440 con 0x55fe43304e70 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.683+0000 7f583a000700 1 -- 172.21.15.16:0/2091304693 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f583c0a1f40 con 0x55fe43304e70 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.683+0000 7f586a92cb80 10 client.15156 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.683+0000 7f586a92cb80 10 client.15156 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-05T23:33:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.683+0000 7f586a92cb80 20 client.15156 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.683+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55fe433191a0 0x55fe4331e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.683+0000 7f586a92cb80 4 client.15156 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.683+0000 7f586a92cb80 1 -- 172.21.15.16:0/2091304693 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe4324d6b0 con 0x55fe433191a0 2024-04-05T23:33:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.683+0000 7f584fe00700 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55fe433191a0 0x55fe4331e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.684+0000 7f584fe00700 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55fe433191a0 0x55fe4331e4c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6624b35f4e67277a server_cookie=a52eb5c6a687d581 in_seq=0 out_seq=0 2024-04-05T23:33:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.684+0000 7f583a000700 10 client.15156 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:33:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.684+0000 7f583a000700 1 -- 172.21.15.16:0/2091304693 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fe4324d6b0 con 0x55fe433191a0 2024-04-05T23:33:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.684+0000 7f583a000700 10 client.15156 handle_command_reply: tid=0 2024-04-05T23:33:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.684+0000 7f586a92cb80 1 client.15156 shutdown 2024-04-05T23:33:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 -- 172.21.15.16:0/2091304693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 msgr2=0x55fe4330e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe4330e520 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f583c0969c0 tx=0x7f583c09a740 comp rx=0 tx=0).stop 2024-04-05T23:33:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 -- 172.21.15.16:0/2091304693 shutdown_connections 2024-04-05T23:33:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55fe433191a0 0x55fe4331e4c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43315560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe4330dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f583a000700 0 client.15156 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:33:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2091304693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe4330e520 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 -- 172.21.15.16:0/2091304693 >> 172.21.15.16:0/2091304693 conn(0x55fe432781d0 msgr2=0x55fe432f9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 -- 172.21.15.16:0/2091304693 shutdown_connections 2024-04-05T23:33:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 1 -- 172.21.15.16:0/2091304693 wait complete. 2024-04-05T23:33:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.685+0000 7f586a92cb80 20 client.15156 trim_cache size 0 max 16384 2024-04-05T23:33:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.703+0000 7f586a92cb80 1 Processor -- start 2024-04-05T23:33:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.703+0000 7f586a92cb80 1 -- start start 2024-04-05T23:33:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe433048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304e70 0x55fe43315480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe4330d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f586a92cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe430d8650 con 0x55fe43305a60 2024-04-05T23:33:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f586a92cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe432fbd00 con 0x55fe43304e70 2024-04-05T23:33:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f586a92cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4324d6b0 con 0x55fe43304510 2024-04-05T23:33:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f5854e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe4330d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f5854e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe4330d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60552/0 (socket says 172.21.15.16:60552) 2024-04-05T23:33:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f5854e00700 1 -- 172.21.15.16:0/204943146 learned_addr learned my addr 172.21.15.16:0/204943146 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f584f400700 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304e70 0x55fe43315480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f584fe00700 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe433048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f5854e00700 1 -- 172.21.15.16:0/204943146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 msgr2=0x55fe433048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f5854e00700 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe433048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f5854e00700 1 -- 172.21.15.16:0/204943146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304e70 msgr2=0x55fe43315480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f5854e00700 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304e70 0x55fe43315480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f5854e00700 1 -- 172.21.15.16:0/204943146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe4321ae50 con 0x55fe43305a60 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.704+0000 7f584fe00700 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe433048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f5854e00700 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe4330d4f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f584801a7b0 tx=0x7f584801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24cac17c63ccb05 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f5846000700 1 -- 172.21.15.16:0/204943146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584800e050 con 0x55fe43305a60 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f5846000700 1 -- 172.21.15.16:0/204943146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f584800f040 con 0x55fe43305a60 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f5846000700 1 -- 172.21.15.16:0/204943146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848027830 con 0x55fe43305a60 2024-04-05T23:33:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f586a92cb80 1 -- 172.21.15.16:0/204943146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 msgr2=0x55fe4330d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f586a92cb80 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe4330d4f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f584801a7b0 tx=0x7f584801d000 comp rx=0 tx=0).stop 2024-04-05T23:33:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f586a92cb80 1 -- 172.21.15.16:0/204943146 shutdown_connections 2024-04-05T23:33:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f586a92cb80 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43305a60 0x55fe4330d4f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f586a92cb80 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe433048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f586a92cb80 1 --2- 172.21.15.16:0/204943146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43304e70 0x55fe43315480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.705+0000 7f586a92cb80 1 -- 172.21.15.16:0/204943146 >> 172.21.15.16:0/204943146 conn(0x55fe432781d0 msgr2=0x55fe432feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.706+0000 7f586a92cb80 1 -- 172.21.15.16:0/204943146 shutdown_connections 2024-04-05T23:33:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.706+0000 7f586a92cb80 1 -- 172.21.15.16:0/204943146 wait complete. 2024-04-05T23:33:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.706+0000 7f586a92cb80 1 -- start start 2024-04-05T23:33:23.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.706+0000 7f586a92cb80 1 Processor -- start 2024-04-05T23:33:23.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe433154e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe43312020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f586a92cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43312560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f586a92cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe43312c40 con 0x55fe43304e70 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f586a92cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4336dab0 con 0x55fe43305a60 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f586a92cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4336dc40 con 0x55fe43304510 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f586a92cb80 10 client.0 _ll_register_callbacks cb 0x7f583c099b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f584f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe43312020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f584f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe43312020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60556/0 (socket says 172.21.15.16:60556) 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f584f400700 1 -- 172.21.15.16:0/2762779007 learned_addr learned my addr 172.21.15.16:0/2762779007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f5854e00700 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43312560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f584fe00700 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe433154e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f584f400700 1 -- 172.21.15.16:0/2762779007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 msgr2=0x55fe433154e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f584f400700 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe433154e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f584f400700 1 -- 172.21.15.16:0/2762779007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 msgr2=0x55fe43312560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f584f400700 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43312560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f584f400700 1 -- 172.21.15.16:0/2762779007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584801a470 con 0x55fe43304e70 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.707+0000 7f5854e00700 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43312560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.708+0000 7f584f400700 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe43312020 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f583c096970 tx=0x7f583c09a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b122f45e1729800f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.708+0000 7f583a000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.708+0000 7f583a000700 1 -- 172.21.15.16:0/2762779007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c095e10 con 0x55fe43304e70 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.708+0000 7f583a000700 1 -- 172.21.15.16:0/2762779007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f583c0a23d0 con 0x55fe43304e70 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.708+0000 7f586a92cb80 1 -- 172.21.15.16:0/2762779007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f584001e040 con 0x55fe43304e70 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.708+0000 7f583a000700 1 -- 172.21.15.16:0/2762779007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f583c0a8050 con 0x55fe43304e70 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.708+0000 7f586a92cb80 10 client.15162 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.708+0000 7f586a92cb80 1 -- 172.21.15.16:0/2762779007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe430dac80 con 0x55fe43304e70 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.709+0000 7f583a000700 1 -- 172.21.15.16:0/2762779007 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f584001e040 con 0x55fe43304e70 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.709+0000 7f583a000700 1 -- 172.21.15.16:0/2762779007 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f583c0994d0 con 0x55fe43304e70 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.709+0000 7f586a92cb80 10 client.15162 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.709+0000 7f586a92cb80 10 client.15162 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.709+0000 7f586a92cb80 20 client.15162 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.709+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55fe433723d0 0x55fe4336ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.709+0000 7f586a92cb80 4 client.15162 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.709+0000 7f586a92cb80 1 -- 172.21.15.16:0/2762779007 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fe430a3ca0 con 0x55fe433723d0 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.709+0000 7f584fe00700 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55fe433723d0 0x55fe4336ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f584fe00700 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55fe433723d0 0x55fe4336ef30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4213aab5aa0b8357 server_cookie=74db6d5e106635b in_seq=0 out_seq=0 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f583a000700 10 client.15162 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f583a000700 1 -- 172.21.15.16:0/2762779007 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55fe430a3ca0 con 0x55fe433723d0 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f583a000700 10 client.15162 handle_command_reply: tid=0 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 client.15162 shutdown 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 -- 172.21.15.16:0/2762779007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 msgr2=0x55fe43312020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe43312020 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f583c096970 tx=0x7f583c09a7b0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 -- 172.21.15.16:0/2762779007 shutdown_connections 2024-04-05T23:33:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55fe433723d0 0x55fe4336ef30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fe43305a60 0x55fe43312560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fe43304510 0x55fe433154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f583a000700 0 client.15162 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:33:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 --2- 172.21.15.16:0/2762779007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fe43304e70 0x55fe43312020 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 -- 172.21.15.16:0/2762779007 >> 172.21.15.16:0/2762779007 conn(0x55fe432781d0 msgr2=0x55fe4330d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 -- 172.21.15.16:0/2762779007 shutdown_connections 2024-04-05T23:33:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 1 -- 172.21.15.16:0/2762779007 wait complete. 2024-04-05T23:33:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.710+0000 7f586a92cb80 20 client.15162 trim_cache size 0 max 16384 2024-04-05T23:33:23.706 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24476, 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 74.300547215999998, 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 101.996631738 2024-04-05T23:33:23.707 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 -- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58340a1230 msgr2=0x7f58340a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58340a1230 0x7f58340a36f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5848009fd0 tx=0x7f5848007b00 comp rx=0 tx=0).stop 2024-04-05T23:33:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 -- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5850102040 msgr2=0x7f58501995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5850102040 0x7f58501995d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f583c00e970 tx=0x7f583c00ec80 comp rx=0 tx=0).stop 2024-04-05T23:33:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 -- 172.21.15.16:0/619686039 shutdown_connections 2024-04-05T23:33:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58340a1230 0x7f58340a36f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5850102a00 0x7f58501a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5850100e60 0x7f5850199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 --2- 172.21.15.16:0/619686039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5850102040 0x7f58501995d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 -- 172.21.15.16:0/619686039 >> 172.21.15.16:0/619686039 conn(0x7f58500fc440 msgr2=0x7f58501078f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 -- 172.21.15.16:0/619686039 shutdown_connections 2024-04-05T23:33:23.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.711+0000 7f5856e00700 1 -- 172.21.15.16:0/619686039 wait complete. 2024-04-05T23:33:23.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:23 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3245373421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:23.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:23 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2936361737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:23.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-05T23:33:23.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:23 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3245373421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:23.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:23 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2936361737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.928+0000 7efc0b600700 1 -- 172.21.15.16:0/2743841746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 msgr2=0x7efc041024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.928+0000 7efc0b600700 1 --2- 172.21.15.16:0/2743841746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 0x7efc041024c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009990 tx=0x7efbf0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.928+0000 7efc0b600700 1 -- 172.21.15.16:0/2743841746 shutdown_connections 2024-04-05T23:33:23.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.928+0000 7efc0b600700 1 --2- 172.21.15.16:0/2743841746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc04102a00 0x7efc04109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.928+0000 7efc0b600700 1 --2- 172.21.15.16:0/2743841746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc04100e60 0x7efc04101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.928+0000 7efc0b600700 1 --2- 172.21.15.16:0/2743841746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 0x7efc041024c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.928+0000 7efc0b600700 1 -- 172.21.15.16:0/2743841746 >> 172.21.15.16:0/2743841746 conn(0x7efc040fc440 msgr2=0x7efc040fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.928+0000 7efc0b600700 1 -- 172.21.15.16:0/2743841746 shutdown_connections 2024-04-05T23:33:23.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.928+0000 7efc0b600700 1 -- 172.21.15.16:0/2743841746 wait complete. 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.929+0000 7efc0b600700 1 Processor -- start 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.929+0000 7efc0b600700 1 -- start start 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.929+0000 7efc0b600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc04100e60 0x7efc04074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.929+0000 7efc0b600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 0x7efc040732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.929+0000 7efc0b600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc04102a00 0x7efc04073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.929+0000 7efc0b600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc04073fb0 con 0x7efc04102040 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.929+0000 7efc0b600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc04074120 con 0x7efc04100e60 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.929+0000 7efc0b600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc041a0ec0 con 0x7efc04102a00 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.929+0000 7efc03e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 0x7efc040732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc03e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 0x7efc040732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60572/0 (socket says 172.21.15.16:60572) 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc03e00700 1 -- 172.21.15.16:0/4257298932 learned_addr learned my addr 172.21.15.16:0/4257298932 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc08c00700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc04100e60 0x7efc04074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc09600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc04102a00 0x7efc04073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc03e00700 1 -- 172.21.15.16:0/4257298932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc04102a00 msgr2=0x7efc04073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc03e00700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc04102a00 0x7efc04073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc03e00700 1 -- 172.21.15.16:0/4257298932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc04100e60 msgr2=0x7efc04074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc03e00700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc04100e60 0x7efc04074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc03e00700 1 -- 172.21.15.16:0/4257298932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf00095f0 con 0x7efc04102040 2024-04-05T23:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc03e00700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 0x7efc040732d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efbf0000c00 tx=0x7efbf0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc01600700 1 -- 172.21.15.16:0/4257298932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf001d070 con 0x7efc04102040 2024-04-05T23:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc01600700 1 -- 172.21.15.16:0/4257298932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbf0021410 con 0x7efc04102040 2024-04-05T23:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.930+0000 7efc01600700 1 -- 172.21.15.16:0/4257298932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0017650 con 0x7efc04102040 2024-04-05T23:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.931+0000 7efc0b600700 1 -- 172.21.15.16:0/4257298932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc041a10f0 con 0x7efc04102040 2024-04-05T23:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.931+0000 7efc0b600700 1 -- 172.21.15.16:0/4257298932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc041a1460 con 0x7efc04102040 2024-04-05T23:33:23.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.934+0000 7efc01600700 1 -- 172.21.15.16:0/4257298932 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7efbf0021580 con 0x7efc04102040 2024-04-05T23:33:23.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.935+0000 7efc1f0b5b80 1 Processor -- start 2024-04-05T23:33:23.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.935+0000 7efc1f0b5b80 1 -- start start 2024-04-05T23:33:23.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.935+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 0x5577c773e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.935+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c773f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.935+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773fa60 0x5577c773fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.935+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c76a4b90 con 0x5577c773ee70 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.935+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c7695940 con 0x5577c773e510 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.935+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c7693300 con 0x5577c773fa60 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.935+0000 7efc01600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efbec0a11b0 0x7efbec0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc08c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c773f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc09600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efbec0a11b0 0x7efbec0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc08c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c773f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60578/0 (socket says 172.21.15.16:60578) 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc08c00700 1 -- 172.21.15.16:0/2731942198 learned_addr learned my addr 172.21.15.16:0/2731942198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc01600700 1 -- 172.21.15.16:0/4257298932 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efbf00cdee0 con 0x7efc04102040 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc09600700 1 --2- 172.21.15.16:0/2731942198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 0x5577c773e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc03e00700 1 --2- 172.21.15.16:0/2731942198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773fa60 0x5577c773fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc08c00700 1 -- 172.21.15.16:0/2731942198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773fa60 msgr2=0x5577c773fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc08c00700 1 --2- 172.21.15.16:0/2731942198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773fa60 0x5577c773fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc08c00700 1 -- 172.21.15.16:0/2731942198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 msgr2=0x5577c773e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc08c00700 1 --2- 172.21.15.16:0/2731942198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 0x5577c773e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc08c00700 1 -- 172.21.15.16:0/2731942198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577c7510360 con 0x5577c773ee70 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc09600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efbec0a11b0 0x7efbec0a3670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x5577c773e2b0 tx=0x7efbfc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.936+0000 7efc08c00700 1 --2- 172.21.15.16:0/2731942198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c773f4d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009d30 tx=0x7efbf8011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38727320e23b67 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efbf5c00700 1 -- 172.21.15.16:0/2731942198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8004220 con 0x5577c773ee70 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efbf5c00700 1 -- 172.21.15.16:0/2731942198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbf8009e80 con 0x5577c773ee70 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efbf5c00700 1 -- 172.21.15.16:0/2731942198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8019670 con 0x5577c773ee70 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/2731942198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 msgr2=0x5577c773f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/2731942198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c773f4d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009d30 tx=0x7efbf8011720 comp rx=0 tx=0).stop 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/2731942198 shutdown_connections 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/2731942198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773fa60 0x5577c773fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/2731942198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 0x5577c773e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/2731942198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c773f4d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.937+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/2731942198 >> 172.21.15.16:0/2731942198 conn(0x5577c76b21d0 msgr2=0x5577c7739050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.938+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/2731942198 shutdown_connections 2024-04-05T23:33:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.938+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/2731942198 wait complete. 2024-04-05T23:33:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.938+0000 7efc1f0b5b80 1 -- start start 2024-04-05T23:33:23.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.938+0000 7efc1f0b5b80 1 Processor -- start 2024-04-05T23:33:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 0x5577c7748180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c77486c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773fa60 0x5577c774f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c768ebf0 con 0x5577c773ee70 2024-04-05T23:33:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c762ad30 con 0x5577c773fa60 2024-04-05T23:33:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c762a9d0 con 0x5577c773e510 2024-04-05T23:33:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc1f0b5b80 10 client.0 _ll_register_callbacks cb 0x5577c7496350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:23.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc03e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c77486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc03e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c77486c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60586/0 (socket says 172.21.15.16:60586) 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc03e00700 1 -- 172.21.15.16:0/867406082 learned_addr learned my addr 172.21.15.16:0/867406082 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc08c00700 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 0x5577c7748180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc09600700 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773fa60 0x5577c774f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc03e00700 1 -- 172.21.15.16:0/867406082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 msgr2=0x5577c7748180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc03e00700 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 0x5577c7748180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc03e00700 1 -- 172.21.15.16:0/867406082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773fa60 msgr2=0x5577c774f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc03e00700 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773fa60 0x5577c774f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.939+0000 7efc03e00700 1 -- 172.21.15.16:0/867406082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf8009960 con 0x5577c773ee70 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.940+0000 7efc03e00700 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c77486c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efbf00058b0 tx=0x7efbf00b2a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1056622ced6e66c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.940+0000 7efbea000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.940+0000 7efbea000700 1 -- 172.21.15.16:0/867406082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf009a960 con 0x5577c773ee70 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.940+0000 7efbea000700 1 -- 172.21.15.16:0/867406082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efbf009b970 con 0x5577c773ee70 2024-04-05T23:33:23.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.940+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/867406082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5577c76876b0 con 0x5577c773ee70 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.940+0000 7efbea000700 1 -- 172.21.15.16:0/867406082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7efbf009abd0 con 0x5577c773ee70 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.940+0000 7efc1f0b5b80 10 client.15174 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.940+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/867406082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5577c74bcb10 con 0x5577c773ee70 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.940+0000 7efbea000700 1 -- 172.21.15.16:0/867406082 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5577c76876b0 con 0x5577c773ee70 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.941+0000 7efbea000700 1 -- 172.21.15.16:0/867406082 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7efbf00b21e0 con 0x5577c773ee70 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.941+0000 7efc1f0b5b80 10 client.15174 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.941+0000 7efc1f0b5b80 10 client.15174 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.941+0000 7efc1f0b5b80 20 client.15174 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.941+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5577c7752e20 0x5577c7758120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.941+0000 7efc1f0b5b80 4 client.15174 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.941+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/867406082 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5577c74ddca0 con 0x5577c7752e20 2024-04-05T23:33:23.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.941+0000 7efc08c00700 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5577c7752e20 0x5577c7758120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.942+0000 7efc08c00700 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5577c7752e20 0x5577c7758120 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8b9b8114ee3a8c4 server_cookie=8852e885d4f5de77 in_seq=0 out_seq=0 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.942+0000 7efbea000700 10 client.15174 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efbea000700 1 -- 172.21.15.16:0/867406082 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5577c74ddca0 con 0x5577c7752e20 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efbea000700 10 client.15174 handle_command_reply: tid=0 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 client.15174 shutdown 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/867406082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 msgr2=0x5577c77486c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c77486c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efbf00058b0 tx=0x7efbf00b2a60 comp rx=0 tx=0).stop 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/867406082 shutdown_connections 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5577c7752e20 0x5577c7758120 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773fa60 0x5577c774f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 0x5577c7748180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efbea000700 0 client.15174 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/867406082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773ee70 0x5577c77486c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/867406082 >> 172.21.15.16:0/867406082 conn(0x5577c76b21d0 msgr2=0x5577c7733410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/867406082 shutdown_connections 2024-04-05T23:33:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/867406082 wait complete. 2024-04-05T23:33:23.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.943+0000 7efc1f0b5b80 20 client.15174 trim_cache size 0 max 16384 2024-04-05T23:33:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.961+0000 7efc1f0b5b80 1 Processor -- start 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc1f0b5b80 1 -- start start 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 0x5577c775c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773ee70 0x5577c774f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c775c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c7512650 con 0x5577c773fa60 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c773aac0 con 0x5577c773e510 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c7754d60 con 0x5577c773ee70 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc09600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c775c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc09600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c775c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60598/0 (socket says 172.21.15.16:60598) 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc09600700 1 -- 172.21.15.16:0/1041673247 learned_addr learned my addr 172.21.15.16:0/1041673247 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc08c00700 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 0x5577c775c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc03e00700 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773ee70 0x5577c774f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc09600700 1 -- 172.21.15.16:0/1041673247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773ee70 msgr2=0x5577c774f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc09600700 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773ee70 0x5577c774f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.962+0000 7efc09600700 1 -- 172.21.15.16:0/1041673247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 msgr2=0x5577c775c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.963+0000 7efc09600700 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 0x5577c775c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.963+0000 7efc09600700 1 -- 172.21.15.16:0/1041673247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577c7654e50 con 0x5577c773fa60 2024-04-05T23:33:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.963+0000 7efc03e00700 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773ee70 0x5577c774f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.963+0000 7efc08c00700 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 0x5577c775c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.963+0000 7efc09600700 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c775c680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0199f0 tx=0x7efbfc009100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291e153ac78fbfe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.963+0000 7efbf6600700 1 -- 172.21.15.16:0/1041673247 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc02b070 con 0x5577c773fa60 2024-04-05T23:33:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.963+0000 7efbf6600700 1 -- 172.21.15.16:0/1041673247 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbfc022040 con 0x5577c773fa60 2024-04-05T23:33:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.963+0000 7efbf6600700 1 -- 172.21.15.16:0/1041673247 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc021050 con 0x5577c773fa60 2024-04-05T23:33:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/1041673247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 msgr2=0x5577c775c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c775c680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0199f0 tx=0x7efbfc009100 comp rx=0 tx=0).stop 2024-04-05T23:33:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/1041673247 shutdown_connections 2024-04-05T23:33:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c775c680 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773e510 0x5577c775c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/1041673247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773ee70 0x5577c774f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/1041673247 >> 172.21.15.16:0/1041673247 conn(0x5577c76b21d0 msgr2=0x5577c7746020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/1041673247 shutdown_connections 2024-04-05T23:33:23.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/1041673247 wait complete. 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 -- start start 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.964+0000 7efc1f0b5b80 1 Processor -- start 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 0x5577c774c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773ee70 0x5577c774cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc1f0b5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c77a7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c774d220 con 0x5577c773fa60 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c77a7c50 con 0x5577c773ee70 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc1f0b5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577c77a7d90 con 0x5577c773e510 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc1f0b5b80 10 client.0 _ll_register_callbacks cb 0x7efbf00975b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc09600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c77a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc09600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c77a7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60610/0 (socket says 172.21.15.16:60610) 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc09600700 1 -- 172.21.15.16:0/4005761832 learned_addr learned my addr 172.21.15.16:0/4005761832 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc03e00700 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773ee70 0x5577c774cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc08c00700 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 0x5577c774c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc09600700 1 -- 172.21.15.16:0/4005761832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 msgr2=0x5577c774c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc09600700 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 0x5577c774c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc09600700 1 -- 172.21.15.16:0/4005761832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773ee70 msgr2=0x5577c774cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc09600700 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773ee70 0x5577c774cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.965+0000 7efc09600700 1 -- 172.21.15.16:0/4005761832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577c7510360 con 0x5577c773fa60 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.966+0000 7efc09600700 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c77a7710 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efbfc005bd0 tx=0x7efbfc01b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fb09cf44971344 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.966+0000 7efbea000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.966+0000 7efbea000700 1 -- 172.21.15.16:0/4005761832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc02b040 con 0x5577c773fa60 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.966+0000 7efbea000700 1 -- 172.21.15.16:0/4005761832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efbfc01be00 con 0x5577c773fa60 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.966+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/4005761832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5577c7731440 con 0x5577c773fa60 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.966+0000 7efbea000700 1 -- 172.21.15.16:0/4005761832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7efbfc021020 con 0x5577c773fa60 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.966+0000 7efc1f0b5b80 10 client.15180 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.966+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/4005761832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7efbfc019730 con 0x5577c773fa60 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.966+0000 7efbea000700 1 -- 172.21.15.16:0/4005761832 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5577c7731440 con 0x5577c773fa60 2024-04-05T23:33:23.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.967+0000 7efbea000700 1 -- 172.21.15.16:0/4005761832 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7efbfc02f800 con 0x5577c773fa60 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.967+0000 7efc1f0b5b80 10 client.15180 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.967+0000 7efc1f0b5b80 10 client.15180 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.967+0000 7efc1f0b5b80 20 client.15180 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.967+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5577c77acb00 0x5577c77a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.967+0000 7efc1f0b5b80 4 client.15180 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.967+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/4005761832 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5577c7512c90 con 0x5577c77acb00 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.967+0000 7efc08c00700 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5577c77acb00 0x5577c77a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efbea000700 10 client.15180 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc08c00700 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5577c77acb00 0x5577c77a94d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2bd9bd9418f4390f server_cookie=8373f52355f8177f in_seq=0 out_seq=0 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efbea000700 1 -- 172.21.15.16:0/4005761832 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5577c7512c90 con 0x5577c77acb00 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efbea000700 10 client.15180 handle_command_reply: tid=0 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 client.15180 shutdown 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/4005761832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 msgr2=0x5577c77a7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c77a7710 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efbfc005bd0 tx=0x7efbfc01b7c0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/4005761832 shutdown_connections 2024-04-05T23:33:23.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5577c77acb00 0x5577c77a94d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5577c773fa60 0x5577c77a7710 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5577c773e510 0x5577c774c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efbea000700 0 client.15180 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 --2- 172.21.15.16:0/4005761832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5577c773ee70 0x5577c774cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/4005761832 >> 172.21.15.16:0/4005761832 conn(0x5577c76b21d0 msgr2=0x5577c775c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/4005761832 shutdown_connections 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.968+0000 7efc1f0b5b80 1 -- 172.21.15.16:0/4005761832 wait complete. 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc1f0b5b80 20 client.15180 trim_cache size 0 max 16384 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:23.964 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24526, 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 74.965191447999999, 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 96.769665294000006 2024-04-05T23:33:23.965 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 -- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efbec0a11b0 msgr2=0x7efbec0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efbec0a11b0 0x7efbec0a3670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x5577c773e2b0 tx=0x7efbfc00b040 comp rx=0 tx=0).stop 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 -- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 msgr2=0x7efc040732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 0x7efc040732d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efbf0000c00 tx=0x7efbf0005740 comp rx=0 tx=0).stop 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 -- 172.21.15.16:0/4257298932 shutdown_connections 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7efbec0a11b0 0x7efbec0a3670 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc04102a00 0x7efc04073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc04100e60 0x7efc04074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 --2- 172.21.15.16:0/4257298932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc04102040 0x7efc040732d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 -- 172.21.15.16:0/4257298932 >> 172.21.15.16:0/4257298932 conn(0x7efc040fc440 msgr2=0x7efc04105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 -- 172.21.15.16:0/4257298932 shutdown_connections 2024-04-05T23:33:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:23.969+0000 7efc0b600700 1 -- 172.21.15.16:0/4257298932 wait complete. 2024-04-05T23:33:23.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-05T23:33:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.188+0000 7f5887800700 1 -- 172.21.15.16:0/2909603787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5880102040 msgr2=0x7f58801024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.188+0000 7f5887800700 1 --2- 172.21.15.16:0/2909603787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5880102040 0x7f58801024c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f586c009990 tx=0x7f586c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.188+0000 7f5887800700 1 -- 172.21.15.16:0/2909603787 shutdown_connections 2024-04-05T23:33:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.188+0000 7f5887800700 1 --2- 172.21.15.16:0/2909603787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5880102a00 0x7f5880109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.188+0000 7f5887800700 1 --2- 172.21.15.16:0/2909603787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5880100e60 0x7f5880101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.188+0000 7f5887800700 1 --2- 172.21.15.16:0/2909603787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5880102040 0x7f58801024c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.188+0000 7f5887800700 1 -- 172.21.15.16:0/2909603787 >> 172.21.15.16:0/2909603787 conn(0x7f58800fc440 msgr2=0x7f58800fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.188+0000 7f5887800700 1 -- 172.21.15.16:0/2909603787 shutdown_connections 2024-04-05T23:33:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.188+0000 7f5887800700 1 -- 172.21.15.16:0/2909603787 wait complete. 2024-04-05T23:33:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.190+0000 7f5887800700 1 Processor -- start 2024-04-05T23:33:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.190+0000 7f5887800700 1 -- start start 2024-04-05T23:33:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.190+0000 7f5887800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5880100e60 0x7f5880196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.190+0000 7f5887800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5880102040 0x7f58801973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.190+0000 7f5887800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5880102a00 0x7f588019e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.190+0000 7f5887800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588019e990 con 0x7f5880100e60 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.190+0000 7f5887800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588019ead0 con 0x7f5880102a00 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.190+0000 7f5887800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588019ec60 con 0x7f5880102040 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5884e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5880100e60 0x7f5880196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5884e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5880100e60 0x7f5880196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60630/0 (socket says 172.21.15.16:60630) 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5884e00700 1 -- 172.21.15.16:0/2073768984 learned_addr learned my addr 172.21.15.16:0/2073768984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f587fe00700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5880102040 0x7f58801973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5885800700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5880102a00 0x7f588019e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5884e00700 1 -- 172.21.15.16:0/2073768984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5880102040 msgr2=0x7f58801973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5884e00700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5880102040 0x7f58801973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5884e00700 1 -- 172.21.15.16:0/2073768984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5880102a00 msgr2=0x7f588019e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5884e00700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5880102a00 0x7f588019e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5884e00700 1 -- 172.21.15.16:0/2073768984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c0095f0 con 0x7f5880100e60 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f5884e00700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5880100e60 0x7f5880196e60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f587400e970 tx=0x7f587400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f587d600700 1 -- 172.21.15.16:0/2073768984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874009790 con 0x7f5880100e60 2024-04-05T23:33:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.191+0000 7f587d600700 1 -- 172.21.15.16:0/2073768984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f587400c7e0 con 0x7f5880100e60 2024-04-05T23:33:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.192+0000 7f5887800700 1 -- 172.21.15.16:0/2073768984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588019ef40 con 0x7f5880100e60 2024-04-05T23:33:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.192+0000 7f587d600700 1 -- 172.21.15.16:0/2073768984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874003c70 con 0x7f5880100e60 2024-04-05T23:33:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.192+0000 7f5887800700 1 -- 172.21.15.16:0/2073768984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588019f370 con 0x7f5880100e60 2024-04-05T23:33:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.195+0000 7f587d600700 1 -- 172.21.15.16:0/2073768984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7f587400cc50 con 0x7f5880100e60 2024-04-05T23:33:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f589b2c2b80 1 Processor -- start 2024-04-05T23:33:24.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f589b2c2b80 1 -- start start 2024-04-05T23:33:24.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f587d600700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58680a1230 0x7f58680a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf36930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f587d600700 1 -- 172.21.15.16:0/2073768984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5874014070 con 0x7f5880100e60 2024-04-05T23:33:24.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36e70 0x55e0faf374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf37a60 0x55e0faf37ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0fae9cb90 con 0x55e0faf37a60 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0fae8d940 con 0x55e0faf36510 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0fae8b300 con 0x55e0faf36e70 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.197+0000 7f5885800700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58680a1230 0x7f58680a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5884e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf36930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5884e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf36930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38402/0 (socket says 172.21.15.16:38402) 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5884e00700 1 -- 172.21.15.16:0/2867329852 learned_addr learned my addr 172.21.15.16:0/2867329852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5885800700 1 --2- 172.21.15.16:0/2867329852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf37a60 0x55e0faf37ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f587fe00700 1 --2- 172.21.15.16:0/2867329852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36e70 0x55e0faf374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5884e00700 1 -- 172.21.15.16:0/2867329852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36e70 msgr2=0x55e0faf374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5884e00700 1 --2- 172.21.15.16:0/2867329852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36e70 0x55e0faf374d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5884e00700 1 -- 172.21.15.16:0/2867329852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf37a60 msgr2=0x55e0faf37ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5884e00700 1 --2- 172.21.15.16:0/2867329852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf37a60 0x55e0faf37ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5884e00700 1 -- 172.21.15.16:0/2867329852 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0fad08360 con 0x55e0faf36510 2024-04-05T23:33:24.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5885800700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58680a1230 0x7f58680a36f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f587800f870 tx=0x7f587800fc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5884e00700 1 --2- 172.21.15.16:0/2867329852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf36930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5874096370 tx=0x7f58740998a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbcec4b2b86f94a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5872000700 1 -- 172.21.15.16:0/2867329852 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58740b1070 con 0x55e0faf36510 2024-04-05T23:33:24.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5872000700 1 -- 172.21.15.16:0/2867329852 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f587409dca0 con 0x55e0faf36510 2024-04-05T23:33:24.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.198+0000 7f5872000700 1 -- 172.21.15.16:0/2867329852 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587409a0d0 con 0x55e0faf36510 2024-04-05T23:33:24.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.199+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2867329852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 msgr2=0x55e0faf36930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.199+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2867329852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf36930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5874096370 tx=0x7f58740998a0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.199+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2867329852 shutdown_connections 2024-04-05T23:33:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.199+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2867329852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf37a60 0x55e0faf37ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.199+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2867329852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf36930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.199+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2867329852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36e70 0x55e0faf374d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.199+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2867329852 >> 172.21.15.16:0/2867329852 conn(0x55e0faeaa1d0 msgr2=0x55e0faf31050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.199+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2867329852 shutdown_connections 2024-04-05T23:33:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.199+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2867329852 wait complete. 2024-04-05T23:33:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.200+0000 7f589b2c2b80 1 -- start start 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.200+0000 7f589b2c2b80 1 Processor -- start 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.200+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36510 0x55e0faf40090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.200+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36e70 0x55e0faf405d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.200+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf47610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0fae86bf0 con 0x55e0faf36510 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0fae22d30 con 0x55e0faf36e70 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0fae229d0 con 0x55e0faf37a60 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f589b2c2b80 10 client.0 _ll_register_callbacks cb 0x55e0fac8e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f5884e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36510 0x55e0faf40090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f5885800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf47610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f587fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36e70 0x55e0faf405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f5885800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf47610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43634/0 (socket says 172.21.15.16:43634) 2024-04-05T23:33:24.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f5885800700 1 -- 172.21.15.16:0/2943469266 learned_addr learned my addr 172.21.15.16:0/2943469266 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f587fe00700 1 -- 172.21.15.16:0/2943469266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 msgr2=0x55e0faf47610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f587fe00700 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf47610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f587fe00700 1 -- 172.21.15.16:0/2943469266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36510 msgr2=0x55e0faf40090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f587fe00700 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36510 0x55e0faf40090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f587fe00700 1 -- 172.21.15.16:0/2943469266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588019f370 con 0x55e0faf36e70 2024-04-05T23:33:24.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.201+0000 7f5885800700 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf47610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.202+0000 7f587fe00700 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36e70 0x55e0faf405d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f586c000c00 tx=0x7f586c00bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35d27d2ec4e7f718 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.202+0000 7f5866a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.202+0000 7f5866a00700 1 -- 172.21.15.16:0/2943469266 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c01d030 con 0x55e0faf36e70 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.202+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2943469266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e0faf29440 con 0x55e0faf36e70 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.202+0000 7f5866a00700 1 -- 172.21.15.16:0/2943469266 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f586c0058e0 con 0x55e0faf36e70 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.202+0000 7f5866a00700 1 -- 172.21.15.16:0/2943469266 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f586c004500 con 0x55e0faf36e70 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.202+0000 7f589b2c2b80 10 client.25217 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.202+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2943469266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e0facb4b10 con 0x55e0faf36e70 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.202+0000 7f5866a00700 1 -- 172.21.15.16:0/2943469266 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e0faf29440 con 0x55e0faf36e70 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.203+0000 7f5866a00700 1 -- 172.21.15.16:0/2943469266 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f586c00f760 con 0x55e0faf36e70 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.203+0000 7f589b2c2b80 10 client.25217 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.203+0000 7f589b2c2b80 10 client.25217 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-05T23:33:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.203+0000 7f589b2c2b80 20 client.25217 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.203+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55e0faf4b220 0x55e0faf50520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.203+0000 7f589b2c2b80 4 client.25217 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.203+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2943469266 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e0fae7f6b0 con 0x55e0faf4b220 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.203+0000 7f5884e00700 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55e0faf4b220 0x55e0faf50520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f5884e00700 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55e0faf4b220 0x55e0faf50520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6392bd8f5f6ca6b7 server_cookie=9ed2f8c2de4a1fd3 in_seq=0 out_seq=0 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f5866a00700 10 client.25217 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f5866a00700 1 -- 172.21.15.16:0/2943469266 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e0fae7f6b0 con 0x55e0faf4b220 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f5866a00700 10 client.25217 handle_command_reply: tid=0 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f589b2c2b80 1 client.25217 shutdown 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2943469266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36e70 msgr2=0x55e0faf405d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36e70 0x55e0faf405d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f586c000c00 tx=0x7f586c00bf60 comp rx=0 tx=0).stop 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2943469266 shutdown_connections 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36510 0x55e0faf40090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.204+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55e0faf4b220 0x55e0faf50520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.205+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf47610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.205+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2943469266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36e70 0x55e0faf405d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.205+0000 7f5866a00700 0 client.25217 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:33:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.205+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2943469266 >> 172.21.15.16:0/2943469266 conn(0x55e0faeaa1d0 msgr2=0x55e0faf2b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.205+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2943469266 shutdown_connections 2024-04-05T23:33:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.205+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2943469266 wait complete. 2024-04-05T23:33:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.205+0000 7f589b2c2b80 20 client.25217 trim_cache size 0 max 16384 2024-04-05T23:33:24.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.224+0000 7f589b2c2b80 1 Processor -- start 2024-04-05T23:33:24.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.224+0000 7f589b2c2b80 1 -- start start 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.224+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf3f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.224+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 0x55e0faf475d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.224+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf3e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.224+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0fad0a650 con 0x55e0faf36e70 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.224+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0fae7f6e0 con 0x55e0faf36510 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.224+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0fae7f990 con 0x55e0faf37a60 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f587fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 0x55e0faf475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5884e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf3f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5884e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf3f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38414/0 (socket says 172.21.15.16:38414) 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5884e00700 1 -- 172.21.15.16:0/1708214619 learned_addr learned my addr 172.21.15.16:0/1708214619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5885800700 1 --2- 172.21.15.16:0/1708214619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf3e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5884e00700 1 -- 172.21.15.16:0/1708214619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 msgr2=0x55e0faf3e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5884e00700 1 --2- 172.21.15.16:0/1708214619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf3e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5884e00700 1 -- 172.21.15.16:0/1708214619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 msgr2=0x55e0faf475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5884e00700 1 --2- 172.21.15.16:0/1708214619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 0x55e0faf475d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5884e00700 1 -- 172.21.15.16:0/1708214619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0fae4ce50 con 0x55e0faf36510 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f587fe00700 1 --2- 172.21.15.16:0/1708214619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 0x55e0faf475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5884e00700 1 --2- 172.21.15.16:0/1708214619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf3f5a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5874099b70 tx=0x7f58740ca780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1486f30cfd8fa34 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5872a00700 1 -- 172.21.15.16:0/1708214619 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58740b1040 con 0x55e0faf36510 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5872a00700 1 -- 172.21.15.16:0/1708214619 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58740ca960 con 0x55e0faf36510 2024-04-05T23:33:24.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.225+0000 7f5872a00700 1 -- 172.21.15.16:0/1708214619 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587409a330 con 0x55e0faf36510 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.226+0000 7f589b2c2b80 1 -- 172.21.15.16:0/1708214619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 msgr2=0x55e0faf3f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.226+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/1708214619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf3f5a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5874099b70 tx=0x7f58740ca780 comp rx=0 tx=0).stop 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.226+0000 7f589b2c2b80 1 -- 172.21.15.16:0/1708214619 shutdown_connections 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.226+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/1708214619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf37a60 0x55e0faf3e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.226+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/1708214619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf36510 0x55e0faf3f5a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.226+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/1708214619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 0x55e0faf475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.226+0000 7f589b2c2b80 1 -- 172.21.15.16:0/1708214619 >> 172.21.15.16:0/1708214619 conn(0x55e0faeaa1d0 msgr2=0x55e0faf31050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.226+0000 7f589b2c2b80 1 -- 172.21.15.16:0/1708214619 shutdown_connections 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.226+0000 7f589b2c2b80 1 -- 172.21.15.16:0/1708214619 wait complete. 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.227+0000 7f589b2c2b80 1 -- start start 2024-04-05T23:33:24.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.227+0000 7f589b2c2b80 1 Processor -- start 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36510 0x55e0faf43840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 0x55e0faf43d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f589b2c2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf37a60 0x55e0faf46c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0faf47150 con 0x55e0faf36e70 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0faf472c0 con 0x55e0faf37a60 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f589b2c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0faf47430 con 0x55e0faf36510 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f589b2c2b80 10 client.0 _ll_register_callbacks cb 0x7f586c009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f587fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 0x55e0faf43d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5885800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf37a60 0x55e0faf46c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5884e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36510 0x55e0faf43840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5884e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36510 0x55e0faf43840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43658/0 (socket says 172.21.15.16:43658) 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5884e00700 1 -- 172.21.15.16:0/2268924463 learned_addr learned my addr 172.21.15.16:0/2268924463 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5885800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf37a60 0x55e0faf46c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38418/0 (socket says 172.21.15.16:38418) 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5884e00700 1 -- 172.21.15.16:0/2268924463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf37a60 msgr2=0x55e0faf46c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5884e00700 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf37a60 0x55e0faf46c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5884e00700 1 -- 172.21.15.16:0/2268924463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 msgr2=0x55e0faf43d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5884e00700 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 0x55e0faf43d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.228+0000 7f5884e00700 1 -- 172.21.15.16:0/2268924463 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e0facb4b10 con 0x55e0faf36510 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f5884e00700 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36510 0x55e0faf43840 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f58740981c0 tx=0x7f5874005ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d595a23f4f53ca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f5866a00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f5866a00700 1 -- 172.21.15.16:0/2268924463 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58740b1070 con 0x55e0faf36510 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f5866a00700 1 -- 172.21.15.16:0/2268924463 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5874099540 con 0x55e0faf36510 2024-04-05T23:33:24.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2268924463 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e0facd5ca0 con 0x55e0faf36510 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f5866a00700 1 -- 172.21.15.16:0/2268924463 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5874096580 con 0x55e0faf36510 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f589b2c2b80 10 client.25243 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2268924463 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e0fad08360 con 0x55e0faf36510 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f5866a00700 1 -- 172.21.15.16:0/2268924463 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e0facd5ca0 con 0x55e0faf36510 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f5866a00700 1 -- 172.21.15.16:0/2268924463 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f5874004220 con 0x55e0faf36510 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f589b2c2b80 10 client.25243 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f589b2c2b80 10 client.25243 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f589b2c2b80 20 client.25243 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55e0fafa3bf0 0x55e0fafa0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f589b2c2b80 4 client.25243 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.229+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2268924463 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e0fad0ac90 con 0x55e0fafa3bf0 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.230+0000 7f587fe00700 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55e0fafa3bf0 0x55e0fafa0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.230+0000 7f5866a00700 10 client.25243 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:33:24.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.230+0000 7f587fe00700 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55e0fafa3bf0 0x55e0fafa0730 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=543241230ddd1a14 server_cookie=1de1e0efb96635ae in_seq=0 out_seq=0 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.230+0000 7f5866a00700 1 -- 172.21.15.16:0/2268924463 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55e0fad0ac90 con 0x55e0fafa3bf0 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.230+0000 7f5866a00700 10 client.25243 handle_command_reply: tid=0 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.230+0000 7f589b2c2b80 1 client.25243 shutdown 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2268924463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36510 msgr2=0x55e0faf43840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36510 0x55e0faf43840 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f58740981c0 tx=0x7f5874005ba0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2268924463 shutdown_connections 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55e0fafa3bf0 0x55e0fafa0730 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0faf37a60 0x55e0faf46c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0faf36510 0x55e0faf43840 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 --2- 172.21.15.16:0/2268924463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0faf36e70 0x55e0faf43d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f5866a00700 0 client.25243 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2268924463 >> 172.21.15.16:0/2268924463 conn(0x55e0faeaa1d0 msgr2=0x55e0faf3ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2268924463 shutdown_connections 2024-04-05T23:33:24.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 1 -- 172.21.15.16:0/2268924463 wait complete. 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.231+0000 7f589b2c2b80 20 client.25243 trim_cache size 0 max 16384 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24550, 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 76.232700914000006, 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 93.357643073999995 2024-04-05T23:33:24.230 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:24.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 -- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58680a1230 msgr2=0x7f58680a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58680a1230 0x7f58680a36f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f587800f870 tx=0x7f587800fc30 comp rx=0 tx=0).stop 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 -- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5880100e60 msgr2=0x7f5880196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5880100e60 0x7f5880196e60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f587400e970 tx=0x7f587400ec80 comp rx=0 tx=0).stop 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 -- 172.21.15.16:0/2073768984 shutdown_connections 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f58680a1230 0x7f58680a36f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5880100e60 0x7f5880196e60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5880102a00 0x7f588019e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 --2- 172.21.15.16:0/2073768984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5880102040 0x7f58801973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 -- 172.21.15.16:0/2073768984 >> 172.21.15.16:0/2073768984 conn(0x7f58800fc440 msgr2=0x7f58801054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 -- 172.21.15.16:0/2073768984 shutdown_connections 2024-04-05T23:33:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.232+0000 7f5887800700 1 -- 172.21.15.16:0/2073768984 wait complete. 2024-04-05T23:33:24.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-05T23:33:24.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.449+0000 7fa065400700 1 -- 172.21.15.16:0/1573561451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060100e60 msgr2=0x7fa060101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.449+0000 7fa065400700 1 --2- 172.21.15.16:0/1573561451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060100e60 0x7fa060101280 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa050009990 tx=0x7fa050009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.450+0000 7fa065400700 1 -- 172.21.15.16:0/1573561451 shutdown_connections 2024-04-05T23:33:24.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.450+0000 7fa065400700 1 --2- 172.21.15.16:0/1573561451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060100e60 0x7fa060101280 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.450+0000 7fa065400700 1 --2- 172.21.15.16:0/1573561451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa060102a00 0x7fa060109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.450+0000 7fa065400700 1 --2- 172.21.15.16:0/1573561451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa060102040 0x7fa0601024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.450+0000 7fa065400700 1 -- 172.21.15.16:0/1573561451 >> 172.21.15.16:0/1573561451 conn(0x7fa0600fc440 msgr2=0x7fa0600fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.450+0000 7fa065400700 1 -- 172.21.15.16:0/1573561451 shutdown_connections 2024-04-05T23:33:24.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.450+0000 7fa065400700 1 -- 172.21.15.16:0/1573561451 wait complete. 2024-04-05T23:33:24.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.451+0000 7fa065400700 1 Processor -- start 2024-04-05T23:33:24.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.451+0000 7fa065400700 1 -- start start 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.451+0000 7fa065400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa060100e60 0x7fa0601990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.451+0000 7fa065400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa060102040 0x7fa060199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.451+0000 7fa065400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060102a00 0x7fa0601a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.451+0000 7fa065400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0601a0bd0 con 0x7fa060102a00 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.451+0000 7fa065400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0601a0d10 con 0x7fa060102040 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.451+0000 7fa065400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0601a0ea0 con 0x7fa060100e60 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.451+0000 7fa05f200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060102a00 0x7fa0601a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05e800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa060100e60 0x7fa0601990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05e800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa060100e60 0x7fa0601990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43678/0 (socket says 172.21.15.16:43678) 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05f200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060102a00 0x7fa0601a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60690/0 (socket says 172.21.15.16:60690) 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05e800700 1 -- 172.21.15.16:0/4004800277 learned_addr learned my addr 172.21.15.16:0/4004800277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05de00700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa060102040 0x7fa060199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05e800700 1 -- 172.21.15.16:0/4004800277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa060102040 msgr2=0x7fa060199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05e800700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa060102040 0x7fa060199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05e800700 1 -- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060102a00 msgr2=0x7fa0601a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05e800700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060102a00 0x7fa0601a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05e800700 1 -- 172.21.15.16:0/4004800277 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0500095f0 con 0x7fa060100e60 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05f200700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060102a00 0x7fa0601a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa05e800700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa060100e60 0x7fa0601990f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa050005950 tx=0x7fa050009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.452+0000 7fa057400700 1 -- 172.21.15.16:0/4004800277 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05001d070 con 0x7fa060100e60 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.453+0000 7fa065400700 1 -- 172.21.15.16:0/4004800277 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0601a1120 con 0x7fa060100e60 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.453+0000 7fa057400700 1 -- 172.21.15.16:0/4004800277 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa050021410 con 0x7fa060100e60 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.453+0000 7fa057400700 1 -- 172.21.15.16:0/4004800277 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa050017650 con 0x7fa060100e60 2024-04-05T23:33:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.453+0000 7fa065400700 1 -- 172.21.15.16:0/4004800277 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0601a1550 con 0x7fa060100e60 2024-04-05T23:33:24.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.456+0000 7fa057400700 1 -- 172.21.15.16:0/4004800277 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fa050021580 con 0x7fa060100e60 2024-04-05T23:33:24.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.457+0000 7fa078e32b80 1 Processor -- start 2024-04-05T23:33:24.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa078e32b80 1 -- start start 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa057400700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa0440a11b0 0x7fa0440a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 0x555847d7f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d804d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d80a60 0x555847d80ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa078e32b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847ce5b90 con 0x555847d80a60 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa078e32b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847cd6940 con 0x555847d7f510 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa078e32b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847cd4300 con 0x555847d7fe70 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa057400700 1 -- 172.21.15.16:0/4004800277 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa0500cdd20 con 0x7fa060100e60 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa05e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 0x555847d7f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa05de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d804d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa05de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d804d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43680/0 (socket says 172.21.15.16:43680) 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa05e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 0x555847d7f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38448/0 (socket says 172.21.15.16:38448) 2024-04-05T23:33:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa05de00700 1 -- 172.21.15.16:0/3816666306 learned_addr learned my addr 172.21.15.16:0/3816666306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa05f200700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa0440a11b0 0x7fa0440a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.458+0000 7fa05f200700 1 --2- 172.21.15.16:0/3816666306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d80a60 0x555847d80ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa05de00700 1 -- 172.21.15.16:0/3816666306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 msgr2=0x555847d7f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa05de00700 1 --2- 172.21.15.16:0/3816666306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 0x555847d7f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa05de00700 1 -- 172.21.15.16:0/3816666306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d80a60 msgr2=0x555847d80ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa05de00700 1 --2- 172.21.15.16:0/3816666306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d80a60 0x555847d80ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa05de00700 1 -- 172.21.15.16:0/3816666306 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555847b51360 con 0x555847d7fe70 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa05de00700 1 --2- 172.21.15.16:0/3816666306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d804d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009d30 tx=0x7fa04c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ffe1e4dc8f70f04 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa05f200700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa0440a11b0 0x7fa0440a3670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa058009fd0 tx=0x7fa058009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa04b400700 1 -- 172.21.15.16:0/3816666306 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c004220 con 0x555847d7fe70 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa04b400700 1 -- 172.21.15.16:0/3816666306 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa04c004380 con 0x555847d7fe70 2024-04-05T23:33:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.459+0000 7fa04b400700 1 -- 172.21.15.16:0/3816666306 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c019810 con 0x555847d7fe70 2024-04-05T23:33:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 -- 172.21.15.16:0/3816666306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 msgr2=0x555847d804d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 --2- 172.21.15.16:0/3816666306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d804d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009d30 tx=0x7fa04c011720 comp rx=0 tx=0).stop 2024-04-05T23:33:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 -- 172.21.15.16:0/3816666306 shutdown_connections 2024-04-05T23:33:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 --2- 172.21.15.16:0/3816666306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d80a60 0x555847d80ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 --2- 172.21.15.16:0/3816666306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 0x555847d7f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 --2- 172.21.15.16:0/3816666306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d804d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 -- 172.21.15.16:0/3816666306 >> 172.21.15.16:0/3816666306 conn(0x555847cf31d0 msgr2=0x555847d7a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 -- 172.21.15.16:0/3816666306 shutdown_connections 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 -- 172.21.15.16:0/3816666306 wait complete. 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.460+0000 7fa078e32b80 1 -- start start 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.461+0000 7fa078e32b80 1 Processor -- start 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.461+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 0x555847d89030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.461+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7fe70 0x555847d89570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.461+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 0x555847d905b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.461+0000 7fa078e32b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847ccfbf0 con 0x555847d7fe70 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.461+0000 7fa078e32b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847c6bd30 con 0x555847d7f510 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.461+0000 7fa078e32b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847c6b9d0 con 0x555847d80a60 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.461+0000 7fa078e32b80 10 client.0 _ll_register_callbacks cb 0x555847ad7350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7fe70 0x555847d89570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7fe70 0x555847d89570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60704/0 (socket says 172.21.15.16:60704) 2024-04-05T23:33:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05de00700 1 -- 172.21.15.16:0/1265386084 learned_addr learned my addr 172.21.15.16:0/1265386084 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05f200700 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 0x555847d905b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05e800700 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 0x555847d89030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05de00700 1 -- 172.21.15.16:0/1265386084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 msgr2=0x555847d905b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05de00700 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 0x555847d905b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05de00700 1 -- 172.21.15.16:0/1265386084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 msgr2=0x555847d89030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05de00700 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 0x555847d89030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05de00700 1 -- 172.21.15.16:0/1265386084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c009960 con 0x555847d7fe70 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa05de00700 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7fe70 0x555847d89570 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa04c006010 tx=0x7fa04c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72ed387e9ca8667 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa03fe00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.462+0000 7fa03fe00700 1 -- 172.21.15.16:0/1265386084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c005040 con 0x555847d7fe70 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa03fe00700 1 -- 172.21.15.16:0/1265386084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa04c0051a0 con 0x555847d7fe70 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa078e32b80 1 -- 172.21.15.16:0/1265386084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555847d72440 con 0x555847d7fe70 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa03fe00700 1 -- 172.21.15.16:0/1265386084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa04c005360 con 0x555847d7fe70 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa078e32b80 10 client.15210 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa078e32b80 1 -- 172.21.15.16:0/1265386084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555847afdb10 con 0x555847d7fe70 2024-04-05T23:33:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa03fe00700 1 -- 172.21.15.16:0/1265386084 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555847d72440 con 0x555847d7fe70 2024-04-05T23:33:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa03fe00700 1 -- 172.21.15.16:0/1265386084 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fa04c021040 con 0x555847d7fe70 2024-04-05T23:33:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa078e32b80 10 client.15210 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa078e32b80 10 client.15210 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-05T23:33:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa078e32b80 20 client.15210 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa078e32b80 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555847d94130 0x555847d99450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa078e32b80 4 client.15210 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.463+0000 7fa078e32b80 1 -- 172.21.15.16:0/1265386084 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555847cc86b0 con 0x555847d94130 2024-04-05T23:33:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.464+0000 7fa05e800700 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555847d94130 0x555847d99450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.464+0000 7fa05e800700 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555847d94130 0x555847d99450 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12cdc06038eb5256 server_cookie=b53dea2392b89fdc in_seq=0 out_seq=0 2024-04-05T23:33:24.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.464+0000 7fa03fe00700 10 client.15210 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:24.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa03fe00700 1 -- 172.21.15.16:0/1265386084 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555847cc86b0 con 0x555847d94130 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa03fe00700 10 client.15210 handle_command_reply: tid=0 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 client.15210 shutdown 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 -- 172.21.15.16:0/1265386084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7fe70 msgr2=0x555847d89570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7fe70 0x555847d89570 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa04c006010 tx=0x7fa04c003680 comp rx=0 tx=0).stop 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 -- 172.21.15.16:0/1265386084 shutdown_connections 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555847d94130 0x555847d99450 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 0x555847d905b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7f510 0x555847d89030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa03fe00700 0 client.15210 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 --2- 172.21.15.16:0/1265386084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7fe70 0x555847d89570 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 -- 172.21.15.16:0/1265386084 >> 172.21.15.16:0/1265386084 conn(0x555847cf31d0 msgr2=0x555847d74410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 -- 172.21.15.16:0/1265386084 shutdown_connections 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.465+0000 7fa078e32b80 1 -- 172.21.15.16:0/1265386084 wait complete. 2024-04-05T23:33:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.466+0000 7fa078e32b80 20 client.15210 trim_cache size 0 max 16384 2024-04-05T23:33:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.484+0000 7fa078e32b80 1 Processor -- start 2024-04-05T23:33:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.484+0000 7fa078e32b80 1 -- start start 2024-04-05T23:33:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.484+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d885b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.484+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d89dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.484+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d80a60 0x555847d90570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.484+0000 7fa078e32b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847b53650 con 0x555847d7f510 2024-04-05T23:33:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.484+0000 7fa078e32b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847d76420 con 0x555847d80a60 2024-04-05T23:33:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.484+0000 7fa078e32b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847cc86b0 con 0x555847d7fe70 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.484+0000 7fa05e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d885b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d80a60 0x555847d90570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d885b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60714/0 (socket says 172.21.15.16:60714) 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05e800700 1 -- 172.21.15.16:0/358519465 learned_addr learned my addr 172.21.15.16:0/358519465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05de00700 1 --2- 172.21.15.16:0/358519465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d89dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05f200700 1 -- 172.21.15.16:0/358519465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 msgr2=0x555847d89dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05f200700 1 --2- 172.21.15.16:0/358519465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d89dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05f200700 1 -- 172.21.15.16:0/358519465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 msgr2=0x555847d885b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05f200700 1 --2- 172.21.15.16:0/358519465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d885b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05f200700 1 -- 172.21.15.16:0/358519465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555847c95e50 con 0x555847d80a60 2024-04-05T23:33:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa05f200700 1 --2- 172.21.15.16:0/358519465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d80a60 0x555847d90570 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa058009550 tx=0x7fa05800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80702cb0874d8bcf server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.485+0000 7fa04be00700 1 -- 172.21.15.16:0/358519465 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05801a6c0 con 0x555847d80a60 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa04be00700 1 -- 172.21.15.16:0/358519465 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa05800f040 con 0x555847d80a60 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa04be00700 1 -- 172.21.15.16:0/358519465 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058025cc0 con 0x555847d80a60 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa078e32b80 1 -- 172.21.15.16:0/358519465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d80a60 msgr2=0x555847d90570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa078e32b80 1 --2- 172.21.15.16:0/358519465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d80a60 0x555847d90570 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa058009550 tx=0x7fa05800b040 comp rx=0 tx=0).stop 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa078e32b80 1 -- 172.21.15.16:0/358519465 shutdown_connections 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa078e32b80 1 --2- 172.21.15.16:0/358519465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d885b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa078e32b80 1 --2- 172.21.15.16:0/358519465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d80a60 0x555847d90570 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa078e32b80 1 --2- 172.21.15.16:0/358519465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d7fe70 0x555847d89dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa078e32b80 1 -- 172.21.15.16:0/358519465 >> 172.21.15.16:0/358519465 conn(0x555847cf31d0 msgr2=0x555847d7a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa078e32b80 1 -- 172.21.15.16:0/358519465 shutdown_connections 2024-04-05T23:33:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.486+0000 7fa078e32b80 1 -- 172.21.15.16:0/358519465 wait complete. 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.487+0000 7fa078e32b80 1 -- start start 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.487+0000 7fa078e32b80 1 Processor -- start 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.487+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d90050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.487+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7fe70 0x555847d90590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.487+0000 7fa078e32b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 0x555847d8cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.487+0000 7fa078e32b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847d8d2e0 con 0x555847d7f510 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.487+0000 7fa078e32b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847d8d450 con 0x555847d7fe70 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.487+0000 7fa078e32b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555847d8d610 con 0x555847d80a60 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.487+0000 7fa078e32b80 10 client.0 _ll_register_callbacks cb 0x7fa050005c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d90050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d90050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60730/0 (socket says 172.21.15.16:60730) 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05e800700 1 -- 172.21.15.16:0/2761752949 learned_addr learned my addr 172.21.15.16:0/2761752949 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05f200700 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 0x555847d8cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05de00700 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7fe70 0x555847d90590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05e800700 1 -- 172.21.15.16:0/2761752949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 msgr2=0x555847d8cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05e800700 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 0x555847d8cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05e800700 1 -- 172.21.15.16:0/2761752949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7fe70 msgr2=0x555847d90590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05e800700 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7fe70 0x555847d90590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05e800700 1 -- 172.21.15.16:0/2761752949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa058008f30 con 0x555847d7f510 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa05e800700 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d90050 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa05009af90 tx=0x7fa050005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34efe70390904b1c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa03fe00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa03fe00700 1 -- 172.21.15.16:0/2761752949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0500be070 con 0x555847d7f510 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa03fe00700 1 -- 172.21.15.16:0/2761752949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa05009b050 con 0x555847d7f510 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.488+0000 7fa078e32b80 1 -- 172.21.15.16:0/2761752949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa04c004380 con 0x555847d7f510 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa03fe00700 1 -- 172.21.15.16:0/2761752949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa05009ee80 con 0x555847d7f510 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa078e32b80 10 client.15216 fetch_fsmap learned FSMap version 27 2024-04-05T23:33:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa078e32b80 1 -- 172.21.15.16:0/2761752949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555847b55c80 con 0x555847d7f510 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa03fe00700 1 -- 172.21.15.16:0/2761752949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0500c9020 con 0x555847d7f510 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa03fe00700 1 -- 172.21.15.16:0/2761752949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fa050021930 con 0x555847d7f510 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa078e32b80 10 client.15216 fetch_fsmap finished waiting for FSMap version 27 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa078e32b80 10 client.15216 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa078e32b80 20 client.15216 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa078e32b80 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555847ded130 0x555847de9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa078e32b80 4 client.15216 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa078e32b80 1 -- 172.21.15.16:0/2761752949 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555847b1eca0 con 0x555847ded130 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.489+0000 7fa05de00700 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555847ded130 0x555847de9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa03fe00700 10 client.15216 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa05de00700 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555847ded130 0x555847de9ae0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10209ca25994c0ab server_cookie=f5fd0856166c3951 in_seq=0 out_seq=0 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa03fe00700 1 -- 172.21.15.16:0/2761752949 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x555847b1eca0 con 0x555847ded130 2024-04-05T23:33:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa03fe00700 10 client.15216 handle_command_reply: tid=0 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa078e32b80 1 client.15216 shutdown 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa078e32b80 1 -- 172.21.15.16:0/2761752949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 msgr2=0x555847d90050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa078e32b80 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d90050 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa05009af90 tx=0x7fa050005e80 comp rx=0 tx=0).stop 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa078e32b80 1 -- 172.21.15.16:0/2761752949 shutdown_connections 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa078e32b80 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555847ded130 0x555847de9ae0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa078e32b80 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555847d7f510 0x555847d90050 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa078e32b80 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555847d80a60 0x555847d8cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa078e32b80 1 --2- 172.21.15.16:0/2761752949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555847d7fe70 0x555847d90590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa03fe00700 0 client.15216 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.490+0000 7fa078e32b80 1 -- 172.21.15.16:0/2761752949 >> 172.21.15.16:0/2761752949 conn(0x555847cf31d0 msgr2=0x555847d9d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa078e32b80 1 -- 172.21.15.16:0/2761752949 shutdown_connections 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa078e32b80 1 -- 172.21.15.16:0/2761752949 wait complete. 2024-04-05T23:33:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa078e32b80 20 client.15216 trim_cache size 0 max 16384 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b3471b30-f3a3-11ee-b649-cb9ed24678a4", 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24566, 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 79.975168840999999, 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 81, 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 79, 2024-04-05T23:33:24.489 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 86.889368907000005 2024-04-05T23:33:24.490 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 -- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa0440a11b0 msgr2=0x7fa0440a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa0440a11b0 0x7fa0440a3670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa058009fd0 tx=0x7fa058009920 comp rx=0 tx=0).stop 2024-04-05T23:33:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 -- 172.21.15.16:0/4004800277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa060100e60 msgr2=0x7fa0601990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa060100e60 0x7fa0601990f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa050005950 tx=0x7fa050009580 comp rx=0 tx=0).stop 2024-04-05T23:33:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 -- 172.21.15.16:0/4004800277 shutdown_connections 2024-04-05T23:33:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fa0440a11b0 0x7fa0440a3670 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060102a00 0x7fa0601a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa060100e60 0x7fa0601990f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 --2- 172.21.15.16:0/4004800277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa060102040 0x7fa060199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 -- 172.21.15.16:0/4004800277 >> 172.21.15.16:0/4004800277 conn(0x7fa0600fc440 msgr2=0x7fa060073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.491+0000 7fa065400700 1 -- 172.21.15.16:0/4004800277 shutdown_connections 2024-04-05T23:33:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.492+0000 7fa065400700 1 -- 172.21.15.16:0/4004800277 wait complete. 2024-04-05T23:33:24.499 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:24 smithi016 ceph-mon[38248]: pgmap v274: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-05T23:33:24.501 INFO:teuthology.run_tasks:Running task sequential... 2024-04-05T23:33:24.512 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-05T23:33:24.512 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-05T23:33:24.512 INFO:teuthology.task.print:Enabling mgr modules 2024-04-05T23:33:24.512 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-05T23:33:24.513 INFO:teuthology.task.exec:Executing custom commands... 2024-04-05T23:33:24.513 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:33:24.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-05T23:33:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.718+0000 7fbfe5c00700 1 -- 172.21.15.16:0/1725346061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfe0100e60 msgr2=0x7fbfe0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.718+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/1725346061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfe0100e60 0x7fbfe0101280 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009990 tx=0x7fbfd0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.718+0000 7fbfe5c00700 1 -- 172.21.15.16:0/1725346061 shutdown_connections 2024-04-05T23:33:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.718+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/1725346061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfe0100e60 0x7fbfe0101280 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.718+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/1725346061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfe0102970 0x7fbfe0109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.718+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/1725346061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe0101fb0 0x7fbfe0102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.718+0000 7fbfe5c00700 1 -- 172.21.15.16:0/1725346061 >> 172.21.15.16:0/1725346061 conn(0x7fbfe00fc3d0 msgr2=0x7fbfe00fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.718+0000 7fbfe5c00700 1 -- 172.21.15.16:0/1725346061 shutdown_connections 2024-04-05T23:33:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.718+0000 7fbfe5c00700 1 -- 172.21.15.16:0/1725346061 wait complete. 2024-04-05T23:33:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.720+0000 7fbfe5c00700 1 Processor -- start 2024-04-05T23:33:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.720+0000 7fbfe5c00700 1 -- start start 2024-04-05T23:33:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.720+0000 7fbfe5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfe0100e60 0x7fbfe0199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.720+0000 7fbfe5c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe0101fb0 0x7fbfe01995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.720+0000 7fbfe5c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfe0102970 0x7fbfe01a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.720+0000 7fbfe5c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe01a0b60 con 0x7fbfe0102970 2024-04-05T23:33:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.720+0000 7fbfe5c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe01a0ca0 con 0x7fbfe0101fb0 2024-04-05T23:33:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.720+0000 7fbfe5c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe01a0e30 con 0x7fbfe0100e60 2024-04-05T23:33:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfde000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe0101fb0 0x7fbfe01995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfdf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfe0102970 0x7fbfe01a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfde000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe0101fb0 0x7fbfe01995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38494/0 (socket says 172.21.15.16:38494) 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfde000700 1 -- 172.21.15.16:0/3787695652 learned_addr learned my addr 172.21.15.16:0/3787695652 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfdea00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfe0100e60 0x7fbfe0199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfdf400700 1 -- 172.21.15.16:0/3787695652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfe0100e60 msgr2=0x7fbfe0199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfdf400700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfe0100e60 0x7fbfe0199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfdf400700 1 -- 172.21.15.16:0/3787695652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe0101fb0 msgr2=0x7fbfe01995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfdf400700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe0101fb0 0x7fbfe01995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfdf400700 1 -- 172.21.15.16:0/3787695652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc009520 con 0x7fbfe0102970 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfdea00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfe0100e60 0x7fbfe0199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfdf400700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfe0102970 0x7fbfe01a0590 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800efc0 tx=0x7fbfd800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfd7400700 1 -- 172.21.15.16:0/3787695652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8018070 con 0x7fbfe0102970 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfd7400700 1 -- 172.21.15.16:0/3787695652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbfd800f040 con 0x7fbfe0102970 2024-04-05T23:33:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd00095f0 con 0x7fbfe0102970 2024-04-05T23:33:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.721+0000 7fbfd7400700 1 -- 172.21.15.16:0/3787695652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd800cdf0 con 0x7fbfe0102970 2024-04-05T23:33:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.722+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe01a1430 con 0x7fbfe0102970 2024-04-05T23:33:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.722+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe0067d70 con 0x7fbfe0102970 2024-04-05T23:33:24.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:24 smithi027 ceph-mon[78597]: pgmap v274: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-05T23:33:24.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.724+0000 7fbfd7400700 1 -- 172.21.15.16:0/3787695652 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 135924+0+0 (secure 0 0 0) 0x7fbfd8007500 con 0x7fbfe0102970 2024-04-05T23:33:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.726+0000 7fbfd7400700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbfc40a11b0 0x7fbfc40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.726+0000 7fbfd7400700 1 -- 172.21.15.16:0/3787695652 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbfd80cc700 con 0x7fbfe0102970 2024-04-05T23:33:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.726+0000 7fbfdea00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbfc40a11b0 0x7fbfc40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.727+0000 7fbfdea00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbfc40a11b0 0x7fbfc40a3670 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000b2e0 tx=0x7fbfd0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:24.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:24.730+0000 7fbfd7400700 1 -- 172.21.15.16:0/3787695652 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbfd8095d70 con 0x7fbfe0102970 2024-04-05T23:33:24.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:24 smithi130 ceph-mon[79417]: pgmap v274: 97 pgs: 97 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-05T23:33:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.009+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fbfe010c880 con 0x7fbfe0102970 2024-04-05T23:33:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.370+0000 7fbfd7400700 1 -- 172.21.15.16:0/3787695652 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 18) v1 ==== 135941+0+0 (secure 0 0 0) 0x7fbfd8096140 con 0x7fbfe0102970 2024-04-05T23:33:25.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.372+0000 7fbfd7400700 1 -- 172.21.15.16:0/3787695652 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v18) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbfd8099800 con 0x7fbfe0102970 2024-04-05T23:33:25.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.374+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbfc40a11b0 msgr2=0x7fbfc40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.374+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbfc40a11b0 0x7fbfc40a3670 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000b2e0 tx=0x7fbfd0005fb0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.374+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfe0102970 msgr2=0x7fbfe01a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.374+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfe0102970 0x7fbfe01a0590 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800efc0 tx=0x7fbfd800c3a0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.375+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 shutdown_connections 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.375+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fbfc40a11b0 0x7fbfc40a3670 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.375+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfe0102970 0x7fbfe01a0590 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.375+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfe0100e60 0x7fbfe0199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.375+0000 7fbfe5c00700 1 --2- 172.21.15.16:0/3787695652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe0101fb0 0x7fbfe01995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.375+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 >> 172.21.15.16:0/3787695652 conn(0x7fbfe00fc3d0 msgr2=0x7fbfe0073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.375+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 shutdown_connections 2024-04-05T23:33:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.375+0000 7fbfe5c00700 1 -- 172.21.15.16:0/3787695652 wait complete. 2024-04-05T23:33:25.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-05T23:33:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.603+0000 7f28dfe00700 1 -- 172.21.15.16:0/1724291890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d81018c0 msgr2=0x7f28d8103cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.603+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1724291890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d81018c0 0x7f28d8103cb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009990 tx=0x7f28cc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.603+0000 7f28dfe00700 1 -- 172.21.15.16:0/1724291890 shutdown_connections 2024-04-05T23:33:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.603+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1724291890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d8106b20 0x7f28d8108f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.603+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1724291890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d81018c0 0x7f28d8103cb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.603+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1724291890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28d81041f0 0x7f28d81065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.603+0000 7f28dfe00700 1 -- 172.21.15.16:0/1724291890 >> 172.21.15.16:0/1724291890 conn(0x7f28d80fb2d0 msgr2=0x7f28d80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.604+0000 7f28dfe00700 1 -- 172.21.15.16:0/1724291890 shutdown_connections 2024-04-05T23:33:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.604+0000 7f28dfe00700 1 -- 172.21.15.16:0/1724291890 wait complete. 2024-04-05T23:33:25.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dfe00700 1 Processor -- start 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dfe00700 1 -- start start 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dfe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28d81018c0 0x7f28d8198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dfe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d81041f0 0x7f28d8199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d8106b20 0x7f28d81a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dfe00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d81a0b10 con 0x7f28d81041f0 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dfe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d81a0c50 con 0x7f28d8106b20 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dfe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d81a0de0 con 0x7f28d81018c0 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dd400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28d81018c0 0x7f28d8198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dd400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28d81018c0 0x7f28d8198fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:43770/0 (socket says 172.21.15.16:43770) 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dd400700 1 -- 172.21.15.16:0/1219987306 learned_addr learned my addr 172.21.15.16:0/1219987306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:25.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dde00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d8106b20 0x7f28d81a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.605+0000 7f28dca00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d81041f0 0x7f28d8199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28dd400700 1 -- 172.21.15.16:0/1219987306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d8106b20 msgr2=0x7f28d81a0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28dd400700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d8106b20 0x7f28d81a0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28dd400700 1 -- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d81041f0 msgr2=0x7f28d8199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28dd400700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d81041f0 0x7f28d8199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28dd400700 1 -- 172.21.15.16:0/1219987306 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d0009c40 con 0x7f28d81018c0 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28dd400700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28d81018c0 0x7f28d8198fe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f28cc005910 tx=0x7f28cc009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28d6000700 1 -- 172.21.15.16:0/1219987306 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc01d070 con 0x7f28d81018c0 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28cc0095f0 con 0x7f28d81018c0 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28d6000700 1 -- 172.21.15.16:0/1219987306 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28cc021410 con 0x7f28d81018c0 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28d6000700 1 -- 172.21.15.16:0/1219987306 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc017650 con 0x7f28d81018c0 2024-04-05T23:33:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.606+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d81a1350 con 0x7f28d81018c0 2024-04-05T23:33:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.607+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d81086f0 con 0x7f28d81018c0 2024-04-05T23:33:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.609+0000 7f28d6000700 1 -- 172.21.15.16:0/1219987306 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135941+0+0 (secure 0 0 0) 0x7f28cc021580 con 0x7f28d81018c0 2024-04-05T23:33:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.610+0000 7f28d6000700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f28c00a1200 0x7f28c00a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.611+0000 7f28dca00700 1 -- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f28c00a1200 msgr2=0x7f28c00a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1565223355 2024-04-05T23:33:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.611+0000 7f28d6000700 1 -- 172.21.15.16:0/1219987306 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f28cc0cdf60 con 0x7f28d81018c0 2024-04-05T23:33:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.611+0000 7f28dca00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f28c00a1200 0x7f28c00a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T23:33:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.613+0000 7f28d6000700 1 -- 172.21.15.16:0/1219987306 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f28cc0975d0 con 0x7f28d81018c0 2024-04-05T23:33:25.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:25 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3787695652' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-05T23:33:25.717 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:25 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: ignoring --setuser ceph since I am not root 2024-04-05T23:33:25.717 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:25 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: ignoring --setgroup ceph since I am not root 2024-04-05T23:33:25.717 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:25 smithi027 ceph-mgr[80070]: -- 172.21.15.27:0/2433070979 <== mon.2 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563c980481a0 con 0x563c9801ec00 2024-04-05T23:33:25.800 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:25 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3787695652' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-05T23:33:25.800 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:25 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: ignoring --setuser ceph since I am not root 2024-04-05T23:33:25.800 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:25 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: ignoring --setgroup ceph since I am not root 2024-04-05T23:33:25.800 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:25 smithi130 ceph-mgr[81477]: -- 172.21.15.130:0/3218324597 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556474bb01a0 con 0x556474b86800 2024-04-05T23:33:25.800 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:25 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:25.806+0000 7fc6ce382200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-05T23:33:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.812+0000 7f28dca00700 1 -- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f28c00a1200 msgr2=0x7f28c00a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1565223355 2024-04-05T23:33:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.812+0000 7f28dca00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f28c00a1200 0x7f28c00a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T23:33:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.830+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f28d8067d70 con 0x7f28d81018c0 2024-04-05T23:33:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.837+0000 7f28d6000700 1 -- 172.21.15.16:0/1219987306 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) v1 ==== 138+0+0 (secure 0 0 0) 0x7f28cc09abf0 con 0x7f28d81018c0 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f28c00a1200 msgr2=0x7f28c00a36c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f28c00a1200 0x7f28c00a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28d81018c0 msgr2=0x7f28d8198fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28d81018c0 0x7f28d8198fe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f28cc005910 tx=0x7f28cc009580 comp rx=0 tx=0).stop 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 shutdown_connections 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f28c00a1200 0x7f28c00a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d8106b20 0x7f28d81a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28d81018c0 0x7f28d8198fe0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 --2- 172.21.15.16:0/1219987306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d81041f0 0x7f28d8199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 >> 172.21.15.16:0/1219987306 conn(0x7f28d80fb2d0 msgr2=0x7f28d8101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 shutdown_connections 2024-04-05T23:33:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:25.840+0000 7f28dfe00700 1 -- 172.21.15.16:0/1219987306 wait complete. 2024-04-05T23:33:25.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:25 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3787695652' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-05T23:33:25.833 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:25 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: ignoring --setuser ceph since I am not root 2024-04-05T23:33:25.833 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:25 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: ignoring --setgroup ceph since I am not root 2024-04-05T23:33:25.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-05T23:33:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.059+0000 7f9328200700 1 -- 172.21.15.16:0/66669145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93201018c0 msgr2=0x7f9320103cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.059+0000 7f9328200700 1 --2- 172.21.15.16:0/66669145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93201018c0 0x7f9320103cb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f93140098f0 tx=0x7f9314009c00 comp rx=0 tx=0).stop 2024-04-05T23:33:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.059+0000 7f9328200700 1 -- 172.21.15.16:0/66669145 shutdown_connections 2024-04-05T23:33:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.059+0000 7f9328200700 1 --2- 172.21.15.16:0/66669145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9320106b20 0x7f9320108f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.059+0000 7f9328200700 1 --2- 172.21.15.16:0/66669145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93201018c0 0x7f9320103cb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.059+0000 7f9328200700 1 --2- 172.21.15.16:0/66669145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93201041f0 0x7f93201065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.059+0000 7f9328200700 1 -- 172.21.15.16:0/66669145 >> 172.21.15.16:0/66669145 conn(0x7f93200fb2d0 msgr2=0x7f93200fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.059+0000 7f9328200700 1 -- 172.21.15.16:0/66669145 shutdown_connections 2024-04-05T23:33:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.059+0000 7f9328200700 1 -- 172.21.15.16:0/66669145 wait complete. 2024-04-05T23:33:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.060+0000 7f9328200700 1 Processor -- start 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.060+0000 7f9328200700 1 -- start start 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.060+0000 7f9328200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93201018c0 0x7f9320199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.060+0000 7f9328200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93201041f0 0x7f9320199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9328200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9320106b20 0x7f93201a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9328200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93201a0b40 con 0x7f9320106b20 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9328200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93201a0c80 con 0x7f93201018c0 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9328200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93201a0e10 con 0x7f93201041f0 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9325800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93201018c0 0x7f9320199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9325800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93201018c0 0x7f9320199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38528/0 (socket says 172.21.15.16:38528) 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9325800700 1 -- 172.21.15.16:0/2169314495 learned_addr learned my addr 172.21.15.16:0/2169314495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9326200700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9320106b20 0x7f93201a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9324e00700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93201041f0 0x7f9320199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9325800700 1 -- 172.21.15.16:0/2169314495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93201041f0 msgr2=0x7f9320199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9325800700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93201041f0 0x7f9320199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9325800700 1 -- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9320106b20 msgr2=0x7f93201a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9325800700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9320106b20 0x7f93201a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9325800700 1 -- 172.21.15.16:0/2169314495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9318009d00 con 0x7f93201018c0 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9326200700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9320106b20 0x7f93201a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.061+0000 7f9325800700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93201018c0 0x7f9320199010 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9314009810 tx=0x7f931400b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.062+0000 7f931e000700 1 -- 172.21.15.16:0/2169314495 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931401c070 con 0x7f93201018c0 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.062+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9314009520 con 0x7f93201018c0 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.062+0000 7f931e000700 1 -- 172.21.15.16:0/2169314495 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f931400f7d0 con 0x7f93201018c0 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.062+0000 7f931e000700 1 -- 172.21.15.16:0/2169314495 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314003a60 con 0x7f93201018c0 2024-04-05T23:33:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.062+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93201a1380 con 0x7f93201018c0 2024-04-05T23:33:26.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.063+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93201086f0 con 0x7f93201018c0 2024-04-05T23:33:26.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.064+0000 7f931e000700 1 -- 172.21.15.16:0/2169314495 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 135941+0+0 (secure 0 0 0) 0x7f9314020030 con 0x7f93201018c0 2024-04-05T23:33:26.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.066+0000 7f931e000700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f93080a1200 0x7f93080a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:26.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.066+0000 7f931e000700 1 -- 172.21.15.16:0/2169314495 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f93140cdb20 con 0x7f93201018c0 2024-04-05T23:33:26.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.066+0000 7f9324e00700 1 -- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f93080a1200 msgr2=0x7f93080a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1565223355 2024-04-05T23:33:26.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.066+0000 7f9324e00700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f93080a1200 0x7f93080a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T23:33:26.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.070+0000 7f931e000700 1 -- 172.21.15.16:0/2169314495 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9314097190 con 0x7f93201018c0 2024-04-05T23:33:26.065 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:25 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:25.900+0000 7fc6ce382200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-05T23:33:26.065 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:25 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:25.987+0000 7fc6ce382200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-05T23:33:26.094 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:25 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:25.824+0000 7f7aec06c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-05T23:33:26.094 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:25 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:25.922+0000 7f7aec06c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-05T23:33:26.094 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:26 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:26.013+0000 7f7aec06c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-05T23:33:26.116 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:25 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:25.997+0000 7fa21dced200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-05T23:33:26.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.267+0000 7f9324e00700 1 -- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f93080a1200 msgr2=0x7f93080a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1565223355 2024-04-05T23:33:26.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.267+0000 7f9324e00700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f93080a1200 0x7f93080a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T23:33:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.287+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f932010c840 con 0x7f93201018c0 2024-04-05T23:33:26.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.292+0000 7f931e000700 1 -- 172.21.15.16:0/2169314495 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) v1 ==== 133+0+0 (secure 0 0 0) 0x7f931409a7b0 con 0x7f93201018c0 2024-04-05T23:33:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f93080a1200 msgr2=0x7f93080a36c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:33:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f93080a1200 0x7f93080a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93201018c0 msgr2=0x7f9320199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93201018c0 0x7f9320199010 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9314009810 tx=0x7f931400b790 comp rx=0 tx=0).stop 2024-04-05T23:33:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 shutdown_connections 2024-04-05T23:33:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7f93080a1200 0x7f93080a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9320106b20 0x7f93201a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93201018c0 0x7f9320199010 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 --2- 172.21.15.16:0/2169314495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93201041f0 0x7f9320199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:26.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 >> 172.21.15.16:0/2169314495 conn(0x7f93200fb2d0 msgr2=0x7f93201010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:26.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 shutdown_connections 2024-04-05T23:33:26.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:26.295+0000 7f9328200700 1 -- 172.21.15.16:0/2169314495 wait complete. 2024-04-05T23:33:26.305 INFO:teuthology.run_tasks:Running task kclient... 2024-04-05T23:33:26.329 INFO:tasks.kclient:Mounting kernel clients... 2024-04-05T23:33:26.329 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-05T23:33:26.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:26.330 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-05T23:33:26.333 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:26 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:26.073+0000 7fc6ce382200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-05T23:33:26.352 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:26 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:26.102+0000 7f7aec06c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-05T23:33:26.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:26.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-05T23:33:26.472 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:26 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3787695652' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-05T23:33:26.472 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:26 smithi016 ceph-mon[38248]: mgrmap e18: x(active, since 6m), standbys: y, z 2024-04-05T23:33:26.472 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:26 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' 2024-04-05T23:33:26.473 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:26 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' 2024-04-05T23:33:26.473 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:26 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:26.124+0000 7fa21dced200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-05T23:33:26.473 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:26 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:26.242+0000 7fa21dced200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-05T23:33:26.473 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:26 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:26.358+0000 7fa21dced200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-05T23:33:26.486 INFO:teuthology.orchestra.run.smithi016.stderr:Cannot find device "ceph-brx" 2024-04-05T23:33:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:33:26.489 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-05T23:33:26.490 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-05T23:33:26.490 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-05T23:33:26.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph fs ls 2024-04-05T23:33:26.690 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:26 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3787695652' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-05T23:33:26.690 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:26 smithi130 ceph-mon[79417]: mgrmap e18: x(active, since 6m), standbys: y, z 2024-04-05T23:33:26.691 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:26 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' 2024-04-05T23:33:26.691 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:26 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' 2024-04-05T23:33:26.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:26 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3787695652' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-05T23:33:26.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:26 smithi027 ceph-mon[78597]: mgrmap e18: x(active, since 6m), standbys: y, z 2024-04-05T23:33:26.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:26 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' 2024-04-05T23:33:26.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:26 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' 2024-04-05T23:33:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:33:26.943 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:26 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:26.698+0000 7fc6ce382200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:33:27.019 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:26 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:26.765+0000 7f7aec06c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:33:27.295 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:26 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:26.951+0000 7fc6ce382200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-05T23:33:27.295 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:27.036+0000 7fc6ce382200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-05T23:33:27.295 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:27.144+0000 7fc6ce382200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-05T23:33:27.332 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:27 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:27.244+0000 7fa21dced200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:33:27.386 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:27 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:27.026+0000 7f7aec06c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-05T23:33:27.386 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:27 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:27.116+0000 7f7aec06c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-05T23:33:27.387 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:27 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:27.230+0000 7f7aec06c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-05T23:33:27.387 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:27 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:27.394+0000 7f7aec06c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-05T23:33:27.584 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:27.303+0000 7fc6ce382200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-05T23:33:27.584 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:27.429+0000 7fc6ce382200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-05T23:33:27.584 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:27.514+0000 7fc6ce382200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-05T23:33:27.716 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:27 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:27.527+0000 7f7aec06c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-05T23:33:27.716 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:27 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:27.616+0000 7f7aec06c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-05T23:33:27.832 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:27 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:27.574+0000 7fa21dced200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-05T23:33:27.832 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:27 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:27.691+0000 7fa21dced200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-05T23:33:28.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.116+0000 7fd745400700 1 -- 172.21.15.16:0/220134520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd740073c30 msgr2=0x7fd74010cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:28.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.116+0000 7fd745400700 1 --2- 172.21.15.16:0/220134520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd740073c30 0x7fd74010cff0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009990 tx=0x7fd72c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.116+0000 7fd745400700 1 -- 172.21.15.16:0/220134520 shutdown_connections 2024-04-05T23:33:28.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.116+0000 7fd745400700 1 --2- 172.21.15.16:0/220134520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd74010d530 0x7fd74010f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.116+0000 7fd745400700 1 --2- 172.21.15.16:0/220134520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd740073310 0x7fd7400736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.116+0000 7fd745400700 1 --2- 172.21.15.16:0/220134520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd740073c30 0x7fd74010cff0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.116+0000 7fd745400700 1 -- 172.21.15.16:0/220134520 >> 172.21.15.16:0/220134520 conn(0x7fd7400fc840 msgr2=0x7fd7400fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.116+0000 7fd745400700 1 -- 172.21.15.16:0/220134520 shutdown_connections 2024-04-05T23:33:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.116+0000 7fd745400700 1 -- 172.21.15.16:0/220134520 wait complete. 2024-04-05T23:33:28.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd745400700 1 Processor -- start 2024-04-05T23:33:28.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd745400700 1 -- start start 2024-04-05T23:33:28.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd745400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd740073310 0x7fd7401993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:28.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd745400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd740073c30 0x7fd740199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:28.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd745400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74010d530 0x7fd74019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd745400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74019e2d0 con 0x7fd74010d530 2024-04-05T23:33:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd745400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74019e440 con 0x7fd740073310 2024-04-05T23:33:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd745400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74019e600 con 0x7fd740073c30 2024-04-05T23:33:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd73fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd740073310 0x7fd7401993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd73fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd740073310 0x7fd7401993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38554/0 (socket says 172.21.15.16:38554) 2024-04-05T23:33:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd73f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd740073c30 0x7fd740199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd73fe00700 1 -- 172.21.15.16:0/2861252399 learned_addr learned my addr 172.21.15.16:0/2861252399 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.118+0000 7fd744a00700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74010d530 0x7fd74019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73fe00700 1 -- 172.21.15.16:0/2861252399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd740073c30 msgr2=0x7fd740199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73fe00700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd740073c30 0x7fd740199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73fe00700 1 -- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74010d530 msgr2=0x7fd74019dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73fe00700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74010d530 0x7fd74019dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73fe00700 1 -- 172.21.15.16:0/2861252399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c0095f0 con 0x7fd740073310 2024-04-05T23:33:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73f400700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd740073c30 0x7fd740199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73fe00700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd740073310 0x7fd7401993d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd73000e9a0 tx=0x7fd73000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73cc00700 1 -- 172.21.15.16:0/2861252399 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730009790 con 0x7fd740073310 2024-04-05T23:33:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74019e8e0 con 0x7fd740073310 2024-04-05T23:33:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73cc00700 1 -- 172.21.15.16:0/2861252399 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd730010430 con 0x7fd740073310 2024-04-05T23:33:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.119+0000 7fd73cc00700 1 -- 172.21.15.16:0/2861252399 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730018680 con 0x7fd740073310 2024-04-05T23:33:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.120+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7401a6140 con 0x7fd740073310 2024-04-05T23:33:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.121+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd740108740 con 0x7fd740073310 2024-04-05T23:33:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.122+0000 7fd73cc00700 1 -- 172.21.15.16:0/2861252399 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 135941+0+0 (secure 0 0 0) 0x7fd7300105a0 con 0x7fd740073310 2024-04-05T23:33:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.124+0000 7fd73cc00700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd7240a1200 0x7fd7240a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.124+0000 7fd73cc00700 1 -- 172.21.15.16:0/2861252399 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd730014070 con 0x7fd740073310 2024-04-05T23:33:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.124+0000 7fd73f400700 1 -- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd7240a1200 msgr2=0x7fd7240a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1565223355 2024-04-05T23:33:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.124+0000 7fd73f400700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd7240a1200 0x7fd7240a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T23:33:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.126+0000 7fd73cc00700 1 -- 172.21.15.16:0/2861252399 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd730096270 con 0x7fd740073310 2024-04-05T23:33:28.229 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:27 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:27.845+0000 7fa21dced200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-05T23:33:28.229 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:28.057+0000 7fa21dced200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-05T23:33:28.278 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:27.883+0000 7fc6ce382200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-05T23:33:28.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.325+0000 7fd73f400700 1 -- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd7240a1200 msgr2=0x7fd7240a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1565223355 2024-04-05T23:33:28.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.325+0000 7fd73f400700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd7240a1200 0x7fd7240a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T23:33:28.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.384+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd74019a6e0 con 0x7fd740073310 2024-04-05T23:33:28.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.385+0000 7fd73cc00700 1 -- 172.21.15.16:0/2861252399 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd730099890 con 0x7fd740073310 2024-04-05T23:33:28.376 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd7240a1200 msgr2=0x7fd7240a36c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd7240a1200 0x7fd7240a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd740073310 msgr2=0x7fd7401993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd740073310 0x7fd7401993d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd73000e9a0 tx=0x7fd73000ecb0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 shutdown_connections 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:6800/1565223355,v1:172.21.15.16:6801/1565223355] conn(0x7fd7240a1200 0x7fd7240a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74010d530 0x7fd74019dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd740073310 0x7fd7401993d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 --2- 172.21.15.16:0/2861252399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd740073c30 0x7fd740199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 >> 172.21.15.16:0/2861252399 conn(0x7fd7400fc840 msgr2=0x7fd7400fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 shutdown_connections 2024-04-05T23:33:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:28.387+0000 7fd745400700 1 -- 172.21.15.16:0/2861252399 wait complete. 2024-04-05T23:33:28.398 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:28 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:28.001+0000 7f7aec06c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-05T23:33:28.398 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:28 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:28.404+0000 7f7aec06c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-05T23:33:28.562 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:28 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2861252399' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T23:33:28.563 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:28.237+0000 7fa21dced200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-05T23:33:28.563 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:28.353+0000 7fa21dced200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-05T23:33:28.568 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:28 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:28.286+0000 7fc6ce382200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-05T23:33:28.568 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:28 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:28.482+0000 7fc6ce382200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-05T23:33:28.568 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:28 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2861252399' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T23:33:28.701 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:28 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2861252399' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T23:33:28.702 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:28 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:28.610+0000 7f7aec06c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-05T23:33:28.702 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:28 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:28.708+0000 7f7aec06c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-05T23:33:28.794 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-05T23:33:28.795 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-05T23:33:28.795 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-04-05T23:33:28.795 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-05T23:33:28.795 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-05T23:33:28.795 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-05T23:33:28.795 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-05T23:33:28.795 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-05T23:33:28.795 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-05T23:33:28.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:28.795 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-04-05T23:33:28.833 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:28 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:28.575+0000 7fc6ce382200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-05T23:33:28.839 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-04-05T23:33:28.840 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-05T23:33:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-04-05T23:33:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-04-05T23:33:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-05T23:33:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41742sec preferred_lft 41742sec 2024-04-05T23:33:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-04-05T23:33:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-05T23:33:28.841 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T23:33:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-04-05T23:33:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-04-05T23:33:28.842 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-05T23:33:28.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:28.843 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2024-04-05T23:33:28.843 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-05T23:33:28.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-04-05T23:33:28.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-04-05T23:33:28.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-04-05T23:33:28.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-05T23:33:28.843 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-05T23:33:29.212 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:28.853+0000 7fa21dced200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-05T23:33:29.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:29.318 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-05T23:33:29.391 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-04-05T23:33:29.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:29.396 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-04-05T23:33:29.464 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-04-05T23:33:29.465 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-05T23:33:29.465 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-05T23:33:29.465 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-05T23:33:29.465 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-05T23:33:29.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:29.466 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2024-04-05T23:33:29.466 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-05T23:33:29.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-05T23:33:29.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-05T23:33:29.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-05T23:33:29.467 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-05T23:33:29.570 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:29.389+0000 7fa21dced200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-05T23:33:29.583 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:29.339+0000 7fc6ce382200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-05T23:33:29.583 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:29.436+0000 7fc6ce382200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-05T23:33:29.584 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:29.584+0000 7fc6ce382200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-05T23:33:29.758 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:29 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:29.507+0000 7f7aec06c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-05T23:33:29.759 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:29 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:29.609+0000 7f7aec06c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-05T23:33:29.759 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:29 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:29.765+0000 7f7aec06c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-05T23:33:29.832 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:29.672+0000 7fa21dced200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-05T23:33:29.832 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:29.803+0000 7fa21dced200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-05T23:33:30.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:30.214 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-05T23:33:30.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:30.263 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-04-05T23:33:30.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:30.309 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2024-04-05T23:33:30.310 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-05T23:33:30.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-05T23:33:30.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-05T23:33:30.310 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-05T23:33:30.502 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-05T23:33:30.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:30.503 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2024-04-05T23:33:30.503 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-05T23:33:30.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-05T23:33:30.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-05T23:33:30.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-05T23:33:30.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-05T23:33:30.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-05T23:33:30.503 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-05T23:33:30.579 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:30.211+0000 7fc6ce382200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-05T23:33:30.579 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:30.391+0000 7fc6ce382200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-05T23:33:30.716 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:30 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:30.422+0000 7f7aec06c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-05T23:33:30.716 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:30 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:30.612+0000 7f7aec06c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-05T23:33:30.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:30.797 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2024-04-05T23:33:30.797 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-05T23:33:30.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-04-05T23:33:30.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-04-05T23:33:30.797 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-05T23:33:30.833 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:30.586+0000 7fc6ce382200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-05T23:33:30.969 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:30.873+0000 7fa21dced200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-05T23:33:30.973 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T23:33:30.973 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-05T23:33:31.015 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-05T23:33:31.015 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T23:33:31.015 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-05T23:33:31.097 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-05T23:33:31.097 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-05T23:33:31.097 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa 2024-04-05T23:33:31.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:31.098 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-05T23:33:31.216 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:30 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:30.815+0000 7f7aec06c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-05T23:33:31.241 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-05T23:33:31.241 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-05T23:33:31.241 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2024-04-05T23:33:31.332 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:31 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:31.010+0000 7fa21dced200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-05T23:33:31.332 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:31 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:31.216+0000 7fa21dced200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-05T23:33:31.796 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:31.518+0000 7fc6ce382200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-05T23:33:31.910 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-05T23:33:31.910 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-05T23:33:31.910 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2024-04-05T23:33:31.910 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.27:3300,172.21.15.130:3300" 2024-04-05T23:33:31.910 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:3300,172.21.15.27:3300,172.21.15.130:3300:/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa 2024-04-05T23:33:31.910 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=b3471b30-f3a3-11ee-b649-cb9ed24678a4" will pass to kernel 2024-04-05T23:33:31.917 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-05T23:33:31.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:31.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-05T23:33:32.080 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:31.804+0000 7fc6ce382200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:33:32.080 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:31.892+0000 7fc6ce382200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-05T23:33:32.080 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:31 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:31.787+0000 7f7aec06c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-05T23:33:32.139 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:32.078+0000 7fa21dced200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-05T23:33:32.334 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:33:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z[81405]: 2024-04-05T23:33:32.087+0000 7fc6ce382200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-05T23:33:32.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:32 smithi130 ceph-mon[79417]: Standby manager daemon z restarted 2024-04-05T23:33:32.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:32 smithi130 ceph-mon[79417]: Standby manager daemon z started 2024-04-05T23:33:32.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:32 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-05T23:33:32.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:32 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:33:32.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:32 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-05T23:33:32.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:32 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:33:32.378 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:32 smithi027 ceph-mon[78597]: Standby manager daemon z restarted 2024-04-05T23:33:32.378 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:32 smithi027 ceph-mon[78597]: Standby manager daemon z started 2024-04-05T23:33:32.378 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:32 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-05T23:33:32.378 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:32 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:33:32.379 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:32 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-05T23:33:32.379 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:32 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:33:32.379 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:32 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:32.088+0000 7f7aec06c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:33:32.379 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:32 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:32.185+0000 7f7aec06c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-05T23:33:32.379 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:33:32 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y[80047]: 2024-04-05T23:33:32.386+0000 7f7aec06c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-05T23:33:32.582 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:32.333+0000 7fa21dced200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-05T23:33:32.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:32 smithi016 ceph-mon[38248]: Standby manager daemon z restarted 2024-04-05T23:33:32.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:32 smithi016 ceph-mon[38248]: Standby manager daemon z started 2024-04-05T23:33:32.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:32 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-05T23:33:32.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:32 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:33:32.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:32 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-05T23:33:32.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:32 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.130:0/3769152096' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:33:33.082 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:32.608+0000 7fa21dced200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-05T23:33:33.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:33 smithi027 ceph-mon[78597]: mgrmap e19: x(active, since 6m), standbys: y, z 2024-04-05T23:33:33.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:33 smithi027 ceph-mon[78597]: Standby manager daemon y restarted 2024-04-05T23:33:33.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:33 smithi027 ceph-mon[78597]: Standby manager daemon y started 2024-04-05T23:33:33.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:33 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-05T23:33:33.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:33 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:33:33.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:33 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-05T23:33:33.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:33 smithi027 ceph-mon[78597]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:33:33.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:33 smithi016 ceph-mon[38248]: mgrmap e19: x(active, since 6m), standbys: y, z 2024-04-05T23:33:33.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:33 smithi016 ceph-mon[38248]: Standby manager daemon y restarted 2024-04-05T23:33:33.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:33 smithi016 ceph-mon[38248]: Standby manager daemon y started 2024-04-05T23:33:33.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:33 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-05T23:33:33.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:33 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:33:33.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:33 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-05T23:33:33.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:33 smithi016 ceph-mon[38248]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:33:33.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:33 smithi130 ceph-mon[79417]: mgrmap e19: x(active, since 6m), standbys: y, z 2024-04-05T23:33:33.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:33 smithi130 ceph-mon[79417]: Standby manager daemon y restarted 2024-04-05T23:33:33.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:33 smithi130 ceph-mon[79417]: Standby manager daemon y started 2024-04-05T23:33:33.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:33 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-05T23:33:33.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:33 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T23:33:33.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:33 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-05T23:33:33.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:33 smithi130 ceph-mon[79417]: from='mgr.? 172.21.15.27:0/976303408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T23:33:34.082 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:33.805+0000 7fa21dced200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-05T23:33:34.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:34 smithi027 ceph-mon[78597]: mgrmap e20: x(active, since 6m), standbys: y, z 2024-04-05T23:33:34.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-mon[38248]: mgrmap e20: x(active, since 6m), standbys: y, z 2024-04-05T23:33:34.582 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.198+0000 7fa21dced200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-05T23:33:34.582 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.321+0000 7fa21dced200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-05T23:33:34.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:34 smithi130 ceph-mon[79417]: mgrmap e20: x(active, since 6m), standbys: y, z 2024-04-05T23:33:34.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:34.660 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-05T23:33:34.772 INFO:teuthology.orchestra.run.smithi016.stdout:client25303 2024-04-05T23:33:34.772 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:33:34.772 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T23:33:34.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:34.836 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-05T23:33:34.922 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.590+0000 7fa21dced200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-05T23:33:34.948 INFO:teuthology.orchestra.run.smithi016.stdout:client25303 2024-04-05T23:33:34.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:34.949 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-05T23:33:35.042 INFO:teuthology.orchestra.run.smithi016.stdout:client25303 2024-04-05T23:33:35.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:33:35.042 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4.client25303/status') 2024-04-05T23:33:35.154 INFO:teuthology.orchestra.run.smithi016.stdout:instance: client.25303 (3)172.21.15.16:0/3170258196 2024-04-05T23:33:35.154 INFO:teuthology.orchestra.run.smithi016.stdout:blocklisted: false 2024-04-05T23:33:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records in 2024-04-05T23:33:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records out 2024-04-05T23:33:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:71 bytes copied, 5.9211e-05 s, 1.2 MB/s 2024-04-05T23:33:35.157 INFO:teuthology.run_tasks:Running task exec... 2024-04-05T23:33:35.167 INFO:teuthology.task.exec:Executing custom commands... 2024-04-05T23:33:35.168 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:33:35.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-05T23:33:35.231 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.929+0000 7fa170800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.231 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.929+0000 7fa170800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.231 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.929+0000 7fa170800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.231 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.929+0000 7fa170800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.232 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.929+0000 7fa170800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.232 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.931+0000 7fa173a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.232 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.931+0000 7fa173a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.232 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.931+0000 7fa173a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.232 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.931+0000 7fa173a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.232 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:33:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:33:34.931+0000 7fa173a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:33:35.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: Active manager daemon x restarted 2024-04-05T23:33:35.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: Activating manager daemon x 2024-04-05T23:33:35.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: osdmap e82: 12 total, 12 up, 12 in 2024-04-05T23:33:35.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: mgrmap e21: x(active, starting, since 0.0120428s), standbys: y, z 2024-04-05T23:33:35.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:33:35.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:33:35.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:33:35.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-05T23:33:35.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:33:35.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T23:33:35.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T23:33:35.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T23:33:35.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: Manager daemon x is now available 2024-04-05T23:33:35.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:33:35.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-05T23:33:35.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:35 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-05T23:33:35.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.380+0000 7f26ab200700 1 -- 172.21.15.16:0/3462265960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26a4100d80 msgr2=0x7f26a41011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:35.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.380+0000 7f26ab200700 1 --2- 172.21.15.16:0/3462265960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26a4100d80 0x7f26a41011e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f26940099c0 tx=0x7f2694009cd0 comp rx=0 tx=0).stop 2024-04-05T23:33:35.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.381+0000 7f26ab200700 1 -- 172.21.15.16:0/3462265960 shutdown_connections 2024-04-05T23:33:35.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.381+0000 7f26ab200700 1 --2- 172.21.15.16:0/3462265960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26a4101720 0x7f26a4107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:35.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.381+0000 7f26ab200700 1 --2- 172.21.15.16:0/3462265960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a40ffb90 0x7f26a40fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:35.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.381+0000 7f26ab200700 1 --2- 172.21.15.16:0/3462265960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26a4100d80 0x7f26a41011e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:35.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.381+0000 7f26ab200700 1 -- 172.21.15.16:0/3462265960 >> 172.21.15.16:0/3462265960 conn(0x7f26a40fb120 msgr2=0x7f26a40fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:35.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.381+0000 7f26ab200700 1 -- 172.21.15.16:0/3462265960 shutdown_connections 2024-04-05T23:33:35.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.381+0000 7f26ab200700 1 -- 172.21.15.16:0/3462265960 wait complete. 2024-04-05T23:33:35.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26ab200700 1 Processor -- start 2024-04-05T23:33:35.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26ab200700 1 -- start start 2024-04-05T23:33:35.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26ab200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26a40ffb90 0x7f26a4198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:35.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26ab200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a4100d80 0x7f26a4199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26ab200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26a4101720 0x7f26a41a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26ab200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a41a0900 con 0x7f26a4101720 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26ab200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a41a0a70 con 0x7f26a4100d80 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26ab200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a41a0c30 con 0x7f26a40ffb90 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26a9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a4100d80 0x7f26a4199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.382+0000 7f26a9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26a40ffb90 0x7f26a4198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26a9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a4100d80 0x7f26a4199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33226/0 (socket says 172.21.15.16:33226) 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26a9400700 1 -- 172.21.15.16:0/2450302196 learned_addr learned my addr 172.21.15.16:0/2450302196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26a9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26a40ffb90 0x7f26a4198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:41356/0 (socket says 172.21.15.16:41356) 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26aa800700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26a4101720 0x7f26a41a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26a9400700 1 -- 172.21.15.16:0/2450302196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26a40ffb90 msgr2=0x7f26a4198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26a9400700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26a40ffb90 0x7f26a4198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26a9400700 1 -- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26a4101720 msgr2=0x7f26a41a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:35.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26a9400700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26a4101720 0x7f26a41a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:35.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26a9400700 1 -- 172.21.15.16:0/2450302196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26940095f0 con 0x7f26a4100d80 2024-04-05T23:33:35.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26aa800700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26a4101720 0x7f26a41a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:35.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f26a9400700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a4100d80 0x7f26a4199350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2694003730 tx=0x7f269400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:35.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.383+0000 7f269ea00700 1 -- 172.21.15.16:0/2450302196 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269401d070 con 0x7f26a4100d80 2024-04-05T23:33:35.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.384+0000 7f269ea00700 1 -- 172.21.15.16:0/2450302196 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2694005050 con 0x7f26a4100d80 2024-04-05T23:33:35.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.384+0000 7f269ea00700 1 -- 172.21.15.16:0/2450302196 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694004bb0 con 0x7f26a4100d80 2024-04-05T23:33:35.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.384+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a41a0eb0 con 0x7f26a4100d80 2024-04-05T23:33:35.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.384+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a41a1220 con 0x7f26a4100d80 2024-04-05T23:33:35.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.385+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a404ebb0 con 0x7f26a4100d80 2024-04-05T23:33:35.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.387+0000 7f269ea00700 1 -- 172.21.15.16:0/2450302196 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 135558+0+0 (secure 0 0 0) 0x7f26940051c0 con 0x7f26a4100d80 2024-04-05T23:33:35.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.388+0000 7f269ea00700 1 -- 172.21.15.16:0/2450302196 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f26940cd910 con 0x7f26a4100d80 2024-04-05T23:33:35.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.390+0000 7f269ea00700 1 -- 172.21.15.16:0/2450302196 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f26940973e0 con 0x7f26a4100d80 2024-04-05T23:33:35.466 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: Active manager daemon x restarted 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: Activating manager daemon x 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: osdmap e82: 12 total, 12 up, 12 in 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: mgrmap e21: x(active, starting, since 0.0120428s), standbys: y, z 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-05T23:33:35.467 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:33:35.468 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: Manager daemon x is now available 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:33:35.469 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-05T23:33:35.470 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:35 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-05T23:33:35.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: Active manager daemon x restarted 2024-04-05T23:33:35.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: Activating manager daemon x 2024-04-05T23:33:35.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: osdmap e82: 12 total, 12 up, 12 in 2024-04-05T23:33:35.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: mgrmap e21: x(active, starting, since 0.0120428s), standbys: y, z 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:33:35.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-05T23:33:35.586 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: Manager daemon x is now available 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-05T23:33:35.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:35 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-05T23:33:35.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.614+0000 7f269ea00700 1 -- 172.21.15.16:0/2450302196 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 22) v1 ==== 135685+0+0 (secure 0 0 0) 0x7f2694096e90 con 0x7f26a4100d80 2024-04-05T23:33:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.615+0000 7f269ea00700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f268c0a7f10 0x7f268c0aa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.622+0000 7f26a9e00700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f268c0a7f10 0x7f268c0aa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:35.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.630+0000 7f26a9e00700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f268c0a7f10 0x7f268c0aa300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f26a419a290 tx=0x7f2698009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:35.687+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7f26a410b550 con 0x7f26a4100d80 2024-04-05T23:33:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.612+0000 7f269ea00700 1 -- 172.21.15.16:0/2450302196 <== mon.2 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v28) v1 ==== 117+0+0 (secure 0 0 0) 0x7f269408c260 con 0x7f26a4100d80 2024-04-05T23:33:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f268c0a7f10 msgr2=0x7f268c0aa300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f268c0a7f10 0x7f268c0aa300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f26a419a290 tx=0x7f2698009260 comp rx=0 tx=0).stop 2024-04-05T23:33:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a4100d80 msgr2=0x7f26a4199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a4100d80 0x7f26a4199350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2694003730 tx=0x7f269400f690 comp rx=0 tx=0).stop 2024-04-05T23:33:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 shutdown_connections 2024-04-05T23:33:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f268c0a7f10 0x7f268c0aa300 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26a4101720 0x7f26a41a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26a40ffb90 0x7f26a4198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 --2- 172.21.15.16:0/2450302196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a4100d80 0x7f26a4199350 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.614+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 >> 172.21.15.16:0/2450302196 conn(0x7f26a40fb120 msgr2=0x7f26a40fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.615+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 shutdown_connections 2024-04-05T23:33:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.615+0000 7f26ab200700 1 -- 172.21.15.16:0/2450302196 wait complete. 2024-04-05T23:33:36.633 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-05T23:33:36.645 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-05T23:33:36.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:33:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.920+0000 7f5658600700 1 -- 172.21.15.16:0/2082750786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5650102940 msgr2=0x7f56501091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.920+0000 7f5658600700 1 --2- 172.21.15.16:0/2082750786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5650102940 0x7f56501091f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f564800b0d0 tx=0x7f564800b3e0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.920+0000 7f5658600700 1 -- 172.21.15.16:0/2082750786 shutdown_connections 2024-04-05T23:33:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.920+0000 7f5658600700 1 --2- 172.21.15.16:0/2082750786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5650102940 0x7f56501091f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.920+0000 7f5658600700 1 --2- 172.21.15.16:0/2082750786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650100db0 0x7f56501011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.920+0000 7f5658600700 1 --2- 172.21.15.16:0/2082750786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5650101fa0 0x7f5650102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.920+0000 7f5658600700 1 -- 172.21.15.16:0/2082750786 >> 172.21.15.16:0/2082750786 conn(0x7f56500fc320 msgr2=0x7f56500fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.920+0000 7f5658600700 1 -- 172.21.15.16:0/2082750786 shutdown_connections 2024-04-05T23:33:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.921+0000 7f5658600700 1 -- 172.21.15.16:0/2082750786 wait complete. 2024-04-05T23:33:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.921+0000 7f5658600700 1 Processor -- start 2024-04-05T23:33:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5658600700 1 -- start start 2024-04-05T23:33:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5658600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650100db0 0x7f5650074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5658600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5650101fa0 0x7f56500732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5658600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5650102940 0x7f5650073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5658600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5650073fb0 con 0x7f5650102940 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5658600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5650074120 con 0x7f5650100db0 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5658600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56501a0e80 con 0x7f5650101fa0 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5655c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650100db0 0x7f5650074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5655c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650100db0 0x7f5650074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33272/0 (socket says 172.21.15.16:33272) 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5655c00700 1 -- 172.21.15.16:0/421503923 learned_addr learned my addr 172.21.15.16:0/421503923 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.922+0000 7f5656600700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5650102940 0x7f5650073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.923+0000 7f5655200700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5650101fa0 0x7f56500732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.923+0000 7f5655c00700 1 -- 172.21.15.16:0/421503923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5650101fa0 msgr2=0x7f56500732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.923+0000 7f5655c00700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5650101fa0 0x7f56500732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.923+0000 7f5655c00700 1 -- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5650102940 msgr2=0x7f5650073810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.923+0000 7f5655c00700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5650102940 0x7f5650073810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:36.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.923+0000 7f5655c00700 1 -- 172.21.15.16:0/421503923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c009520 con 0x7f5650100db0 2024-04-05T23:33:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.923+0000 7f5655c00700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650100db0 0x7f5650074c40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f564400e970 tx=0x7f564400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.923+0000 7f564e000700 1 -- 172.21.15.16:0/421503923 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644009790 con 0x7f5650100db0 2024-04-05T23:33:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.923+0000 7f564e000700 1 -- 172.21.15.16:0/421503923 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f564400c780 con 0x7f5650100db0 2024-04-05T23:33:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.924+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5648009d00 con 0x7f5650100db0 2024-04-05T23:33:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.924+0000 7f564e000700 1 -- 172.21.15.16:0/421503923 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644003be0 con 0x7f5650100db0 2024-04-05T23:33:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.924+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56501a1490 con 0x7f5650100db0 2024-04-05T23:33:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.925+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56501081e0 con 0x7f5650100db0 2024-04-05T23:33:36.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.927+0000 7f564e000700 1 -- 172.21.15.16:0/421503923 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) v1 ==== 135685+0+0 (secure 0 0 0) 0x7f564400cbf0 con 0x7f5650100db0 2024-04-05T23:33:36.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.928+0000 7f564e000700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56380a1070 0x7f56380a3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:36.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.928+0000 7f564e000700 1 -- 172.21.15.16:0/421503923 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f5644014070 con 0x7f5650100db0 2024-04-05T23:33:36.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.928+0000 7f5655200700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56380a1070 0x7f56380a3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:36.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.929+0000 7f5655200700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56380a1070 0x7f56380a3530 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f563c009df0 tx=0x7f563c009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:36.932+0000 7f564e000700 1 -- 172.21.15.16:0/421503923 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5644095fb0 con 0x7f5650100db0 2024-04-05T23:33:36.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: mgrmap e22: x(active, since 1.02906s), standbys: y, z 2024-04-05T23:33:36.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: [05/Apr/2024:23:33:35] ENGINE Bus STARTING 2024-04-05T23:33:36.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2450302196' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-05T23:33:36.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-05T23:33:36.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: [05/Apr/2024:23:33:35] ENGINE Serving on http://172.21.15.16:8765 2024-04-05T23:33:36.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: [05/Apr/2024:23:33:35] ENGINE Serving on https://172.21.15.16:7150 2024-04-05T23:33:36.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: [05/Apr/2024:23:33:35] ENGINE Bus STARTED 2024-04-05T23:33:36.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:36.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:36.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:36.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:36 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: mgrmap e22: x(active, since 1.02906s), standbys: y, z 2024-04-05T23:33:37.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: [05/Apr/2024:23:33:35] ENGINE Bus STARTING 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2450302196' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: [05/Apr/2024:23:33:35] ENGINE Serving on http://172.21.15.16:8765 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: [05/Apr/2024:23:33:35] ENGINE Serving on https://172.21.15.16:7150 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: [05/Apr/2024:23:33:35] ENGINE Bus STARTED 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:36 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: mgrmap e22: x(active, since 1.02906s), standbys: y, z 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: [05/Apr/2024:23:33:35] ENGINE Bus STARTING 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2450302196' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: [05/Apr/2024:23:33:35] ENGINE Serving on http://172.21.15.16:8765 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: [05/Apr/2024:23:33:35] ENGINE Serving on https://172.21.15.16:7150 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: [05/Apr/2024:23:33:35] ENGINE Bus STARTED 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:36 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.159+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5650067b50 con 0x7f5650100db0 2024-04-05T23:33:37.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.161+0000 7f564e000700 1 -- 172.21.15.16:0/421503923 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22958 (secure 0 0 0) 0x7f56440995d0 con 0x7f5650100db0 2024-04-05T23:33:37.154 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:37.154 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:33:34.593258+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/985211078":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1740984099":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/4157043736":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/1565223355":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2846240830":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/4184690359":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/1565223355":"2024-04-06T23:33:34.593227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:33:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56380a1070 msgr2=0x7f56380a3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56380a1070 0x7f56380a3530 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f563c009df0 tx=0x7f563c009260 comp rx=0 tx=0).stop 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650100db0 msgr2=0x7f5650074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650100db0 0x7f5650074c40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f564400e970 tx=0x7f564400ec80 comp rx=0 tx=0).stop 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 shutdown_connections 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56380a1070 0x7f56380a3530 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5650102940 0x7f5650073810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650100db0 0x7f5650074c40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 --2- 172.21.15.16:0/421503923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5650101fa0 0x7f56500732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 >> 172.21.15.16:0/421503923 conn(0x7f56500fc320 msgr2=0x7f5650105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 shutdown_connections 2024-04-05T23:33:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.163+0000 7f5658600700 1 -- 172.21.15.16:0/421503923 wait complete. 2024-04-05T23:33:37.176 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-05T23:32:02.641673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-05T23:32:03.972521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:33:37.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:33:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.396+0000 7f4cf0800700 1 -- 172.21.15.16:0/2168926348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ce8073940 msgr2=0x7f4ce8073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.396+0000 7f4cf0800700 1 --2- 172.21.15.16:0/2168926348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ce8073940 0x7f4ce8073dc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009990 tx=0x7f4cd8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.396+0000 7f4cf0800700 1 -- 172.21.15.16:0/2168926348 shutdown_connections 2024-04-05T23:33:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.396+0000 7f4cf0800700 1 --2- 172.21.15.16:0/2168926348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce8108d70 0x7f4ce810b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.396+0000 7f4cf0800700 1 --2- 172.21.15.16:0/2168926348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ce8074f20 0x7f4ce8073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.396+0000 7f4cf0800700 1 --2- 172.21.15.16:0/2168926348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ce8073940 0x7f4ce8073dc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.396+0000 7f4cf0800700 1 -- 172.21.15.16:0/2168926348 >> 172.21.15.16:0/2168926348 conn(0x7f4ce80fc420 msgr2=0x7f4ce80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.397+0000 7f4cf0800700 1 -- 172.21.15.16:0/2168926348 shutdown_connections 2024-04-05T23:33:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.397+0000 7f4cf0800700 1 -- 172.21.15.16:0/2168926348 wait complete. 2024-04-05T23:33:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cf0800700 1 Processor -- start 2024-04-05T23:33:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cf0800700 1 -- start start 2024-04-05T23:33:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cf0800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ce8073940 0x7f4ce81990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cf0800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ce8074f20 0x7f4ce81995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cf0800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce8108d70 0x7f4ce81a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cf0800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81a0b90 con 0x7f4ce8074f20 2024-04-05T23:33:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cf0800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81a0cd0 con 0x7f4ce8108d70 2024-04-05T23:33:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cf0800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81a0e60 con 0x7f4ce8073940 2024-04-05T23:33:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cee800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce8108d70 0x7f4ce81a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4ced400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ce8074f20 0x7f4ce81995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cee800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce8108d70 0x7f4ce81a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33282/0 (socket says 172.21.15.16:33282) 2024-04-05T23:33:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cee800700 1 -- 172.21.15.16:0/3192505286 learned_addr learned my addr 172.21.15.16:0/3192505286 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4ced400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ce8074f20 0x7f4ce81995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48308/0 (socket says 172.21.15.16:48308) 2024-04-05T23:33:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.398+0000 7f4cede00700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ce8073940 0x7f4ce81990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4cee800700 1 -- 172.21.15.16:0/3192505286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ce8073940 msgr2=0x7f4ce81990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4cee800700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ce8073940 0x7f4ce81990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4cee800700 1 -- 172.21.15.16:0/3192505286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ce8074f20 msgr2=0x7f4ce81995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4cee800700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ce8074f20 0x7f4ce81995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4cee800700 1 -- 172.21.15.16:0/3192505286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd80095f0 con 0x7f4ce8108d70 2024-04-05T23:33:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4cee800700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce8108d70 0x7f4ce81a05c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400ef30 tx=0x7f4ce400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4ce2a00700 1 -- 172.21.15.16:0/3192505286 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4018070 con 0x7f4ce8108d70 2024-04-05T23:33:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4ce2a00700 1 -- 172.21.15.16:0/3192505286 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ce400f040 con 0x7f4ce8108d70 2024-04-05T23:33:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce81a10e0 con 0x7f4ce8108d70 2024-04-05T23:33:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4ce2a00700 1 -- 172.21.15.16:0/3192505286 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400cd00 con 0x7f4ce8108d70 2024-04-05T23:33:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.399+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce81a15a0 con 0x7f4ce8108d70 2024-04-05T23:33:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.400+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce8107b30 con 0x7f4ce8108d70 2024-04-05T23:33:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.402+0000 7f4ce2a00700 1 -- 172.21.15.16:0/3192505286 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 135993+0+0 (secure 0 0 0) 0x7f4ce4003a00 con 0x7f4ce8108d70 2024-04-05T23:33:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.403+0000 7f4ce2a00700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4cd00a1290 0x7f4cd00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.403+0000 7f4ce2a00700 1 -- 172.21.15.16:0/3192505286 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f4ce40cc850 con 0x7f4ce8108d70 2024-04-05T23:33:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.404+0000 7f4cede00700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4cd00a1290 0x7f4cd00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.404+0000 7f4cede00700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4cd00a1290 0x7f4cd00a3750 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009c60 tx=0x7f4cdc009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.407+0000 7f4ce2a00700 1 -- 172.21.15.16:0/3192505286 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4ce4095d00 con 0x7f4ce8108d70 2024-04-05T23:33:37.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.629+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ce810eaa0 con 0x7f4ce8108d70 2024-04-05T23:33:37.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.629+0000 7f4ce2a00700 1 -- 172.21.15.16:0/3192505286 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ce4099320 con 0x7f4ce8108d70 2024-04-05T23:33:37.622 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:33:37.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4cd00a1290 msgr2=0x7f4cd00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4cd00a1290 0x7f4cd00a3750 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009c60 tx=0x7f4cdc009260 comp rx=0 tx=0).stop 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce8108d70 msgr2=0x7f4ce81a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce8108d70 0x7f4ce81a05c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400ef30 tx=0x7f4ce400c3a0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 shutdown_connections 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4cd00a1290 0x7f4cd00a3750 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ce8108d70 0x7f4ce81a05c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ce8073940 0x7f4ce81990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 --2- 172.21.15.16:0/3192505286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ce8074f20 0x7f4ce81995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 >> 172.21.15.16:0/3192505286 conn(0x7f4ce80fc420 msgr2=0x7f4ce8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 shutdown_connections 2024-04-05T23:33:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.632+0000 7f4cf0800700 1 -- 172.21.15.16:0/3192505286 wait complete. 2024-04-05T23:33:37.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-05T23:33:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.859+0000 7f2e77a00700 1 -- 172.21.15.16:0/449995934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e70108d80 msgr2=0x7f2e7010b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.859+0000 7f2e77a00700 1 --2- 172.21.15.16:0/449995934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e70108d80 0x7f2e7010b210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800b0a0 tx=0x7f2e6800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.859+0000 7f2e77a00700 1 -- 172.21.15.16:0/449995934 shutdown_connections 2024-04-05T23:33:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.859+0000 7f2e77a00700 1 --2- 172.21.15.16:0/449995934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e70108d80 0x7f2e7010b210 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.859+0000 7f2e77a00700 1 --2- 172.21.15.16:0/449995934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e70074f20 0x7f2e70073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.859+0000 7f2e77a00700 1 --2- 172.21.15.16:0/449995934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70073940 0x7f2e70073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.859+0000 7f2e77a00700 1 -- 172.21.15.16:0/449995934 >> 172.21.15.16:0/449995934 conn(0x7f2e700fc430 msgr2=0x7f2e700fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:37.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.859+0000 7f2e77a00700 1 -- 172.21.15.16:0/449995934 shutdown_connections 2024-04-05T23:33:37.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.859+0000 7f2e77a00700 1 -- 172.21.15.16:0/449995934 wait complete. 2024-04-05T23:33:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.860+0000 7f2e77a00700 1 Processor -- start 2024-04-05T23:33:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.860+0000 7f2e77a00700 1 -- start start 2024-04-05T23:33:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e77a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e70073940 0x7f2e70199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e77a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70074f20 0x7f2e701996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e77a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e70108d80 0x7f2e701a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e77a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e701a0c90 con 0x7f2e70074f20 2024-04-05T23:33:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e77a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e701a0dd0 con 0x7f2e70073940 2024-04-05T23:33:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e77a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e701a0f60 con 0x7f2e70108d80 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e6fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70074f20 0x7f2e701996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e6fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70074f20 0x7f2e701996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48318/0 (socket says 172.21.15.16:48318) 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e6fe00700 1 -- 172.21.15.16:0/2400994945 learned_addr learned my addr 172.21.15.16:0/2400994945 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e75000700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e70073940 0x7f2e70199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e75a00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e70108d80 0x7f2e701a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e6fe00700 1 -- 172.21.15.16:0/2400994945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e70108d80 msgr2=0x7f2e701a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e6fe00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e70108d80 0x7f2e701a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e6fe00700 1 -- 172.21.15.16:0/2400994945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e70073940 msgr2=0x7f2e70199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e6fe00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e70073940 0x7f2e70199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.861+0000 7f2e6fe00700 1 -- 172.21.15.16:0/2400994945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68009d00 con 0x7f2e70074f20 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.862+0000 7f2e6fe00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70074f20 0x7f2e701996a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00e970 tx=0x7f2e5c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.862+0000 7f2e6d600700 1 -- 172.21.15.16:0/2400994945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c009790 con 0x7f2e70074f20 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.862+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e701a1240 con 0x7f2e70074f20 2024-04-05T23:33:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.862+0000 7f2e6d600700 1 -- 172.21.15.16:0/2400994945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e5c00eec0 con 0x7f2e70074f20 2024-04-05T23:33:37.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.862+0000 7f2e6d600700 1 -- 172.21.15.16:0/2400994945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c003cd0 con 0x7f2e70074f20 2024-04-05T23:33:37.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.862+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e701a17a0 con 0x7f2e70074f20 2024-04-05T23:33:37.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.863+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e70107b40 con 0x7f2e70074f20 2024-04-05T23:33:37.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.865+0000 7f2e6d600700 1 -- 172.21.15.16:0/2400994945 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 135993+0+0 (secure 0 0 0) 0x7f2e5c00ccd0 con 0x7f2e70074f20 2024-04-05T23:33:37.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.867+0000 7f2e6d600700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2e580a1290 0x7f2e580a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:37.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.868+0000 7f2e6d600700 1 -- 172.21.15.16:0/2400994945 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f2e5c014070 con 0x7f2e70074f20 2024-04-05T23:33:37.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.868+0000 7f2e75000700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2e580a1290 0x7f2e580a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.868+0000 7f2e75000700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2e580a1290 0x7f2e580a3750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009c60 tx=0x7f2e64009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:37.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:37.872+0000 7f2e6d600700 1 -- 172.21.15.16:0/2400994945 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2e5c096300 con 0x7f2e70074f20 2024-04-05T23:33:37.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi130 to 182.4M 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: Unable to set osd_memory_target on smithi130 to 191357644: error parsing value: Value '191357644' is below minimum 939524096 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi027 to 182.4M 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: Unable to set osd_memory_target on smithi027 to 191349760: error parsing value: Value '191349760' is below minimum 939524096 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/421503923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:37.968 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:37 smithi027 ceph-mon[78597]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-05T23:33:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.088+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2e7010ebb0 con 0x7f2e70074f20 2024-04-05T23:33:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.089+0000 7f2e6d600700 1 -- 172.21.15.16:0/2400994945 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2e5c099920 con 0x7f2e70074f20 2024-04-05T23:33:38.082 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-05T23:33:38.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:33:38.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-05T23:33:38.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:33:38.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi130 to 182.4M 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: Unable to set osd_memory_target on smithi130 to 191357644: error parsing value: Value '191357644' is below minimum 939524096 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi027 to 182.4M 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: Unable to set osd_memory_target on smithi027 to 191349760: error parsing value: Value '191349760' is below minimum 939524096 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/421503923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:38.084 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:37 smithi016 ceph-mon[38248]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-05T23:33:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:33:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-05T23:33:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:33:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi130 to 182.4M 2024-04-05T23:33:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: Unable to set osd_memory_target on smithi130 to 191357644: error parsing value: Value '191357644' is below minimum 939524096 2024-04-05T23:33:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi027 to 182.4M 2024-04-05T23:33:38.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: Unable to set osd_memory_target on smithi027 to 191349760: error parsing value: Value '191349760' is below minimum 939524096 2024-04-05T23:33:38.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/421503923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:38.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:37 smithi130 ceph-mon[79417]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-05T23:33:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2e580a1290 msgr2=0x7f2e580a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2e580a1290 0x7f2e580a3750 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009c60 tx=0x7f2e64009260 comp rx=0 tx=0).stop 2024-04-05T23:33:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70074f20 msgr2=0x7f2e701996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70074f20 0x7f2e701996a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00e970 tx=0x7f2e5c00ec80 comp rx=0 tx=0).stop 2024-04-05T23:33:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 shutdown_connections 2024-04-05T23:33:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2e580a1290 0x7f2e580a3750 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e70108d80 0x7f2e701a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e70073940 0x7f2e70199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 --2- 172.21.15.16:0/2400994945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70074f20 0x7f2e701996a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 >> 172.21.15.16:0/2400994945 conn(0x7f2e700fc430 msgr2=0x7f2e70104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 shutdown_connections 2024-04-05T23:33:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.092+0000 7f2e77a00700 1 -- 172.21.15.16:0/2400994945 wait complete. 2024-04-05T23:33:38.104 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-05T23:33:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.321+0000 7f16bc000700 1 -- 172.21.15.16:0/2898610320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4102a10 msgr2=0x7f16b4109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.321+0000 7f16bc000700 1 --2- 172.21.15.16:0/2898610320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4102a10 0x7f16b4109330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b0a0 tx=0x7f16ac00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.321+0000 7f16bc000700 1 -- 172.21.15.16:0/2898610320 shutdown_connections 2024-04-05T23:33:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.321+0000 7f16bc000700 1 --2- 172.21.15.16:0/2898610320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b4100e70 0x7f16b4101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.321+0000 7f16bc000700 1 --2- 172.21.15.16:0/2898610320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4102a10 0x7f16b4109330 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.321+0000 7f16bc000700 1 --2- 172.21.15.16:0/2898610320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16b4102050 0x7f16b41024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.321+0000 7f16bc000700 1 -- 172.21.15.16:0/2898610320 >> 172.21.15.16:0/2898610320 conn(0x7f16b40fc430 msgr2=0x7f16b40fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.321+0000 7f16bc000700 1 -- 172.21.15.16:0/2898610320 shutdown_connections 2024-04-05T23:33:38.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.321+0000 7f16bc000700 1 -- 172.21.15.16:0/2898610320 wait complete. 2024-04-05T23:33:38.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.322+0000 7f16bc000700 1 Processor -- start 2024-04-05T23:33:38.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.322+0000 7f16bc000700 1 -- start start 2024-04-05T23:33:38.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16bc000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4100e70 0x7f16b4074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:38.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16bc000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b4102050 0x7f16b40732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:38.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16bc000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16b4102a10 0x7f16b4073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:38.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16bc000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b4073fb0 con 0x7f16b4102050 2024-04-05T23:33:38.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16bc000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b4074120 con 0x7f16b4102a10 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16bc000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b41a0f90 con 0x7f16b4100e70 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16b9600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4100e70 0x7f16b4074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16ba000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16b4102a10 0x7f16b4073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16b9600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4100e70 0x7f16b4074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:41454/0 (socket says 172.21.15.16:41454) 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16b9600700 1 -- 172.21.15.16:0/917766935 learned_addr learned my addr 172.21.15.16:0/917766935 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.323+0000 7f16b8c00700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b4102050 0x7f16b40732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16ba000700 1 -- 172.21.15.16:0/917766935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4100e70 msgr2=0x7f16b4074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16ba000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4100e70 0x7f16b4074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16ba000700 1 -- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b4102050 msgr2=0x7f16b40732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16ba000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b4102050 0x7f16b40732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16ba000700 1 -- 172.21.15.16:0/917766935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac009d00 con 0x7f16b4102a10 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16b9600700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4100e70 0x7f16b4074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16ba000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16b4102a10 0x7f16b4073810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f16ac007b90 tx=0x7f16ac007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:38.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16b2000700 1 -- 172.21.15.16:0/917766935 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac00e040 con 0x7f16b4102a10 2024-04-05T23:33:38.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16b2000700 1 -- 172.21.15.16:0/917766935 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16ac00be40 con 0x7f16b4102a10 2024-04-05T23:33:38.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b41a11c0 con 0x7f16b4102a10 2024-04-05T23:33:38.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16b2000700 1 -- 172.21.15.16:0/917766935 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac01b460 con 0x7f16b4102a10 2024-04-05T23:33:38.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.324+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b41a1590 con 0x7f16b4102a10 2024-04-05T23:33:38.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.325+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b41082a0 con 0x7f16b4102a10 2024-04-05T23:33:38.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.327+0000 7f16b2000700 1 -- 172.21.15.16:0/917766935 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 135993+0+0 (secure 0 0 0) 0x7f16ac01d470 con 0x7f16b4102a10 2024-04-05T23:33:38.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.329+0000 7f16b2000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f169c0a1290 0x7f169c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:38.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.329+0000 7f16b2000700 1 -- 172.21.15.16:0/917766935 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f16ac0ce5e0 con 0x7f16b4102a10 2024-04-05T23:33:38.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.329+0000 7f16b9600700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f169c0a1290 0x7f169c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:38.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.330+0000 7f16b9600700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f169c0a1290 0x7f169c0a3750 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f16a8005950 tx=0x7f16a8009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:38.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.331+0000 7f16b2000700 1 -- 172.21.15.16:0/917766935 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f16ac097bb0 con 0x7f16b4102a10 2024-04-05T23:33:38.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.553+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16b410c730 con 0x7f16b4102a10 2024-04-05T23:33:38.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.554+0000 7f16b2000700 1 -- 172.21.15.16:0/917766935 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f16ac09b1d0 con 0x7f16b4102a10 2024-04-05T23:33:38.547 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-05T23:33:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f169c0a1290 msgr2=0x7f169c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f169c0a1290 0x7f169c0a3750 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f16a8005950 tx=0x7f16a8009260 comp rx=0 tx=0).stop 2024-04-05T23:33:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16b4102a10 msgr2=0x7f16b4073810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16b4102a10 0x7f16b4073810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f16ac007b90 tx=0x7f16ac007930 comp rx=0 tx=0).stop 2024-04-05T23:33:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 shutdown_connections 2024-04-05T23:33:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f169c0a1290 0x7f169c0a3750 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16b4102a10 0x7f16b4073810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16b4100e70 0x7f16b4074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 --2- 172.21.15.16:0/917766935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b4102050 0x7f16b40732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 >> 172.21.15.16:0/917766935 conn(0x7f16b40fc430 msgr2=0x7f16b41053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 shutdown_connections 2024-04-05T23:33:38.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.556+0000 7f16bc000700 1 -- 172.21.15.16:0/917766935 wait complete. 2024-04-05T23:33:38.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:33:38.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.789+0000 7f6a7b400700 1 -- 172.21.15.16:0/2706458462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74101880 msgr2=0x7f6a74103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.789+0000 7f6a7b400700 1 --2- 172.21.15.16:0/2706458462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74101880 0x7f6a74103c70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009990 tx=0x7f6a68009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.789+0000 7f6a7b400700 1 -- 172.21.15.16:0/2706458462 shutdown_connections 2024-04-05T23:33:38.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.789+0000 7f6a7b400700 1 --2- 172.21.15.16:0/2706458462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a74106ae0 0x7f6a74108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.789+0000 7f6a7b400700 1 --2- 172.21.15.16:0/2706458462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74101880 0x7f6a74103c70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.789+0000 7f6a7b400700 1 --2- 172.21.15.16:0/2706458462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a741041b0 0x7f6a741065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.789+0000 7f6a7b400700 1 -- 172.21.15.16:0/2706458462 >> 172.21.15.16:0/2706458462 conn(0x7f6a740fb230 msgr2=0x7f6a740fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:38.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.789+0000 7f6a7b400700 1 -- 172.21.15.16:0/2706458462 shutdown_connections 2024-04-05T23:33:38.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.789+0000 7f6a7b400700 1 -- 172.21.15.16:0/2706458462 wait complete. 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.790+0000 7f6a7b400700 1 Processor -- start 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.790+0000 7f6a7b400700 1 -- start start 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a7b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a74101880 0x7f6a74198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a7b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a741041b0 0x7f6a74199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a7b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74106ae0 0x7f6a741a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a7b400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a741a0a70 con 0x7f6a741041b0 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a7b400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a741a0bb0 con 0x7f6a74106ae0 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a7b400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a741a0d40 con 0x7f6a74101880 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74106ae0 0x7f6a741a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:38.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a73e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a741041b0 0x7f6a74199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:38.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a78a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a74101880 0x7f6a74198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:38.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74106ae0 0x7f6a741a04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33340/0 (socket says 172.21.15.16:33340) 2024-04-05T23:33:38.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a79400700 1 -- 172.21.15.16:0/4123529656 learned_addr learned my addr 172.21.15.16:0/4123529656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:38.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a78a00700 1 -- 172.21.15.16:0/4123529656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74106ae0 msgr2=0x7f6a741a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a78a00700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74106ae0 0x7f6a741a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a78a00700 1 -- 172.21.15.16:0/4123529656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a741041b0 msgr2=0x7f6a74199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:38.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a78a00700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a741041b0 0x7f6a74199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:38.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a78a00700 1 -- 172.21.15.16:0/4123529656 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680095f0 con 0x7f6a74101880 2024-04-05T23:33:38.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.791+0000 7f6a79400700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74106ae0 0x7f6a741a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:38.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.792+0000 7f6a78a00700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a74101880 0x7f6a74198f40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6a680098e0 tx=0x7f6a68009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:38.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.792+0000 7f6a71600700 1 -- 172.21.15.16:0/4123529656 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6801d070 con 0x7f6a74101880 2024-04-05T23:33:38.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.792+0000 7f6a71600700 1 -- 172.21.15.16:0/4123529656 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a68021410 con 0x7f6a74101880 2024-04-05T23:33:38.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.792+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a741a0fc0 con 0x7f6a74101880 2024-04-05T23:33:38.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.792+0000 7f6a71600700 1 -- 172.21.15.16:0/4123529656 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68005720 con 0x7f6a74101880 2024-04-05T23:33:38.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.792+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a741a1380 con 0x7f6a74101880 2024-04-05T23:33:38.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.793+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a74108680 con 0x7f6a74101880 2024-04-05T23:33:38.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.795+0000 7f6a71600700 1 -- 172.21.15.16:0/4123529656 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 23) v1 ==== 135993+0+0 (secure 0 0 0) 0x7f6a68021580 con 0x7f6a74101880 2024-04-05T23:33:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.796+0000 7f6a71600700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a5c0a1310 0x7f6a5c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.797+0000 7f6a71600700 1 -- 172.21.15.16:0/4123529656 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f6a680cdf50 con 0x7f6a74101880 2024-04-05T23:33:38.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.797+0000 7f6a73e00700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a5c0a1310 0x7f6a5c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:38.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.797+0000 7f6a73e00700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a5c0a1310 0x7f6a5c0a37d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009c60 tx=0x7f6a60009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:38.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:38.800+0000 7f6a71600700 1 -- 172.21.15.16:0/4123529656 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6a68097400 con 0x7f6a74101880 2024-04-05T23:33:38.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3192505286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:38.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2400994945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi016 to 222.6M 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: Unable to set osd_memory_target on smithi016 to 233463910: error parsing value: Value '233463910' is below minimum 939524096 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-05T23:33:38.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:38 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/917766935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:38.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3192505286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:38.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2400994945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:38.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi016 to 222.6M 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: Unable to set osd_memory_target on smithi016 to 233463910: error parsing value: Value '233463910' is below minimum 939524096 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-05T23:33:38.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:38 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/917766935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.023+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a74067b50 con 0x7f6a74101880 2024-04-05T23:33:39.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.025+0000 7f6a71600700 1 -- 172.21.15.16:0/4123529656 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22958 (secure 0 0 0) 0x7f6a6809aa20 con 0x7f6a74101880 2024-04-05T23:33:39.018 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:39.019 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:33:34.593258+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/985211078":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1740984099":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/4157043736":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/1565223355":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2846240830":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/4184690359":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/1565223355":"2024-04-06T23:33:34.593227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a5c0a1310 msgr2=0x7f6a5c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a5c0a1310 0x7f6a5c0a37d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009c60 tx=0x7f6a60009260 comp rx=0 tx=0).stop 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a74101880 msgr2=0x7f6a74198f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a74101880 0x7f6a74198f40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6a680098e0 tx=0x7f6a68009580 comp rx=0 tx=0).stop 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 shutdown_connections 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a5c0a1310 0x7f6a5c0a37d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a74106ae0 0x7f6a741a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a74101880 0x7f6a74198f40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 --2- 172.21.15.16:0/4123529656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a741041b0 0x7f6a74199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 >> 172.21.15.16:0/4123529656 conn(0x7f6a740fb230 msgr2=0x7f6a74101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 shutdown_connections 2024-04-05T23:33:39.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.030+0000 7f6a7b400700 1 -- 172.21.15.16:0/4123529656 wait complete. 2024-04-05T23:33:39.046 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-05T23:32:02.641673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-05T23:32:03.972521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:33:39.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:33:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3192505286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2400994945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:39.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi016 to 222.6M 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: Unable to set osd_memory_target on smithi016 to 233463910: error parsing value: Value '233463910' is below minimum 939524096 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-05T23:33:39.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:38 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/917766935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:39.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.275+0000 7f9e3ce00700 1 -- 172.21.15.16:0/3099929014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e380ff710 msgr2=0x7f9e380ffb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.275+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/3099929014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e380ff710 0x7f9e380ffb30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009990 tx=0x7f9e28009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.275+0000 7f9e3ce00700 1 -- 172.21.15.16:0/3099929014 shutdown_connections 2024-04-05T23:33:39.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.275+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/3099929014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e38100a30 0x7f9e38100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.275+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/3099929014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e380ff710 0x7f9e380ffb30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.275+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/3099929014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e38100070 0x7f9e381004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.275+0000 7f9e3ce00700 1 -- 172.21.15.16:0/3099929014 >> 172.21.15.16:0/3099929014 conn(0x7f9e380fb330 msgr2=0x7f9e380fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:39.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.275+0000 7f9e3ce00700 1 -- 172.21.15.16:0/3099929014 shutdown_connections 2024-04-05T23:33:39.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.275+0000 7f9e3ce00700 1 -- 172.21.15.16:0/3099929014 wait complete. 2024-04-05T23:33:39.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.276+0000 7f9e3ce00700 1 Processor -- start 2024-04-05T23:33:39.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.276+0000 7f9e3ce00700 1 -- start start 2024-04-05T23:33:39.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e3ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e380ff710 0x7f9e38199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:39.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e3ce00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e38100070 0x7f9e38199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e3ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e38100a30 0x7f9e381a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e3ce00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e381a0b70 con 0x7f9e380ff710 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e3ce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e381a0cb0 con 0x7f9e38100a30 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e3ce00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e381a0e40 con 0x7f9e38100070 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e36800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e38100a30 0x7f9e381a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e35e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e380ff710 0x7f9e38199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e35400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e38100070 0x7f9e38199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e36800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e38100a30 0x7f9e381a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33360/0 (socket says 172.21.15.16:33360) 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e35e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e380ff710 0x7f9e38199040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48374/0 (socket says 172.21.15.16:48374) 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e36800700 1 -- 172.21.15.16:0/2300216631 learned_addr learned my addr 172.21.15.16:0/2300216631 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e36800700 1 -- 172.21.15.16:0/2300216631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e38100070 msgr2=0x7f9e38199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e36800700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e38100070 0x7f9e38199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e36800700 1 -- 172.21.15.16:0/2300216631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e380ff710 msgr2=0x7f9e38199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e36800700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e380ff710 0x7f9e38199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.277+0000 7f9e36800700 1 -- 172.21.15.16:0/2300216631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280095f0 con 0x7f9e38100a30 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.278+0000 7f9e36800700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e38100a30 0x7f9e381a05a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9e3000ef30 tx=0x7f9e3000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.278+0000 7f9e2ea00700 1 -- 172.21.15.16:0/2300216631 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e30018070 con 0x7f9e38100a30 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.278+0000 7f9e2ea00700 1 -- 172.21.15.16:0/2300216631 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e3000f040 con 0x7f9e38100a30 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.278+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e381a10c0 con 0x7f9e38100a30 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.278+0000 7f9e2ea00700 1 -- 172.21.15.16:0/2300216631 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3000cd00 con 0x7f9e38100a30 2024-04-05T23:33:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.278+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e381a15d0 con 0x7f9e38100a30 2024-04-05T23:33:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.279+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3810a780 con 0x7f9e38100a30 2024-04-05T23:33:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.281+0000 7f9e2ea00700 1 -- 172.21.15.16:0/2300216631 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f9e30003a00 con 0x7f9e38100a30 2024-04-05T23:33:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.283+0000 7f9e2ea00700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e1c0a1320 0x7f9e1c0a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.283+0000 7f9e2ea00700 1 -- 172.21.15.16:0/2300216631 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f9e300cc9f0 con 0x7f9e38100a30 2024-04-05T23:33:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.283+0000 7f9e35e00700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e1c0a1320 0x7f9e1c0a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.284+0000 7f9e35e00700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e1c0a1320 0x7f9e1c0a37e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800b2e0 tx=0x7f9e28005fd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:39.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.286+0000 7f9e2ea00700 1 -- 172.21.15.16:0/2300216631 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9e30095d30 con 0x7f9e38100a30 2024-04-05T23:33:39.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.522+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e3810a160 con 0x7f9e38100a30 2024-04-05T23:33:39.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.522+0000 7f9e2ea00700 1 -- 172.21.15.16:0/2300216631 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e30099350 con 0x7f9e38100a30 2024-04-05T23:33:39.515 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:33:39.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e1c0a1320 msgr2=0x7f9e1c0a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e1c0a1320 0x7f9e1c0a37e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800b2e0 tx=0x7f9e28005fd0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e38100a30 msgr2=0x7f9e381a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e38100a30 0x7f9e381a05a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9e3000ef30 tx=0x7f9e3000c3a0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 shutdown_connections 2024-04-05T23:33:39.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e1c0a1320 0x7f9e1c0a37e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e380ff710 0x7f9e38199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e38100a30 0x7f9e381a05a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 --2- 172.21.15.16:0/2300216631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e38100070 0x7f9e38199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 >> 172.21.15.16:0/2300216631 conn(0x7f9e380fb330 msgr2=0x7f9e380fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:39.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 shutdown_connections 2024-04-05T23:33:39.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.526+0000 7f9e3ce00700 1 -- 172.21.15.16:0/2300216631 wait complete. 2024-04-05T23:33:39.539 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.759+0000 7ffa2c000700 1 -- 172.21.15.16:0/947988197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa24100e60 msgr2=0x7ffa24101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.759+0000 7ffa2c000700 1 --2- 172.21.15.16:0/947988197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa24100e60 0x7ffa24101280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009990 tx=0x7ffa18009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.759+0000 7ffa2c000700 1 -- 172.21.15.16:0/947988197 shutdown_connections 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.759+0000 7ffa2c000700 1 --2- 172.21.15.16:0/947988197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa24102a00 0x7ffa24109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.759+0000 7ffa2c000700 1 --2- 172.21.15.16:0/947988197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa24100e60 0x7ffa24101280 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.759+0000 7ffa2c000700 1 --2- 172.21.15.16:0/947988197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa24102040 0x7ffa241024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.759+0000 7ffa2c000700 1 -- 172.21.15.16:0/947988197 >> 172.21.15.16:0/947988197 conn(0x7ffa240fc440 msgr2=0x7ffa240fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.759+0000 7ffa2c000700 1 -- 172.21.15.16:0/947988197 shutdown_connections 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.759+0000 7ffa2c000700 1 -- 172.21.15.16:0/947988197 wait complete. 2024-04-05T23:33:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.760+0000 7ffa2c000700 1 Processor -- start 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.760+0000 7ffa2c000700 1 -- start start 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.760+0000 7ffa2c000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa24100e60 0x7ffa24106c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.760+0000 7ffa2c000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa24102040 0x7ffa24107150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.760+0000 7ffa2c000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa24102a00 0x7ffa241037a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.760+0000 7ffa2c000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa24103f40 con 0x7ffa24102a00 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.760+0000 7ffa2c000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa241040b0 con 0x7ffa24100e60 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.760+0000 7ffa2c000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa24104270 con 0x7ffa24102040 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa2a000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa24102a00 0x7ffa241037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa2a000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa24102a00 0x7ffa241037a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48388/0 (socket says 172.21.15.16:48388) 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa2a000700 1 -- 172.21.15.16:0/2739476837 learned_addr learned my addr 172.21.15.16:0/2739476837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa28c00700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa24102040 0x7ffa24107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa29600700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa24100e60 0x7ffa24106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa2a000700 1 -- 172.21.15.16:0/2739476837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa24102040 msgr2=0x7ffa24107150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa2a000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa24102040 0x7ffa24107150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa2a000700 1 -- 172.21.15.16:0/2739476837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa24100e60 msgr2=0x7ffa24106c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa2a000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa24100e60 0x7ffa24106c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa2a000700 1 -- 172.21.15.16:0/2739476837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa180095f0 con 0x7ffa24102a00 2024-04-05T23:33:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.761+0000 7ffa2a000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa24102a00 0x7ffa241037a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00ef30 tx=0x7ffa1c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.762+0000 7ffa22000700 1 -- 172.21.15.16:0/2739476837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c018070 con 0x7ffa24102a00 2024-04-05T23:33:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.762+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa241a5570 con 0x7ffa24102a00 2024-04-05T23:33:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.762+0000 7ffa22000700 1 -- 172.21.15.16:0/2739476837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa1c00f040 con 0x7ffa24102a00 2024-04-05T23:33:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.762+0000 7ffa22000700 1 -- 172.21.15.16:0/2739476837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c00cca0 con 0x7ffa24102a00 2024-04-05T23:33:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.762+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa241a5aa0 con 0x7ffa24102a00 2024-04-05T23:33:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.763+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa24067c30 con 0x7ffa24102a00 2024-04-05T23:33:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.765+0000 7ffa22000700 1 -- 172.21.15.16:0/2739476837 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7ffa1c003a00 con 0x7ffa24102a00 2024-04-05T23:33:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.766+0000 7ffa22000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ffa0c0a1320 0x7ffa0c0a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.766+0000 7ffa22000700 1 -- 172.21.15.16:0/2739476837 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ffa1c0cc980 con 0x7ffa24102a00 2024-04-05T23:33:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.766+0000 7ffa29600700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ffa0c0a1320 0x7ffa0c0a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.767+0000 7ffa29600700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ffa0c0a1320 0x7ffa0c0a37e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800b680 tx=0x7ffa18005fd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:39.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:39.771+0000 7ffa22000700 1 -- 172.21.15.16:0/2739476837 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ffa1c095cc0 con 0x7ffa24102a00 2024-04-05T23:33:39.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:33:39.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:33:39.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:33:39.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:33:39.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:33:39.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4123529656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:39 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2300216631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:39.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:33:39.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:33:39.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:33:39.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:33:39.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4123529656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:39.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:39 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2300216631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:40.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.014+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa241a5d60 con 0x7ffa24102a00 2024-04-05T23:33:40.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.015+0000 7ffa22000700 1 -- 172.21.15.16:0/2739476837 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffa1c0992e0 con 0x7ffa24102a00 2024-04-05T23:33:40.008 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-05T23:33:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ffa0c0a1320 msgr2=0x7ffa0c0a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ffa0c0a1320 0x7ffa0c0a37e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800b680 tx=0x7ffa18005fd0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa24102a00 msgr2=0x7ffa241037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa24102a00 0x7ffa241037a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00ef30 tx=0x7ffa1c00c3a0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 shutdown_connections 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ffa0c0a1320 0x7ffa0c0a37e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa24102a00 0x7ffa241037a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa24100e60 0x7ffa24106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 --2- 172.21.15.16:0/2739476837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa24102040 0x7ffa24107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.017+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 >> 172.21.15.16:0/2739476837 conn(0x7ffa240fc440 msgr2=0x7ffa241078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.018+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 shutdown_connections 2024-04-05T23:33:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.018+0000 7ffa2c000700 1 -- 172.21.15.16:0/2739476837 wait complete. 2024-04-05T23:33:40.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-05T23:33:40.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:33:40.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:33:40.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:33:40.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T23:33:40.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:33:40.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:33:40.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4123529656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:40.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: Updating smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:33:40.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: Updating smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:33:40.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-05T23:33:40.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-05T23:33:40.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:39 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2300216631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.251+0000 7f7a9a000700 1 -- 172.21.15.16:0/3044120139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a94111e60 msgr2=0x7f7a94114250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.251+0000 7f7a9a000700 1 --2- 172.21.15.16:0/3044120139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a94111e60 0x7f7a94114250 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00b0a0 tx=0x7f7a8c00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.251+0000 7f7a9a000700 1 -- 172.21.15.16:0/3044120139 shutdown_connections 2024-04-05T23:33:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.251+0000 7f7a9a000700 1 --2- 172.21.15.16:0/3044120139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a94111e60 0x7f7a94114250 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.251+0000 7f7a9a000700 1 --2- 172.21.15.16:0/3044120139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a94073280 0x7f7a94073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.251+0000 7f7a9a000700 1 --2- 172.21.15.16:0/3044120139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a94073ba0 0x7f7a94111920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.252+0000 7f7a9a000700 1 -- 172.21.15.16:0/3044120139 >> 172.21.15.16:0/3044120139 conn(0x7f7a940fc920 msgr2=0x7f7a940fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.252+0000 7f7a9a000700 1 -- 172.21.15.16:0/3044120139 shutdown_connections 2024-04-05T23:33:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.252+0000 7f7a9a000700 1 -- 172.21.15.16:0/3044120139 wait complete. 2024-04-05T23:33:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.253+0000 7f7a9a000700 1 Processor -- start 2024-04-05T23:33:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.253+0000 7f7a9a000700 1 -- start start 2024-04-05T23:33:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.253+0000 7f7a9a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a94073280 0x7f7a941a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.253+0000 7f7a9a000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a94073ba0 0x7f7a941a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.253+0000 7f7a9a000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a94111e60 0x7f7a9419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.253+0000 7f7a9a000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9419df30 con 0x7f7a94073ba0 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.253+0000 7f7a9a000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9419e0a0 con 0x7f7a94073280 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.253+0000 7f7a9a000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9419e260 con 0x7f7a94111e60 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.253+0000 7f7a92a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a94073ba0 0x7f7a941a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a94073280 0x7f7a941a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a94073280 0x7f7a941a37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33398/0 (socket says 172.21.15.16:33398) 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93400700 1 -- 172.21.15.16:0/4276538745 learned_addr learned my addr 172.21.15.16:0/4276538745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93e00700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a94111e60 0x7f7a9419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93400700 1 -- 172.21.15.16:0/4276538745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a94111e60 msgr2=0x7f7a9419d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93400700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a94111e60 0x7f7a9419d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93400700 1 -- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a94073ba0 msgr2=0x7f7a941a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93400700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a94073ba0 0x7f7a941a3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93400700 1 -- 172.21.15.16:0/4276538745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a80009520 con 0x7f7a94073280 2024-04-05T23:33:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a92a00700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a94073ba0 0x7f7a941a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.254+0000 7f7a93400700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a94073280 0x7f7a941a37a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400e9a0 tx=0x7f7a8400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.255+0000 7f7a8be00700 1 -- 172.21.15.16:0/4276538745 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84009790 con 0x7f7a94073280 2024-04-05T23:33:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.255+0000 7f7a8be00700 1 -- 172.21.15.16:0/4276538745 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a84010430 con 0x7f7a94073280 2024-04-05T23:33:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.255+0000 7f7a8be00700 1 -- 172.21.15.16:0/4276538745 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84018680 con 0x7f7a94073280 2024-04-05T23:33:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.255+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8c009d00 con 0x7f7a94073280 2024-04-05T23:33:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.255+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a941aa5b0 con 0x7f7a94073280 2024-04-05T23:33:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.256+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a94104340 con 0x7f7a94073280 2024-04-05T23:33:40.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.259+0000 7f7a8be00700 1 -- 172.21.15.16:0/4276538745 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f7a840105a0 con 0x7f7a94073280 2024-04-05T23:33:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.260+0000 7f7a8be00700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7a780a1320 0x7f7a780a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.260+0000 7f7a8be00700 1 -- 172.21.15.16:0/4276538745 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f7a84014070 con 0x7f7a94073280 2024-04-05T23:33:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.261+0000 7f7a92a00700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7a780a1320 0x7f7a780a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:40.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.261+0000 7f7a92a00700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7a780a1320 0x7f7a780a37e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7a9419ea90 tx=0x7f7a80009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:40.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.262+0000 7f7a8be00700 1 -- 172.21.15.16:0/4276538745 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7a84096500 con 0x7f7a94073280 2024-04-05T23:33:40.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.483+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7a9419ed90 con 0x7f7a94073280 2024-04-05T23:33:40.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.484+0000 7f7a8be00700 1 -- 172.21.15.16:0/4276538745 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7a840260a0 con 0x7f7a94073280 2024-04-05T23:33:40.477 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-05T23:33:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7a780a1320 msgr2=0x7f7a780a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7a780a1320 0x7f7a780a37e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7a9419ea90 tx=0x7f7a80009260 comp rx=0 tx=0).stop 2024-04-05T23:33:40.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a94073280 msgr2=0x7f7a941a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a94073280 0x7f7a941a37a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400e9a0 tx=0x7f7a8400ecb0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 shutdown_connections 2024-04-05T23:33:40.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7a780a1320 0x7f7a780a37e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a94111e60 0x7f7a9419d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a94073280 0x7f7a941a37a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 --2- 172.21.15.16:0/4276538745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a94073ba0 0x7f7a941a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 >> 172.21.15.16:0/4276538745 conn(0x7f7a940fc920 msgr2=0x7f7a940fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:40.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.486+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 shutdown_connections 2024-04-05T23:33:40.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.487+0000 7f7a9a000700 1 -- 172.21.15.16:0/4276538745 wait complete. 2024-04-05T23:33:40.497 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-05T23:33:40.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:33:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.723+0000 7f9e41c00700 1 -- 172.21.15.16:0/3525617539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e3c1060e0 msgr2=0x7f9e3c10dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.723+0000 7f9e41c00700 1 --2- 172.21.15.16:0/3525617539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e3c1060e0 0x7f9e3c10dc60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400b0a0 tx=0x7f9e3400b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.723+0000 7f9e41c00700 1 -- 172.21.15.16:0/3525617539 shutdown_connections 2024-04-05T23:33:40.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.723+0000 7f9e41c00700 1 --2- 172.21.15.16:0/3525617539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e3c1060e0 0x7f9e3c10dc60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.723+0000 7f9e41c00700 1 --2- 172.21.15.16:0/3525617539 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e3c100fc0 0x7f9e3c1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.723+0000 7f9e41c00700 1 --2- 172.21.15.16:0/3525617539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e3c101970 0x7f9e3c1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.723+0000 7f9e41c00700 1 -- 172.21.15.16:0/3525617539 >> 172.21.15.16:0/3525617539 conn(0x7f9e3c0fc880 msgr2=0x7f9e3c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:40.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.723+0000 7f9e41c00700 1 -- 172.21.15.16:0/3525617539 shutdown_connections 2024-04-05T23:33:40.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.723+0000 7f9e41c00700 1 -- 172.21.15.16:0/3525617539 wait complete. 2024-04-05T23:33:40.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.724+0000 7f9e41c00700 1 Processor -- start 2024-04-05T23:33:40.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.724+0000 7f9e41c00700 1 -- start start 2024-04-05T23:33:40.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e41c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e3c100fc0 0x7f9e3c199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:40.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e41c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e3c1060e0 0x7f9e3c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:40.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e41c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e3c19de70 0x7f9e3c19e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:40.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e41c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c19e8f0 con 0x7f9e3c19de70 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e41c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c19ea60 con 0x7f9e3c100fc0 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e41c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c19ec20 con 0x7f9e3c1060e0 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e3aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e3c100fc0 0x7f9e3c199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e3b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e3c19de70 0x7f9e3c19e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e3aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e3c100fc0 0x7f9e3c199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33404/0 (socket says 172.21.15.16:33404) 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e3aa00700 1 -- 172.21.15.16:0/1832099073 learned_addr learned my addr 172.21.15.16:0/1832099073 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e3a000700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e3c1060e0 0x7f9e3c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e3b400700 1 -- 172.21.15.16:0/1832099073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e3c1060e0 msgr2=0x7f9e3c199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e3b400700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e3c1060e0 0x7f9e3c199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.725+0000 7f9e3b400700 1 -- 172.21.15.16:0/1832099073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e3c100fc0 msgr2=0x7f9e3c199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.726+0000 7f9e3b400700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e3c100fc0 0x7f9e3c199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.726+0000 7f9e3b400700 1 -- 172.21.15.16:0/1832099073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e34009d00 con 0x7f9e3c19de70 2024-04-05T23:33:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.726+0000 7f9e3aa00700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e3c100fc0 0x7f9e3c199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.726+0000 7f9e3b400700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e3c19de70 0x7f9e3c19e2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400b070 tx=0x7f9e3400bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.726+0000 7f9e33400700 1 -- 172.21.15.16:0/1832099073 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3400e040 con 0x7f9e3c19de70 2024-04-05T23:33:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.726+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3c1a6120 con 0x7f9e3c19de70 2024-04-05T23:33:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.726+0000 7f9e33400700 1 -- 172.21.15.16:0/1832099073 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e3401d480 con 0x7f9e3c19de70 2024-04-05T23:33:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.726+0000 7f9e33400700 1 -- 172.21.15.16:0/1832099073 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e340052a0 con 0x7f9e3c19de70 2024-04-05T23:33:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.726+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3c1a64f0 con 0x7f9e3c19de70 2024-04-05T23:33:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.727+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3c102490 con 0x7f9e3c19de70 2024-04-05T23:33:40.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.729+0000 7f9e33400700 1 -- 172.21.15.16:0/1832099073 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f9e3401d5f0 con 0x7f9e3c19de70 2024-04-05T23:33:40.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.730+0000 7f9e33400700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e200a1320 0x7f9e200a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:40.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.731+0000 7f9e33400700 1 -- 172.21.15.16:0/1832099073 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f9e340ce130 con 0x7f9e3c19de70 2024-04-05T23:33:40.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.731+0000 7f9e3aa00700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e200a1320 0x7f9e200a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:40.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.731+0000 7f9e3aa00700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e200a1320 0x7f9e200a37e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c011550 tx=0x7f9e2c009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:40.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.733+0000 7f9e33400700 1 -- 172.21.15.16:0/1832099073 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9e34097470 con 0x7f9e3c19de70 2024-04-05T23:33:40.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:33:40.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2739476837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:40.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4276538745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:40.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:33:40.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:33:40.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:33:40.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:40 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:33:40.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2739476837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:40.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4276538745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:40.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:33:40.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:33:40.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:33:40.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:40 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.995+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e3c067b50 con 0x7f9e3c19de70 2024-04-05T23:33:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.996+0000 7f9e33400700 1 -- 172.21.15.16:0/1832099073 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10902 (secure 0 0 0) 0x7f9e3409aa90 con 0x7f9e3c19de70 2024-04-05T23:33:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:40.989 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:33:36.604775+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:33:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e200a1320 msgr2=0x7f9e200a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e200a1320 0x7f9e200a37e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c011550 tx=0x7f9e2c009260 comp rx=0 tx=0).stop 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e3c19de70 msgr2=0x7f9e3c19e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e3c19de70 0x7f9e3c19e2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400b070 tx=0x7f9e3400bcf0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 shutdown_connections 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e200a1320 0x7f9e200a37e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e3c19de70 0x7f9e3c19e2f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e3c100fc0 0x7f9e3c199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 --2- 172.21.15.16:0/1832099073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e3c1060e0 0x7f9e3c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 >> 172.21.15.16:0/1832099073 conn(0x7f9e3c0fc880 msgr2=0x7f9e3c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 shutdown_connections 2024-04-05T23:33:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:40.998+0000 7f9e41c00700 1 -- 172.21.15.16:0/1832099073 wait complete. 2024-04-05T23:33:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-04-05T23:33:41.010 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-05T23:33:41.010 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-05T23:33:41.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: Updating smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2739476837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4276538745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:33:41.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:40 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:33:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.217+0000 7f5761600700 1 -- 172.21.15.16:0/2183186380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f575c101970 msgr2=0x7f575c1059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.217+0000 7f5761600700 1 --2- 172.21.15.16:0/2183186380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f575c101970 0x7f575c1059b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5748009990 tx=0x7f5748009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.217+0000 7f5761600700 1 -- 172.21.15.16:0/2183186380 shutdown_connections 2024-04-05T23:33:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.217+0000 7f5761600700 1 --2- 172.21.15.16:0/2183186380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f575c1060e0 0x7f575c10dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.217+0000 7f5761600700 1 --2- 172.21.15.16:0/2183186380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575c100fc0 0x7f575c1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.217+0000 7f5761600700 1 --2- 172.21.15.16:0/2183186380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f575c101970 0x7f575c1059b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.217+0000 7f5761600700 1 -- 172.21.15.16:0/2183186380 >> 172.21.15.16:0/2183186380 conn(0x7f575c0fc860 msgr2=0x7f575c0fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.217+0000 7f5761600700 1 -- 172.21.15.16:0/2183186380 shutdown_connections 2024-04-05T23:33:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.217+0000 7f5761600700 1 -- 172.21.15.16:0/2183186380 wait complete. 2024-04-05T23:33:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.218+0000 7f5761600700 1 Processor -- start 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.219+0000 7f5761600700 1 -- start start 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.219+0000 7f5761600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f575c100fc0 0x7f575c1972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.219+0000 7f5761600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575c1060e0 0x7f575c197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.219+0000 7f5761600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f575c19bb60 0x7f575c19bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.219+0000 7f5761600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c19c5e0 con 0x7f575c100fc0 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.219+0000 7f5761600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c19c750 con 0x7f575c1060e0 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.219+0000 7f5761600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c19c910 con 0x7f575c19bb60 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f575c100fc0 0x7f575c1972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f575c100fc0 0x7f575c1972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48458/0 (socket says 172.21.15.16:48458) 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575aa00700 1 -- 172.21.15.16:0/3827827308 learned_addr learned my addr 172.21.15.16:0/3827827308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575a000700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575c1060e0 0x7f575c197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575b400700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f575c19bb60 0x7f575c19bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575aa00700 1 -- 172.21.15.16:0/3827827308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f575c19bb60 msgr2=0x7f575c19bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575aa00700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f575c19bb60 0x7f575c19bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575aa00700 1 -- 172.21.15.16:0/3827827308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575c1060e0 msgr2=0x7f575c197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575aa00700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575c1060e0 0x7f575c197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575aa00700 1 -- 172.21.15.16:0/3827827308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480095f0 con 0x7f575c100fc0 2024-04-05T23:33:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.220+0000 7f575aa00700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f575c100fc0 0x7f575c1972d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f574c00e9a0 tx=0x7f574c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.221+0000 7f5753400700 1 -- 172.21.15.16:0/3827827308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c009790 con 0x7f575c100fc0 2024-04-05T23:33:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.221+0000 7f5753400700 1 -- 172.21.15.16:0/3827827308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f574c010430 con 0x7f575c100fc0 2024-04-05T23:33:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.221+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c1a3e70 con 0x7f575c100fc0 2024-04-05T23:33:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.221+0000 7f5753400700 1 -- 172.21.15.16:0/3827827308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c018680 con 0x7f575c100fc0 2024-04-05T23:33:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.221+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c1a42f0 con 0x7f575c100fc0 2024-04-05T23:33:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.222+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c1093b0 con 0x7f575c100fc0 2024-04-05T23:33:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.224+0000 7f5753400700 1 -- 172.21.15.16:0/3827827308 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f574c0105a0 con 0x7f575c100fc0 2024-04-05T23:33:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.226+0000 7f5753400700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f57400a1320 0x7f57400a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.226+0000 7f5753400700 1 -- 172.21.15.16:0/3827827308 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f574c014070 con 0x7f575c100fc0 2024-04-05T23:33:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.226+0000 7f575a000700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f57400a1320 0x7f57400a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.226+0000 7f575a000700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f57400a1320 0x7f57400a37e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f575c1987d0 tx=0x7f5748005960 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.230+0000 7f5753400700 1 -- 172.21.15.16:0/3827827308 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f574c052ab0 con 0x7f575c100fc0 2024-04-05T23:33:41.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.445+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 --> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f575c101970 con 0x7f57400a1320 2024-04-05T23:33:41.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.459+0000 7f5753400700 1 -- 172.21.15.16:0/3827827308 <== mgr.15240 v2:172.21.15.16:6800/3895420073 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343358 (secure 0 0 0) 0x7f575c101970 con 0x7f57400a1320 2024-04-05T23:33:41.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f57400a1320 msgr2=0x7f57400a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f57400a1320 0x7f57400a37e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f575c1987d0 tx=0x7f5748005960 comp rx=0 tx=0).stop 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f575c100fc0 msgr2=0x7f575c1972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f575c100fc0 0x7f575c1972d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f574c00e9a0 tx=0x7f574c00ecb0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 shutdown_connections 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f57400a1320 0x7f57400a37e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f575c100fc0 0x7f575c1972d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f575c19bb60 0x7f575c19bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 --2- 172.21.15.16:0/3827827308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575c1060e0 0x7f575c197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 >> 172.21.15.16:0/3827827308 conn(0x7f575c0fc860 msgr2=0x7f575c0fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.463+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 shutdown_connections 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.464+0000 7f5761600700 1 -- 172.21.15.16:0/3827827308 wait complete. 2024-04-05T23:33:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-05T23:33:41.475 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-05T23:33:40.613366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190190,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1133,"num_read_kb":731,"num_write":543,"num_write_kb":4398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":932,"ondisk_log_size":932,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370856,"kb_used_data":13800,"kb_used_omap":18,"kb_used_meta":356845,"kb_avail":1124570968,"statfs":{"total":1151940427776,"available":1151560671232,"internally_reserved":0,"allocated":14131200,"data_stored":8299437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":365409605},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.976347"},"pg_stats":[{"pgid":"3.3f","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673349+0000","last_change":"2024-04-05T23:32:04.536879+0000","last_active":"2024-04-05T23:33:34.673349+0000","last_peered":"2024-04-05T23:33:34.673349+0000","last_clean":"2024-04-05T23:33:34.673349+0000","last_became_active":"2024-04-05T23:32:04.536183+0000","last_became_peered":"2024-04-05T23:32:04.536183+0000","last_unstale":"2024-04-05T23:33:34.673349+0000","last_undegraded":"2024-04-05T23:33:34.673349+0000","last_fullsized":"2024-04-05T23:33:34.673349+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:13:22.589464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"82'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670374+0000","last_change":"2024-04-05T23:32:04.540791+0000","last_active":"2024-04-05T23:33:34.670374+0000","last_peered":"2024-04-05T23:33:34.670374+0000","last_clean":"2024-04-05T23:33:34.670374+0000","last_became_active":"2024-04-05T23:32:04.540455+0000","last_became_peered":"2024-04-05T23:32:04.540455+0000","last_unstale":"2024-04-05T23:33:34.670374+0000","last_undegraded":"2024-04-05T23:33:34.670374+0000","last_fullsized":"2024-04-05T23:33:34.670374+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:06:19.090672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.671697+0000","last_change":"2024-04-05T23:32:04.543428+0000","last_active":"2024-04-05T23:33:34.671697+0000","last_peered":"2024-04-05T23:33:34.671697+0000","last_clean":"2024-04-05T23:33:34.671697+0000","last_became_active":"2024-04-05T23:32:04.543071+0000","last_became_peered":"2024-04-05T23:32:04.543071+0000","last_unstale":"2024-04-05T23:33:34.671697+0000","last_undegraded":"2024-04-05T23:33:34.671697+0000","last_fullsized":"2024-04-05T23:33:34.671697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:35:44.911749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651922+0000","last_change":"2024-04-05T23:32:04.531444+0000","last_active":"2024-04-05T23:33:34.651922+0000","last_peered":"2024-04-05T23:33:34.651922+0000","last_clean":"2024-04-05T23:33:34.651922+0000","last_became_active":"2024-04-05T23:32:04.529556+0000","last_became_peered":"2024-04-05T23:32:04.529556+0000","last_unstale":"2024-04-05T23:33:34.651922+0000","last_undegraded":"2024-04-05T23:33:34.651922+0000","last_fullsized":"2024-04-05T23:33:34.651922+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:48.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"82'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669652+0000","last_change":"2024-04-05T23:32:04.529565+0000","last_active":"2024-04-05T23:33:34.669652+0000","last_peered":"2024-04-05T23:33:34.669652+0000","last_clean":"2024-04-05T23:33:34.669652+0000","last_became_active":"2024-04-05T23:32:04.529226+0000","last_became_peered":"2024-04-05T23:32:04.529226+0000","last_unstale":"2024-04-05T23:33:34.669652+0000","last_undegraded":"2024-04-05T23:33:34.669652+0000","last_fullsized":"2024-04-05T23:33:34.669652+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:44:25.208996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.655487+0000","last_change":"2024-04-05T23:32:04.546260+0000","last_active":"2024-04-05T23:33:34.655487+0000","last_peered":"2024-04-05T23:33:34.655487+0000","last_clean":"2024-04-05T23:33:34.655487+0000","last_became_active":"2024-04-05T23:32:04.545851+0000","last_became_peered":"2024-04-05T23:32:04.545851+0000","last_unstale":"2024-04-05T23:33:34.655487+0000","last_undegraded":"2024-04-05T23:33:34.655487+0000","last_fullsized":"2024-04-05T23:33:34.655487+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:59:16.735635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.665495+0000","last_change":"2024-04-05T23:32:04.536348+0000","last_active":"2024-04-05T23:33:34.665495+0000","last_peered":"2024-04-05T23:33:34.665495+0000","last_clean":"2024-04-05T23:33:34.665495+0000","last_became_active":"2024-04-05T23:32:04.536015+0000","last_became_peered":"2024-04-05T23:32:04.536015+0000","last_unstale":"2024-04-05T23:33:34.665495+0000","last_undegraded":"2024-04-05T23:33:34.665495+0000","last_fullsized":"2024-04-05T23:33:34.665495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:02:24.897957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666748+0000","last_change":"2024-04-05T23:32:04.532528+0000","last_active":"2024-04-05T23:33:34.666748+0000","last_peered":"2024-04-05T23:33:34.666748+0000","last_clean":"2024-04-05T23:33:34.666748+0000","last_became_active":"2024-04-05T23:32:04.532104+0000","last_became_peered":"2024-04-05T23:32:04.532104+0000","last_unstale":"2024-04-05T23:33:34.666748+0000","last_undegraded":"2024-04-05T23:33:34.666748+0000","last_fullsized":"2024-04-05T23:33:34.666748+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:06.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"82'11","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651120+0000","last_change":"2024-04-05T23:32:04.531223+0000","last_active":"2024-04-05T23:33:34.651120+0000","last_peered":"2024-04-05T23:33:34.651120+0000","last_clean":"2024-04-05T23:33:34.651120+0000","last_became_active":"2024-04-05T23:32:04.530863+0000","last_became_peered":"2024-04-05T23:32:04.530863+0000","last_unstale":"2024-04-05T23:33:34.651120+0000","last_undegraded":"2024-04-05T23:33:34.651120+0000","last_fullsized":"2024-04-05T23:33:34.651120+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:03:35.912984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"82'16","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.653658+0000","last_change":"2024-04-05T23:32:04.541481+0000","last_active":"2024-04-05T23:33:34.653658+0000","last_peered":"2024-04-05T23:33:34.653658+0000","last_clean":"2024-04-05T23:33:34.653658+0000","last_became_active":"2024-04-05T23:32:04.541133+0000","last_became_peered":"2024-04-05T23:32:04.541133+0000","last_unstale":"2024-04-05T23:33:34.653658+0000","last_undegraded":"2024-04-05T23:33:34.653658+0000","last_fullsized":"2024-04-05T23:33:34.653658+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:01:22.825587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.660781+0000","last_change":"2024-04-05T23:32:04.542390+0000","last_active":"2024-04-05T23:33:34.660781+0000","last_peered":"2024-04-05T23:33:34.660781+0000","last_clean":"2024-04-05T23:33:34.660781+0000","last_became_active":"2024-04-05T23:32:04.542026+0000","last_became_peered":"2024-04-05T23:32:04.542026+0000","last_unstale":"2024-04-05T23:33:34.660781+0000","last_undegraded":"2024-04-05T23:33:34.660781+0000","last_fullsized":"2024-04-05T23:33:34.660781+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:15:42.243807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651490+0000","last_change":"2024-04-05T23:32:04.540096+0000","last_active":"2024-04-05T23:33:34.651490+0000","last_peered":"2024-04-05T23:33:34.651490+0000","last_clean":"2024-04-05T23:33:34.651490+0000","last_became_active":"2024-04-05T23:32:04.539764+0000","last_became_peered":"2024-04-05T23:32:04.539764+0000","last_unstale":"2024-04-05T23:33:34.651490+0000","last_undegraded":"2024-04-05T23:33:34.651490+0000","last_fullsized":"2024-04-05T23:33:34.651490+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:57:36.442393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.665139+0000","last_change":"2024-04-05T23:32:04.540839+0000","last_active":"2024-04-05T23:33:34.665139+0000","last_peered":"2024-04-05T23:33:34.665139+0000","last_clean":"2024-04-05T23:33:34.665139+0000","last_became_active":"2024-04-05T23:32:04.540511+0000","last_became_peered":"2024-04-05T23:32:04.540511+0000","last_unstale":"2024-04-05T23:33:34.665139+0000","last_undegraded":"2024-04-05T23:33:34.665139+0000","last_fullsized":"2024-04-05T23:33:34.665139+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:58:24.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.667960+0000","last_change":"2024-04-05T23:32:04.539222+0000","last_active":"2024-04-05T23:33:34.667960+0000","last_peered":"2024-04-05T23:33:34.667960+0000","last_clean":"2024-04-05T23:33:34.667960+0000","last_became_active":"2024-04-05T23:32:04.538906+0000","last_became_peered":"2024-04-05T23:32:04.538906+0000","last_unstale":"2024-04-05T23:33:34.667960+0000","last_undegraded":"2024-04-05T23:33:34.667960+0000","last_fullsized":"2024-04-05T23:33:34.667960+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:05:24.429153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666925+0000","last_change":"2024-04-05T23:32:04.544155+0000","last_active":"2024-04-05T23:33:34.666925+0000","last_peered":"2024-04-05T23:33:34.666925+0000","last_clean":"2024-04-05T23:33:34.666925+0000","last_became_active":"2024-04-05T23:32:04.543852+0000","last_became_peered":"2024-04-05T23:32:04.543852+0000","last_unstale":"2024-04-05T23:33:34.666925+0000","last_undegraded":"2024-04-05T23:33:34.666925+0000","last_fullsized":"2024-04-05T23:33:34.666925+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:18:34.353542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.672831+0000","last_change":"2024-04-05T23:32:04.533772+0000","last_active":"2024-04-05T23:33:34.672831+0000","last_peered":"2024-04-05T23:33:34.672831+0000","last_clean":"2024-04-05T23:33:34.672831+0000","last_became_active":"2024-04-05T23:32:04.533416+0000","last_became_peered":"2024-04-05T23:32:04.533416+0000","last_unstale":"2024-04-05T23:33:34.672831+0000","last_undegraded":"2024-04-05T23:33:34.672831+0000","last_fullsized":"2024-04-05T23:33:34.672831+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:36:55.072349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673455+0000","last_change":"2024-04-05T23:32:04.551902+0000","last_active":"2024-04-05T23:33:34.673455+0000","last_peered":"2024-04-05T23:33:34.673455+0000","last_clean":"2024-04-05T23:33:34.673455+0000","last_became_active":"2024-04-05T23:32:04.551595+0000","last_became_peered":"2024-04-05T23:32:04.551595+0000","last_unstale":"2024-04-05T23:33:34.673455+0000","last_undegraded":"2024-04-05T23:33:34.673455+0000","last_fullsized":"2024-04-05T23:33:34.673455+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:14:46.648789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651142+0000","last_change":"2024-04-05T23:32:04.542967+0000","last_active":"2024-04-05T23:33:34.651142+0000","last_peered":"2024-04-05T23:33:34.651142+0000","last_clean":"2024-04-05T23:33:34.651142+0000","last_became_active":"2024-04-05T23:32:04.541767+0000","last_became_peered":"2024-04-05T23:32:04.541767+0000","last_unstale":"2024-04-05T23:33:34.651142+0000","last_undegraded":"2024-04-05T23:33:34.651142+0000","last_fullsized":"2024-04-05T23:33:34.651142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:27:46.150249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666772+0000","last_change":"2024-04-05T23:32:04.549512+0000","last_active":"2024-04-05T23:33:34.666772+0000","last_peered":"2024-04-05T23:33:34.666772+0000","last_clean":"2024-04-05T23:33:34.666772+0000","last_became_active":"2024-04-05T23:32:04.549184+0000","last_became_peered":"2024-04-05T23:32:04.549184+0000","last_unstale":"2024-04-05T23:33:34.666772+0000","last_undegraded":"2024-04-05T23:33:34.666772+0000","last_fullsized":"2024-04-05T23:33:34.666772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:48:24.186408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'14","reported_seq":115,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.941928+0000","last_change":"2024-04-05T23:32:04.005188+0000","last_active":"2024-04-05T23:33:34.941928+0000","last_peered":"2024-04-05T23:33:34.941928+0000","last_clean":"2024-04-05T23:33:34.941928+0000","last_became_active":"2024-04-05T23:32:04.004818+0000","last_became_peered":"2024-04-05T23:32:04.004818+0000","last_unstale":"2024-04-05T23:33:34.941928+0000","last_undegraded":"2024-04-05T23:33:34.941928+0000","last_fullsized":"2024-04-05T23:33:34.941928+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:49:41.146761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661564+0000","last_change":"2024-04-05T23:32:04.544881+0000","last_active":"2024-04-05T23:33:34.661564+0000","last_peered":"2024-04-05T23:33:34.661564+0000","last_clean":"2024-04-05T23:33:34.661564+0000","last_became_active":"2024-04-05T23:32:04.542149+0000","last_became_peered":"2024-04-05T23:32:04.542149+0000","last_unstale":"2024-04-05T23:33:34.661564+0000","last_undegraded":"2024-04-05T23:33:34.661564+0000","last_fullsized":"2024-04-05T23:33:34.661564+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:28:51.713053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.374976+0000","last_change":"2024-04-05T23:32:04.011450+0000","last_active":"2024-04-05T23:32:38.374976+0000","last_peered":"2024-04-05T23:32:38.374976+0000","last_clean":"2024-04-05T23:32:38.374976+0000","last_became_active":"2024-04-05T23:32:04.009607+0000","last_became_peered":"2024-04-05T23:32:04.009607+0000","last_unstale":"2024-04-05T23:32:38.374976+0000","last_undegraded":"2024-04-05T23:32:38.374976+0000","last_fullsized":"2024-04-05T23:32:38.374976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:16:45.483108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673025+0000","last_change":"2024-04-05T23:32:04.550370+0000","last_active":"2024-04-05T23:33:34.673025+0000","last_peered":"2024-04-05T23:33:34.673025+0000","last_clean":"2024-04-05T23:33:34.673025+0000","last_became_active":"2024-04-05T23:32:04.549980+0000","last_became_peered":"2024-04-05T23:32:04.549980+0000","last_unstale":"2024-04-05T23:33:34.673025+0000","last_undegraded":"2024-04-05T23:33:34.673025+0000","last_fullsized":"2024-04-05T23:33:34.673025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:44:30.704366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379630+0000","last_change":"2024-04-05T23:32:04.004169+0000","last_active":"2024-04-05T23:32:38.379630+0000","last_peered":"2024-04-05T23:32:38.379630+0000","last_clean":"2024-04-05T23:32:38.379630+0000","last_became_active":"2024-04-05T23:32:04.003646+0000","last_became_peered":"2024-04-05T23:32:04.003646+0000","last_unstale":"2024-04-05T23:32:38.379630+0000","last_undegraded":"2024-04-05T23:32:38.379630+0000","last_fullsized":"2024-04-05T23:32:38.379630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:57:02.781815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.662342+0000","last_change":"2024-04-05T23:32:04.542066+0000","last_active":"2024-04-05T23:33:34.662342+0000","last_peered":"2024-04-05T23:33:34.662342+0000","last_clean":"2024-04-05T23:33:34.662342+0000","last_became_active":"2024-04-05T23:32:04.540735+0000","last_became_peered":"2024-04-05T23:32:04.540735+0000","last_unstale":"2024-04-05T23:33:34.662342+0000","last_undegraded":"2024-04-05T23:33:34.662342+0000","last_fullsized":"2024-04-05T23:33:34.662342+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:02:03.595888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:41.225108+0000","last_change":"2024-04-05T23:32:38.396641+0000","last_active":"2024-04-05T23:32:41.225108+0000","last_peered":"2024-04-05T23:32:41.225108+0000","last_clean":"2024-04-05T23:32:41.225108+0000","last_became_active":"2024-04-05T23:32:38.379708+0000","last_became_peered":"2024-04-05T23:32:38.379708+0000","last_unstale":"2024-04-05T23:32:41.225108+0000","last_undegraded":"2024-04-05T23:32:41.225108+0000","last_fullsized":"2024-04-05T23:32:41.225108+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:00.370725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"82'15","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652724+0000","last_change":"2024-04-05T23:32:04.545780+0000","last_active":"2024-04-05T23:33:34.652724+0000","last_peered":"2024-04-05T23:33:34.652724+0000","last_clean":"2024-04-05T23:33:34.652724+0000","last_became_active":"2024-04-05T23:32:04.545279+0000","last_became_peered":"2024-04-05T23:32:04.545279+0000","last_unstale":"2024-04-05T23:33:34.652724+0000","last_undegraded":"2024-04-05T23:33:34.652724+0000","last_fullsized":"2024-04-05T23:33:34.652724+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:10.577280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":111,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.537192+0000","last_change":"2024-04-05T23:32:04.004189+0000","last_active":"2024-04-05T23:33:35.537192+0000","last_peered":"2024-04-05T23:33:35.537192+0000","last_clean":"2024-04-05T23:33:35.537192+0000","last_became_active":"2024-04-05T23:32:04.003841+0000","last_became_peered":"2024-04-05T23:32:04.003841+0000","last_unstale":"2024-04-05T23:33:35.537192+0000","last_undegraded":"2024-04-05T23:33:35.537192+0000","last_fullsized":"2024-04-05T23:33:35.537192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:33:26.898657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.649038+0000","last_change":"2024-04-05T23:32:04.529538+0000","last_active":"2024-04-05T23:33:34.649038+0000","last_peered":"2024-04-05T23:33:34.649038+0000","last_clean":"2024-04-05T23:33:34.649038+0000","last_became_active":"2024-04-05T23:32:04.529231+0000","last_became_peered":"2024-04-05T23:32:04.529231+0000","last_unstale":"2024-04-05T23:33:34.649038+0000","last_undegraded":"2024-04-05T23:33:34.649038+0000","last_fullsized":"2024-04-05T23:33:34.649038+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:49:15.608966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377155+0000","last_change":"2024-04-05T23:32:04.002763+0000","last_active":"2024-04-05T23:32:38.377155+0000","last_peered":"2024-04-05T23:32:38.377155+0000","last_clean":"2024-04-05T23:32:38.377155+0000","last_became_active":"2024-04-05T23:32:04.002410+0000","last_became_peered":"2024-04-05T23:32:04.002410+0000","last_unstale":"2024-04-05T23:32:38.377155+0000","last_undegraded":"2024-04-05T23:32:38.377155+0000","last_fullsized":"2024-04-05T23:32:38.377155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:15.193813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.667881+0000","last_change":"2024-04-05T23:32:38.461447+0000","last_active":"2024-04-05T23:33:34.667881+0000","last_peered":"2024-04-05T23:33:34.667881+0000","last_clean":"2024-04-05T23:33:34.667881+0000","last_became_active":"2024-04-05T23:32:38.460968+0000","last_became_peered":"2024-04-05T23:32:38.460968+0000","last_unstale":"2024-04-05T23:33:34.667881+0000","last_undegraded":"2024-04-05T23:33:34.667881+0000","last_fullsized":"2024-04-05T23:33:34.667881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:42:21.093881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377383+0000","last_change":"2024-04-05T23:32:04.001937+0000","last_active":"2024-04-05T23:32:38.377383+0000","last_peered":"2024-04-05T23:32:38.377383+0000","last_clean":"2024-04-05T23:32:38.377383+0000","last_became_active":"2024-04-05T23:32:04.001508+0000","last_became_peered":"2024-04-05T23:32:04.001508+0000","last_unstale":"2024-04-05T23:32:38.377383+0000","last_undegraded":"2024-04-05T23:32:38.377383+0000","last_fullsized":"2024-04-05T23:32:38.377383+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:09:20.682084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.668565+0000","last_change":"2024-04-05T23:32:04.531576+0000","last_active":"2024-04-05T23:33:34.668565+0000","last_peered":"2024-04-05T23:33:34.668565+0000","last_clean":"2024-04-05T23:33:34.668565+0000","last_became_active":"2024-04-05T23:32:04.531234+0000","last_became_peered":"2024-04-05T23:32:04.531234+0000","last_unstale":"2024-04-05T23:33:34.668565+0000","last_undegraded":"2024-04-05T23:33:34.668565+0000","last_fullsized":"2024-04-05T23:33:34.668565+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:21:57.769861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.376961+0000","last_change":"2024-04-05T23:32:04.001570+0000","last_active":"2024-04-05T23:32:38.376961+0000","last_peered":"2024-04-05T23:32:38.376961+0000","last_clean":"2024-04-05T23:32:38.376961+0000","last_became_active":"2024-04-05T23:32:04.001224+0000","last_became_peered":"2024-04-05T23:32:04.001224+0000","last_unstale":"2024-04-05T23:32:38.376961+0000","last_undegraded":"2024-04-05T23:32:38.376961+0000","last_fullsized":"2024-04-05T23:32:38.376961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:44:41.070205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"82'11","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.660070+0000","last_change":"2024-04-05T23:32:04.542029+0000","last_active":"2024-04-05T23:33:34.660070+0000","last_peered":"2024-04-05T23:33:34.660070+0000","last_clean":"2024-04-05T23:33:34.660070+0000","last_became_active":"2024-04-05T23:32:04.541684+0000","last_became_peered":"2024-04-05T23:32:04.541684+0000","last_unstale":"2024-04-05T23:33:34.660070+0000","last_undegraded":"2024-04-05T23:33:34.660070+0000","last_fullsized":"2024-04-05T23:33:34.660070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:07:06.456917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'6","reported_seq":193,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.536712+0000","last_change":"2024-04-05T23:32:04.004986+0000","last_active":"2024-04-05T23:33:35.536712+0000","last_peered":"2024-04-05T23:33:35.536712+0000","last_clean":"2024-04-05T23:33:35.536712+0000","last_became_active":"2024-04-05T23:32:04.004606+0000","last_became_peered":"2024-04-05T23:32:04.004606+0000","last_unstale":"2024-04-05T23:33:35.536712+0000","last_undegraded":"2024-04-05T23:33:35.536712+0000","last_fullsized":"2024-04-05T23:33:35.536712+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:23:41.060500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.654797+0000","last_change":"2024-04-05T23:32:04.541334+0000","last_active":"2024-04-05T23:33:34.654797+0000","last_peered":"2024-04-05T23:33:34.654797+0000","last_clean":"2024-04-05T23:33:34.654797+0000","last_became_active":"2024-04-05T23:32:04.540967+0000","last_became_peered":"2024-04-05T23:32:04.540967+0000","last_unstale":"2024-04-05T23:33:34.654797+0000","last_undegraded":"2024-04-05T23:33:34.654797+0000","last_fullsized":"2024-04-05T23:33:34.654797+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:16:33.045172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.466958+0000","last_change":"2024-04-05T23:32:38.466417+0000","last_active":"2024-04-05T23:32:38.466958+0000","last_peered":"2024-04-05T23:32:38.466958+0000","last_clean":"2024-04-05T23:32:38.466958+0000","last_became_active":"2024-04-05T23:32:38.462293+0000","last_became_peered":"2024-04-05T23:32:38.462293+0000","last_unstale":"2024-04-05T23:32:38.466958+0000","last_undegraded":"2024-04-05T23:32:38.466958+0000","last_fullsized":"2024-04-05T23:32:38.466958+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:31.118636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"82'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669698+0000","last_change":"2024-04-05T23:32:38.385818+0000","last_active":"2024-04-05T23:33:34.669698+0000","last_peered":"2024-04-05T23:33:34.669698+0000","last_clean":"2024-04-05T23:33:34.669698+0000","last_became_active":"2024-04-05T23:32:38.385423+0000","last_became_peered":"2024-04-05T23:32:38.385423+0000","last_unstale":"2024-04-05T23:33:34.669698+0000","last_undegraded":"2024-04-05T23:33:34.669698+0000","last_fullsized":"2024-04-05T23:33:34.669698+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:07:11.374783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377265+0000","last_change":"2024-04-05T23:32:04.000697+0000","last_active":"2024-04-05T23:32:38.377265+0000","last_peered":"2024-04-05T23:32:38.377265+0000","last_clean":"2024-04-05T23:32:38.377265+0000","last_became_active":"2024-04-05T23:32:04.000377+0000","last_became_peered":"2024-04-05T23:32:04.000377+0000","last_unstale":"2024-04-05T23:32:38.377265+0000","last_undegraded":"2024-04-05T23:32:38.377265+0000","last_fullsized":"2024-04-05T23:32:38.377265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:55:11.227637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.662559+0000","last_change":"2024-04-05T23:32:38.383367+0000","last_active":"2024-04-05T23:33:34.662559+0000","last_peered":"2024-04-05T23:33:34.662559+0000","last_clean":"2024-04-05T23:33:34.662559+0000","last_became_active":"2024-04-05T23:32:38.383002+0000","last_became_peered":"2024-04-05T23:32:38.383002+0000","last_unstale":"2024-04-05T23:33:34.662559+0000","last_undegraded":"2024-04-05T23:33:34.662559+0000","last_fullsized":"2024-04-05T23:33:34.662559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:30:59.947387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'36","reported_seq":234,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.547756+0000","last_change":"2024-04-05T23:32:04.007322+0000","last_active":"2024-04-05T23:33:35.547756+0000","last_peered":"2024-04-05T23:33:35.547756+0000","last_clean":"2024-04-05T23:33:35.547756+0000","last_became_active":"2024-04-05T23:32:04.006971+0000","last_became_peered":"2024-04-05T23:32:04.006971+0000","last_unstale":"2024-04-05T23:33:35.547756+0000","last_undegraded":"2024-04-05T23:33:35.547756+0000","last_fullsized":"2024-04-05T23:33:35.547756+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:04:38.413482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65485,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":183,"num_read_kb":155,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661408+0000","last_change":"2024-04-05T23:32:04.535342+0000","last_active":"2024-04-05T23:33:34.661408+0000","last_peered":"2024-04-05T23:33:34.661408+0000","last_clean":"2024-04-05T23:33:34.661408+0000","last_became_active":"2024-04-05T23:32:04.535053+0000","last_became_peered":"2024-04-05T23:32:04.535053+0000","last_unstale":"2024-04-05T23:33:34.661408+0000","last_undegraded":"2024-04-05T23:33:34.661408+0000","last_fullsized":"2024-04-05T23:33:34.661408+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:19:20.683836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:41.225091+0000","last_change":"2024-04-05T23:32:38.391418+0000","last_active":"2024-04-05T23:32:41.225091+0000","last_peered":"2024-04-05T23:32:41.225091+0000","last_clean":"2024-04-05T23:32:41.225091+0000","last_became_active":"2024-04-05T23:32:38.379643+0000","last_became_peered":"2024-04-05T23:32:38.379643+0000","last_unstale":"2024-04-05T23:32:41.225091+0000","last_undegraded":"2024-04-05T23:32:41.225091+0000","last_fullsized":"2024-04-05T23:32:41.225091+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:37:12.343153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.668971+0000","last_change":"2024-04-05T23:32:04.536455+0000","last_active":"2024-04-05T23:33:34.668971+0000","last_peered":"2024-04-05T23:33:34.668971+0000","last_clean":"2024-04-05T23:33:34.668971+0000","last_became_active":"2024-04-05T23:32:04.533994+0000","last_became_peered":"2024-04-05T23:32:04.533994+0000","last_unstale":"2024-04-05T23:33:34.668971+0000","last_undegraded":"2024-04-05T23:33:34.668971+0000","last_fullsized":"2024-04-05T23:33:34.668971+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:38:18.952823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.376958+0000","last_change":"2024-04-05T23:32:04.000708+0000","last_active":"2024-04-05T23:32:38.376958+0000","last_peered":"2024-04-05T23:32:38.376958+0000","last_clean":"2024-04-05T23:32:38.376958+0000","last_became_active":"2024-04-05T23:32:04.000354+0000","last_became_peered":"2024-04-05T23:32:04.000354+0000","last_unstale":"2024-04-05T23:32:38.376958+0000","last_undegraded":"2024-04-05T23:32:38.376958+0000","last_fullsized":"2024-04-05T23:32:38.376958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:05:34.291607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"82'15","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666642+0000","last_change":"2024-04-05T23:32:04.532146+0000","last_active":"2024-04-05T23:33:34.666642+0000","last_peered":"2024-04-05T23:33:34.666642+0000","last_clean":"2024-04-05T23:33:34.666642+0000","last_became_active":"2024-04-05T23:32:04.531735+0000","last_became_peered":"2024-04-05T23:32:04.531735+0000","last_unstale":"2024-04-05T23:33:34.666642+0000","last_undegraded":"2024-04-05T23:33:34.666642+0000","last_fullsized":"2024-04-05T23:33:34.666642+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:52:44.898871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'8","reported_seq":114,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.540783+0000","last_change":"2024-04-05T23:32:04.000144+0000","last_active":"2024-04-05T23:33:35.540783+0000","last_peered":"2024-04-05T23:33:35.540783+0000","last_clean":"2024-04-05T23:33:35.540783+0000","last_became_active":"2024-04-05T23:32:03.999712+0000","last_became_peered":"2024-04-05T23:32:03.999712+0000","last_unstale":"2024-04-05T23:33:35.540783+0000","last_undegraded":"2024-04-05T23:33:35.540783+0000","last_fullsized":"2024-04-05T23:33:35.540783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:43.775018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1632,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":92,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.658416+0000","last_change":"2024-04-05T23:32:04.537623+0000","last_active":"2024-04-05T23:33:34.658416+0000","last_peered":"2024-04-05T23:33:34.658416+0000","last_clean":"2024-04-05T23:33:34.658416+0000","last_became_active":"2024-04-05T23:32:04.537399+0000","last_became_peered":"2024-04-05T23:32:04.537399+0000","last_unstale":"2024-04-05T23:33:34.658416+0000","last_undegraded":"2024-04-05T23:33:34.658416+0000","last_fullsized":"2024-04-05T23:33:34.658416+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:01:01.643133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378658+0000","last_change":"2024-04-05T23:32:04.006806+0000","last_active":"2024-04-05T23:32:38.378658+0000","last_peered":"2024-04-05T23:32:38.378658+0000","last_clean":"2024-04-05T23:32:38.378658+0000","last_became_active":"2024-04-05T23:32:04.006397+0000","last_became_peered":"2024-04-05T23:32:04.006397+0000","last_unstale":"2024-04-05T23:32:38.378658+0000","last_undegraded":"2024-04-05T23:32:38.378658+0000","last_fullsized":"2024-04-05T23:32:38.378658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:42:01.758863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.653257+0000","last_change":"2024-04-05T23:32:04.545952+0000","last_active":"2024-04-05T23:33:34.653257+0000","last_peered":"2024-04-05T23:33:34.653257+0000","last_clean":"2024-04-05T23:33:34.653257+0000","last_became_active":"2024-04-05T23:32:04.545528+0000","last_became_peered":"2024-04-05T23:32:04.545528+0000","last_unstale":"2024-04-05T23:33:34.653257+0000","last_undegraded":"2024-04-05T23:33:34.653257+0000","last_fullsized":"2024-04-05T23:33:34.653257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:33:06.074173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:33:32.204346+0000","last_change":"2024-04-05T23:32:03.999841+0000","last_active":"2024-04-05T23:33:32.204346+0000","last_peered":"2024-04-05T23:33:32.204346+0000","last_clean":"2024-04-05T23:33:32.204346+0000","last_became_active":"2024-04-05T23:32:03.999418+0000","last_became_peered":"2024-04-05T23:32:03.999418+0000","last_unstale":"2024-04-05T23:33:32.204346+0000","last_undegraded":"2024-04-05T23:33:32.204346+0000","last_fullsized":"2024-04-05T23:33:32.204346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:00:41.452167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.650643+0000","last_change":"2024-04-05T23:32:04.530012+0000","last_active":"2024-04-05T23:33:34.650643+0000","last_peered":"2024-04-05T23:33:34.650643+0000","last_clean":"2024-04-05T23:33:34.650643+0000","last_became_active":"2024-04-05T23:32:04.529628+0000","last_became_peered":"2024-04-05T23:32:04.529628+0000","last_unstale":"2024-04-05T23:33:34.650643+0000","last_undegraded":"2024-04-05T23:33:34.650643+0000","last_fullsized":"2024-04-05T23:33:34.650643+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:40:22.036170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379485+0000","last_change":"2024-04-05T23:32:04.006157+0000","last_active":"2024-04-05T23:32:38.379485+0000","last_peered":"2024-04-05T23:32:38.379485+0000","last_clean":"2024-04-05T23:32:38.379485+0000","last_became_active":"2024-04-05T23:32:04.005853+0000","last_became_peered":"2024-04-05T23:32:04.005853+0000","last_unstale":"2024-04-05T23:32:38.379485+0000","last_undegraded":"2024-04-05T23:32:38.379485+0000","last_fullsized":"2024-04-05T23:32:38.379485+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:48:39.849306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.658363+0000","last_change":"2024-04-05T23:32:04.547367+0000","last_active":"2024-04-05T23:33:34.658363+0000","last_peered":"2024-04-05T23:33:34.658363+0000","last_clean":"2024-04-05T23:33:34.658363+0000","last_became_active":"2024-04-05T23:32:04.547040+0000","last_became_peered":"2024-04-05T23:32:04.547040+0000","last_unstale":"2024-04-05T23:33:34.658363+0000","last_undegraded":"2024-04-05T23:33:34.658363+0000","last_fullsized":"2024-04-05T23:33:34.658363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:41:41.998984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'7","reported_seq":109,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.559448+0000","last_change":"2024-04-05T23:32:04.009851+0000","last_active":"2024-04-05T23:33:35.559448+0000","last_peered":"2024-04-05T23:33:35.559448+0000","last_clean":"2024-04-05T23:33:35.559448+0000","last_became_active":"2024-04-05T23:32:04.009460+0000","last_became_peered":"2024-04-05T23:32:04.009460+0000","last_unstale":"2024-04-05T23:33:35.559448+0000","last_undegraded":"2024-04-05T23:33:35.559448+0000","last_fullsized":"2024-04-05T23:33:35.559448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:39:41.000439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1326,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652783+0000","last_change":"2024-04-05T23:32:04.549621+0000","last_active":"2024-04-05T23:33:34.652783+0000","last_peered":"2024-04-05T23:33:34.652783+0000","last_clean":"2024-04-05T23:33:34.652783+0000","last_became_active":"2024-04-05T23:32:04.549325+0000","last_became_peered":"2024-04-05T23:32:04.549325+0000","last_unstale":"2024-04-05T23:33:34.652783+0000","last_undegraded":"2024-04-05T23:33:34.652783+0000","last_fullsized":"2024-04-05T23:33:34.652783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:11:07.101463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652570+0000","last_change":"2024-04-05T23:32:04.546768+0000","last_active":"2024-04-05T23:33:34.652570+0000","last_peered":"2024-04-05T23:33:34.652570+0000","last_clean":"2024-04-05T23:33:34.652570+0000","last_became_active":"2024-04-05T23:32:04.546471+0000","last_became_peered":"2024-04-05T23:32:04.546471+0000","last_unstale":"2024-04-05T23:33:34.652570+0000","last_undegraded":"2024-04-05T23:33:34.652570+0000","last_fullsized":"2024-04-05T23:33:34.652570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:48:22.728976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":109,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.621543+0000","last_change":"2024-04-05T23:32:04.007093+0000","last_active":"2024-04-05T23:33:34.621543+0000","last_peered":"2024-04-05T23:33:34.621543+0000","last_clean":"2024-04-05T23:33:34.621543+0000","last_became_active":"2024-04-05T23:32:04.006836+0000","last_became_peered":"2024-04-05T23:32:04.006836+0000","last_unstale":"2024-04-05T23:33:34.621543+0000","last_undegraded":"2024-04-05T23:33:34.621543+0000","last_fullsized":"2024-04-05T23:33:34.621543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:20:42.489368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669973+0000","last_change":"2024-04-05T23:32:04.538204+0000","last_active":"2024-04-05T23:33:34.669973+0000","last_peered":"2024-04-05T23:33:34.669973+0000","last_clean":"2024-04-05T23:33:34.669973+0000","last_became_active":"2024-04-05T23:32:04.535254+0000","last_became_peered":"2024-04-05T23:32:04.535254+0000","last_unstale":"2024-04-05T23:33:34.669973+0000","last_undegraded":"2024-04-05T23:33:34.669973+0000","last_fullsized":"2024-04-05T23:33:34.669973+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:06:01.059339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.460974+0000","last_change":"2024-04-05T23:32:03.997704+0000","last_active":"2024-04-05T23:32:38.460974+0000","last_peered":"2024-04-05T23:32:38.460974+0000","last_clean":"2024-04-05T23:32:38.460974+0000","last_became_active":"2024-04-05T23:32:03.997275+0000","last_became_peered":"2024-04-05T23:32:03.997275+0000","last_unstale":"2024-04-05T23:32:38.460974+0000","last_undegraded":"2024-04-05T23:32:38.460974+0000","last_fullsized":"2024-04-05T23:32:38.460974+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:58.355104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":337,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378621+0000","last_change":"2024-04-05T23:31:10.395041+0000","last_active":"2024-04-05T23:32:38.378621+0000","last_peered":"2024-04-05T23:32:38.378621+0000","last_clean":"2024-04-05T23:32:38.378621+0000","last_became_active":"2024-04-05T23:31:10.394479+0000","last_became_peered":"2024-04-05T23:31:10.394479+0000","last_unstale":"2024-04-05T23:32:38.378621+0000","last_undegraded":"2024-04-05T23:32:38.378621+0000","last_fullsized":"2024-04-05T23:32:38.378621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_clean_scrub_stamp":"2024-04-05T23:28:33.945697+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:54:10.683526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.649757+0000","last_change":"2024-04-05T23:32:38.385626+0000","last_active":"2024-04-05T23:33:34.649757+0000","last_peered":"2024-04-05T23:33:34.649757+0000","last_clean":"2024-04-05T23:33:34.649757+0000","last_became_active":"2024-04-05T23:32:38.384654+0000","last_became_peered":"2024-04-05T23:32:38.384654+0000","last_unstale":"2024-04-05T23:33:34.649757+0000","last_undegraded":"2024-04-05T23:33:34.649757+0000","last_fullsized":"2024-04-05T23:33:34.649757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:55:42.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378826+0000","last_change":"2024-04-05T23:32:04.005978+0000","last_active":"2024-04-05T23:32:38.378826+0000","last_peered":"2024-04-05T23:32:38.378826+0000","last_clean":"2024-04-05T23:32:38.378826+0000","last_became_active":"2024-04-05T23:32:04.005638+0000","last_became_peered":"2024-04-05T23:32:04.005638+0000","last_unstale":"2024-04-05T23:32:38.378826+0000","last_undegraded":"2024-04-05T23:32:38.378826+0000","last_fullsized":"2024-04-05T23:32:38.378826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:39:35.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379513+0000","last_change":"2024-04-05T23:32:04.006727+0000","last_active":"2024-04-05T23:32:38.379513+0000","last_peered":"2024-04-05T23:32:38.379513+0000","last_clean":"2024-04-05T23:32:38.379513+0000","last_became_active":"2024-04-05T23:32:04.006414+0000","last_became_peered":"2024-04-05T23:32:04.006414+0000","last_unstale":"2024-04-05T23:32:38.379513+0000","last_undegraded":"2024-04-05T23:32:38.379513+0000","last_fullsized":"2024-04-05T23:32:38.379513+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:41:28.661118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.664992+0000","last_change":"2024-04-05T23:32:04.536893+0000","last_active":"2024-04-05T23:33:34.664992+0000","last_peered":"2024-04-05T23:33:34.664992+0000","last_clean":"2024-04-05T23:33:34.664992+0000","last_became_active":"2024-04-05T23:32:04.536629+0000","last_became_peered":"2024-04-05T23:32:04.536629+0000","last_unstale":"2024-04-05T23:33:34.664992+0000","last_undegraded":"2024-04-05T23:33:34.664992+0000","last_fullsized":"2024-04-05T23:33:34.664992+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:27:58.279768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'44","reported_seq":82,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.876599+0000","last_change":"2024-04-05T23:32:04.010202+0000","last_active":"2024-04-05T23:33:34.876599+0000","last_peered":"2024-04-05T23:33:34.876599+0000","last_clean":"2024-04-05T23:33:34.876599+0000","last_became_active":"2024-04-05T23:32:04.009889+0000","last_became_peered":"2024-04-05T23:32:04.009889+0000","last_unstale":"2024-04-05T23:33:34.876599+0000","last_undegraded":"2024-04-05T23:33:34.876599+0000","last_fullsized":"2024-04-05T23:33:34.876599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:12:13.121449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.667696+0000","last_change":"2024-04-05T23:32:04.533576+0000","last_active":"2024-04-05T23:33:34.667696+0000","last_peered":"2024-04-05T23:33:34.667696+0000","last_clean":"2024-04-05T23:33:34.667696+0000","last_became_active":"2024-04-05T23:32:04.533066+0000","last_became_peered":"2024-04-05T23:32:04.533066+0000","last_unstale":"2024-04-05T23:33:34.667696+0000","last_undegraded":"2024-04-05T23:33:34.667696+0000","last_fullsized":"2024-04-05T23:33:34.667696+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:01:44.019210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.461143+0000","last_change":"2024-04-05T23:32:04.005096+0000","last_active":"2024-04-05T23:32:38.461143+0000","last_peered":"2024-04-05T23:32:38.461143+0000","last_clean":"2024-04-05T23:32:38.461143+0000","last_became_active":"2024-04-05T23:32:04.004702+0000","last_became_peered":"2024-04-05T23:32:04.004702+0000","last_unstale":"2024-04-05T23:32:38.461143+0000","last_undegraded":"2024-04-05T23:32:38.461143+0000","last_fullsized":"2024-04-05T23:32:38.461143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:33:09.934039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670944+0000","last_change":"2024-04-05T23:32:04.535623+0000","last_active":"2024-04-05T23:33:34.670944+0000","last_peered":"2024-04-05T23:33:34.670944+0000","last_clean":"2024-04-05T23:33:34.670944+0000","last_became_active":"2024-04-05T23:32:04.535223+0000","last_became_peered":"2024-04-05T23:32:04.535223+0000","last_unstale":"2024-04-05T23:33:34.670944+0000","last_undegraded":"2024-04-05T23:33:34.670944+0000","last_fullsized":"2024-04-05T23:33:34.670944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:24:43.543119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377010+0000","last_change":"2024-04-05T23:32:04.004097+0000","last_active":"2024-04-05T23:32:38.377010+0000","last_peered":"2024-04-05T23:32:38.377010+0000","last_clean":"2024-04-05T23:32:38.377010+0000","last_became_active":"2024-04-05T23:32:04.003713+0000","last_became_peered":"2024-04-05T23:32:04.003713+0000","last_unstale":"2024-04-05T23:32:38.377010+0000","last_undegraded":"2024-04-05T23:32:38.377010+0000","last_fullsized":"2024-04-05T23:32:38.377010+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:47:43.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"82'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.627575+0000","last_change":"2024-04-05T23:32:04.538843+0000","last_active":"2024-04-05T23:33:34.627575+0000","last_peered":"2024-04-05T23:33:34.627575+0000","last_clean":"2024-04-05T23:33:34.627575+0000","last_became_active":"2024-04-05T23:32:04.538472+0000","last_became_peered":"2024-04-05T23:32:04.538472+0000","last_unstale":"2024-04-05T23:33:34.627575+0000","last_undegraded":"2024-04-05T23:33:34.627575+0000","last_fullsized":"2024-04-05T23:33:34.627575+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:40:40.863815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.373470+0000","last_change":"2024-04-05T23:32:04.013064+0000","last_active":"2024-04-05T23:32:38.373470+0000","last_peered":"2024-04-05T23:32:38.373470+0000","last_clean":"2024-04-05T23:32:38.373470+0000","last_became_active":"2024-04-05T23:32:04.012751+0000","last_became_peered":"2024-04-05T23:32:04.012751+0000","last_unstale":"2024-04-05T23:32:38.373470+0000","last_undegraded":"2024-04-05T23:32:38.373470+0000","last_fullsized":"2024-04-05T23:32:38.373470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:06:10.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673173+0000","last_change":"2024-04-05T23:32:04.529014+0000","last_active":"2024-04-05T23:33:34.673173+0000","last_peered":"2024-04-05T23:33:34.673173+0000","last_clean":"2024-04-05T23:33:34.673173+0000","last_became_active":"2024-04-05T23:32:04.528622+0000","last_became_peered":"2024-04-05T23:32:04.528622+0000","last_unstale":"2024-04-05T23:33:34.673173+0000","last_undegraded":"2024-04-05T23:33:34.673173+0000","last_fullsized":"2024-04-05T23:33:34.673173+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:46:26.390934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377465+0000","last_change":"2024-04-05T23:32:04.002651+0000","last_active":"2024-04-05T23:32:38.377465+0000","last_peered":"2024-04-05T23:32:38.377465+0000","last_clean":"2024-04-05T23:32:38.377465+0000","last_became_active":"2024-04-05T23:32:04.002346+0000","last_became_peered":"2024-04-05T23:32:04.002346+0000","last_unstale":"2024-04-05T23:32:38.377465+0000","last_undegraded":"2024-04-05T23:32:38.377465+0000","last_fullsized":"2024-04-05T23:32:38.377465+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:26:48.176254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669156+0000","last_change":"2024-04-05T23:32:04.538785+0000","last_active":"2024-04-05T23:33:34.669156+0000","last_peered":"2024-04-05T23:33:34.669156+0000","last_clean":"2024-04-05T23:33:34.669156+0000","last_became_active":"2024-04-05T23:32:04.538463+0000","last_became_peered":"2024-04-05T23:32:04.538463+0000","last_unstale":"2024-04-05T23:33:34.669156+0000","last_undegraded":"2024-04-05T23:33:34.669156+0000","last_fullsized":"2024-04-05T23:33:34.669156+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:44:38.941857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379009+0000","last_change":"2024-04-05T23:32:04.003760+0000","last_active":"2024-04-05T23:32:38.379009+0000","last_peered":"2024-04-05T23:32:38.379009+0000","last_clean":"2024-04-05T23:32:38.379009+0000","last_became_active":"2024-04-05T23:32:04.003389+0000","last_became_peered":"2024-04-05T23:32:04.003389+0000","last_unstale":"2024-04-05T23:32:38.379009+0000","last_undegraded":"2024-04-05T23:32:38.379009+0000","last_fullsized":"2024-04-05T23:32:38.379009+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:42:00.353651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661087+0000","last_change":"2024-04-05T23:32:04.540051+0000","last_active":"2024-04-05T23:33:34.661087+0000","last_peered":"2024-04-05T23:33:34.661087+0000","last_clean":"2024-04-05T23:33:34.661087+0000","last_became_active":"2024-04-05T23:32:04.539736+0000","last_became_peered":"2024-04-05T23:32:04.539736+0000","last_unstale":"2024-04-05T23:33:34.661087+0000","last_undegraded":"2024-04-05T23:33:34.661087+0000","last_fullsized":"2024-04-05T23:33:34.661087+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:54:14.086904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'7","reported_seq":110,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.559133+0000","last_change":"2024-04-05T23:32:04.006492+0000","last_active":"2024-04-05T23:33:35.559133+0000","last_peered":"2024-04-05T23:33:35.559133+0000","last_clean":"2024-04-05T23:33:35.559133+0000","last_became_active":"2024-04-05T23:32:04.006154+0000","last_became_peered":"2024-04-05T23:32:04.006154+0000","last_unstale":"2024-04-05T23:33:35.559133+0000","last_undegraded":"2024-04-05T23:33:35.559133+0000","last_fullsized":"2024-04-05T23:33:35.559133+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:53:23.431400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1908,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661003+0000","last_change":"2024-04-05T23:32:04.540757+0000","last_active":"2024-04-05T23:33:34.661003+0000","last_peered":"2024-04-05T23:33:34.661003+0000","last_clean":"2024-04-05T23:33:34.661003+0000","last_became_active":"2024-04-05T23:32:04.540359+0000","last_became_peered":"2024-04-05T23:32:04.540359+0000","last_unstale":"2024-04-05T23:33:34.661003+0000","last_undegraded":"2024-04-05T23:33:34.661003+0000","last_fullsized":"2024-04-05T23:33:34.661003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:43:52.533489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'41","reported_seq":64,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.873377+0000","last_change":"2024-04-05T23:32:04.002736+0000","last_active":"2024-04-05T23:33:34.873377+0000","last_peered":"2024-04-05T23:33:34.873377+0000","last_clean":"2024-04-05T23:33:34.873377+0000","last_became_active":"2024-04-05T23:32:04.002277+0000","last_became_peered":"2024-04-05T23:32:04.002277+0000","last_unstale":"2024-04-05T23:33:34.873377+0000","last_undegraded":"2024-04-05T23:33:34.873377+0000","last_fullsized":"2024-04-05T23:33:34.873377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:56:39.001732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670914+0000","last_change":"2024-04-05T23:32:04.537076+0000","last_active":"2024-04-05T23:33:34.670914+0000","last_peered":"2024-04-05T23:33:34.670914+0000","last_clean":"2024-04-05T23:33:34.670914+0000","last_became_active":"2024-04-05T23:32:04.536707+0000","last_became_peered":"2024-04-05T23:32:04.536707+0000","last_unstale":"2024-04-05T23:33:34.670914+0000","last_undegraded":"2024-04-05T23:33:34.670914+0000","last_fullsized":"2024-04-05T23:33:34.670914+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:51:09.395408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378236+0000","last_change":"2024-04-05T23:32:04.006122+0000","last_active":"2024-04-05T23:32:38.378236+0000","last_peered":"2024-04-05T23:32:38.378236+0000","last_clean":"2024-04-05T23:32:38.378236+0000","last_became_active":"2024-04-05T23:32:04.005683+0000","last_became_peered":"2024-04-05T23:32:04.005683+0000","last_unstale":"2024-04-05T23:32:38.378236+0000","last_undegraded":"2024-04-05T23:32:38.378236+0000","last_fullsized":"2024-04-05T23:32:38.378236+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:21:40.922614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661469+0000","last_change":"2024-04-05T23:32:04.533321+0000","last_active":"2024-04-05T23:33:34.661469+0000","last_peered":"2024-04-05T23:33:34.661469+0000","last_clean":"2024-04-05T23:33:34.661469+0000","last_became_active":"2024-04-05T23:32:04.532949+0000","last_became_peered":"2024-04-05T23:32:04.532949+0000","last_unstale":"2024-04-05T23:33:34.661469+0000","last_undegraded":"2024-04-05T23:33:34.661469+0000","last_fullsized":"2024-04-05T23:33:34.661469+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:31.500132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.648745+0000","last_change":"2024-04-05T23:32:04.536508+0000","last_active":"2024-04-05T23:33:34.648745+0000","last_peered":"2024-04-05T23:33:34.648745+0000","last_clean":"2024-04-05T23:33:34.648745+0000","last_became_active":"2024-04-05T23:32:04.536142+0000","last_became_peered":"2024-04-05T23:32:04.536142+0000","last_unstale":"2024-04-05T23:33:34.648745+0000","last_undegraded":"2024-04-05T23:33:34.648745+0000","last_fullsized":"2024-04-05T23:33:34.648745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:18:28.497807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"82'12","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652638+0000","last_change":"2024-04-05T23:32:04.545511+0000","last_active":"2024-04-05T23:33:34.652638+0000","last_peered":"2024-04-05T23:33:34.652638+0000","last_clean":"2024-04-05T23:33:34.652638+0000","last_became_active":"2024-04-05T23:32:04.545191+0000","last_became_peered":"2024-04-05T23:32:04.545191+0000","last_unstale":"2024-04-05T23:33:34.652638+0000","last_undegraded":"2024-04-05T23:33:34.652638+0000","last_fullsized":"2024-04-05T23:33:34.652638+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:17:06.431429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.668751+0000","last_change":"2024-04-05T23:32:04.549673+0000","last_active":"2024-04-05T23:33:34.668751+0000","last_peered":"2024-04-05T23:33:34.668751+0000","last_clean":"2024-04-05T23:33:34.668751+0000","last_became_active":"2024-04-05T23:32:04.549391+0000","last_became_peered":"2024-04-05T23:32:04.549391+0000","last_unstale":"2024-04-05T23:33:34.668751+0000","last_undegraded":"2024-04-05T23:33:34.668751+0000","last_fullsized":"2024-04-05T23:33:34.668751+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:30:22.689107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666260+0000","last_change":"2024-04-05T23:32:04.535735+0000","last_active":"2024-04-05T23:33:34.666260+0000","last_peered":"2024-04-05T23:33:34.666260+0000","last_clean":"2024-04-05T23:33:34.666260+0000","last_became_active":"2024-04-05T23:32:04.534461+0000","last_became_peered":"2024-04-05T23:32:04.534461+0000","last_unstale":"2024-04-05T23:33:34.666260+0000","last_undegraded":"2024-04-05T23:33:34.666260+0000","last_fullsized":"2024-04-05T23:33:34.666260+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:37:41.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"82'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.671261+0000","last_change":"2024-04-05T23:32:04.532340+0000","last_active":"2024-04-05T23:33:34.671261+0000","last_peered":"2024-04-05T23:33:34.671261+0000","last_clean":"2024-04-05T23:33:34.671261+0000","last_became_active":"2024-04-05T23:32:04.530053+0000","last_became_peered":"2024-04-05T23:32:04.530053+0000","last_unstale":"2024-04-05T23:33:34.671261+0000","last_undegraded":"2024-04-05T23:33:34.671261+0000","last_fullsized":"2024-04-05T23:33:34.671261+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:16:11.168366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.646897+0000","last_change":"2024-04-05T23:32:04.529972+0000","last_active":"2024-04-05T23:33:34.646897+0000","last_peered":"2024-04-05T23:33:34.646897+0000","last_clean":"2024-04-05T23:33:34.646897+0000","last_became_active":"2024-04-05T23:32:04.529551+0000","last_became_peered":"2024-04-05T23:32:04.529551+0000","last_unstale":"2024-04-05T23:33:34.646897+0000","last_undegraded":"2024-04-05T23:33:34.646897+0000","last_fullsized":"2024-04-05T23:33:34.646897+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:13.764971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"82'15","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.648772+0000","last_change":"2024-04-05T23:32:04.529166+0000","last_active":"2024-04-05T23:33:34.648772+0000","last_peered":"2024-04-05T23:33:34.648772+0000","last_clean":"2024-04-05T23:33:34.648772+0000","last_became_active":"2024-04-05T23:32:04.528785+0000","last_became_peered":"2024-04-05T23:32:04.528785+0000","last_unstale":"2024-04-05T23:33:34.648772+0000","last_undegraded":"2024-04-05T23:33:34.648772+0000","last_fullsized":"2024-04-05T23:33:34.648772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:29:43.964265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"82'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.660708+0000","last_change":"2024-04-05T23:32:04.540631+0000","last_active":"2024-04-05T23:33:34.660708+0000","last_peered":"2024-04-05T23:33:34.660708+0000","last_clean":"2024-04-05T23:33:34.660708+0000","last_became_active":"2024-04-05T23:32:04.540327+0000","last_became_peered":"2024-04-05T23:32:04.540327+0000","last_unstale":"2024-04-05T23:33:34.660708+0000","last_undegraded":"2024-04-05T23:33:34.660708+0000","last_fullsized":"2024-04-05T23:33:34.660708+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:22:48.825324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.656475+0000","last_change":"2024-04-05T23:32:04.544845+0000","last_active":"2024-04-05T23:33:34.656475+0000","last_peered":"2024-04-05T23:33:34.656475+0000","last_clean":"2024-04-05T23:33:34.656475+0000","last_became_active":"2024-04-05T23:32:04.544472+0000","last_became_peered":"2024-04-05T23:32:04.544472+0000","last_unstale":"2024-04-05T23:33:34.656475+0000","last_undegraded":"2024-04-05T23:33:34.656475+0000","last_fullsized":"2024-04-05T23:33:34.656475+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:34:09.529644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652655+0000","last_change":"2024-04-05T23:32:38.382577+0000","last_active":"2024-04-05T23:33:34.652655+0000","last_peered":"2024-04-05T23:33:34.652655+0000","last_clean":"2024-04-05T23:33:34.652655+0000","last_became_active":"2024-04-05T23:32:38.380286+0000","last_became_peered":"2024-04-05T23:32:38.380286+0000","last_unstale":"2024-04-05T23:33:34.652655+0000","last_undegraded":"2024-04-05T23:33:34.652655+0000","last_fullsized":"2024-04-05T23:33:34.652655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:42:52.270935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670550+0000","last_change":"2024-04-05T23:32:04.534631+0000","last_active":"2024-04-05T23:33:34.670550+0000","last_peered":"2024-04-05T23:33:34.670550+0000","last_clean":"2024-04-05T23:33:34.670550+0000","last_became_active":"2024-04-05T23:32:04.534330+0000","last_became_peered":"2024-04-05T23:32:04.534330+0000","last_unstale":"2024-04-05T23:33:34.670550+0000","last_undegraded":"2024-04-05T23:33:34.670550+0000","last_fullsized":"2024-04-05T23:33:34.670550+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:20:47.592174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"82'11","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669338+0000","last_change":"2024-04-05T23:32:04.531459+0000","last_active":"2024-04-05T23:33:34.669338+0000","last_peered":"2024-04-05T23:33:34.669338+0000","last_clean":"2024-04-05T23:33:34.669338+0000","last_became_active":"2024-04-05T23:32:04.530828+0000","last_became_peered":"2024-04-05T23:32:04.530828+0000","last_unstale":"2024-04-05T23:33:34.669338+0000","last_undegraded":"2024-04-05T23:33:34.669338+0000","last_fullsized":"2024-04-05T23:33:34.669338+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:51:34.090030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652270+0000","last_change":"2024-04-05T23:32:04.527711+0000","last_active":"2024-04-05T23:33:34.652270+0000","last_peered":"2024-04-05T23:33:34.652270+0000","last_clean":"2024-04-05T23:33:34.652270+0000","last_became_active":"2024-04-05T23:32:04.527374+0000","last_became_peered":"2024-04-05T23:32:04.527374+0000","last_unstale":"2024-04-05T23:33:34.652270+0000","last_undegraded":"2024-04-05T23:33:34.652270+0000","last_fullsized":"2024-04-05T23:33:34.652270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:15:10.764340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599588,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":997,"num_read_kb":612,"num_write":360,"num_write_kb":1458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2162688,"data_stored":1798764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579931,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":440050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.60599999999999998,"15min":0.59599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.968,"5min":3.968,"15min":3.968},"last":0.434}]},{"osd":1,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.64400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80400000000000005,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.64100000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69399999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.48199999999999998,"15min":0.48999999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67400000000000004,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":4,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.79300000000000004,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.75},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80400000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.66400000000000003}]},{"osd":5,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":1.353,"15min":1.353},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79400000000000004,"5min":1.306,"15min":1.306},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.74299999999999999,"15min":0.751},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85399999999999998,"5min":1.109,"15min":1.109},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.751,"15min":0.76600000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.89400000000000002,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.71699999999999997}]},{"osd":8,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64300000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.78500000000000003,"15min":0.80300000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.92800000000000005,"5min":1.71,"15min":1.71},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.74199999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.95699999999999996,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.78300000000000003}]},{"osd":10,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.78500000000000003,"15min":0.79700000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.94299999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.80100000000000005,"15min":0.81799999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.877,"5min":1.667,"15min":1.667},"last":0.73599999999999999}]}]},{"osd":10,"up_from":67,"seq":287762808863,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1744896,"data_stored":1228526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64000000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.35699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.878,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.372}]},{"osd":2,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66700000000000004,"15min":0.66800000000000004},"min":{"1min":0.52900000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65500000000000003,"15min":0.65900000000000003},"min":{"1min":0.435,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.81699999999999995,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.72399999999999998,"15min":0.70399999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.8660000000000001,"5min":3.8660000000000001,"15min":3.8660000000000001},"last":0.43099999999999999}]},{"osd":4,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81499999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.308},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83099999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.39000000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.94299999999999995,"5min":1.145,"15min":1.145},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0269999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.48499999999999999,"5min":0.441,"15min":0.441},"max":{"1min":0.96199999999999997,"5min":1.228,"15min":1.228},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.76700000000000002,"15min":0.77100000000000002},"min":{"1min":0.51200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.92600000000000005,"5min":1.129,"15min":1.129},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79900000000000004,"15min":0.80100000000000005},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0129999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.79200000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.042,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.41999999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.85399999999999998,"15min":0.86699999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0109999999999999,"5min":1.444,"15min":1.444},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.82299999999999995,"15min":0.83299999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.89900000000000002,"5min":1.258,"15min":1.258},"last":0.505}]},{"osd":11,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.82399999999999995,"15min":0.82499999999999996},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0149999999999999,"5min":1.351,"15min":1.351},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.80600000000000005,"15min":0.80900000000000005},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.94999999999999996,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.629}]}]},{"osd":9,"up_from":62,"seq":266287972387,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":439298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.55500000000000005,"15min":0.57099999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67700000000000005,"5min":0.746,"15min":0.746},"last":0.45800000000000002}]},{"osd":1,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.234,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77900000000000003,"15min":0.79700000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1060000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.70599999999999996}]},{"osd":2,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.74299999999999999,"15min":0.76300000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90900000000000003,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.754,"15min":0.77200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.036,"5min":1.256,"15min":1.256},"last":0.72299999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.78700000000000003,"15min":0.81100000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95699999999999996,"5min":1.27,"15min":1.27},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.71899999999999997,"15min":0.74099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.97599999999999998,"5min":1.405,"15min":1.405},"last":0.65300000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0329999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.87,"15min":0.88600000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.86699999999999999,"15min":0.89300000000000002},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.98399999999999999,"5min":1.637,"15min":1.637},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.89600000000000002,"15min":0.91900000000000004},"min":{"1min":0.55800000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1299999999999999,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.745}]},{"osd":6,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.94099999999999995,"15min":0.96899999999999997},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.071,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.94499999999999995,"15min":0.95899999999999996},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1619999999999999,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.68300000000000005}]},{"osd":8,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.0069999999999999,"15min":1.0309999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.173,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.996,"15min":1.02},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.127,"5min":1.778,"15min":1.778},"last":0.84799999999999998}]},{"osd":10,"last update":"Fri Apr 5 23:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.95499999999999996,"15min":0.97699999999999998},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.2110000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.05},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.89500000000000002,"15min":0.91100000000000003},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.111,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.79200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]}]},{"osd":8,"up_from":55,"seq":236223201320,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1314816,"data_stored":832310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56200000000000006,"15min":0.57399999999999995},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68300000000000005,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.63800000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.79400000000000004}]},{"osd":2,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.46200000000000002,"15min":0.44800000000000001},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.46400000000000002,"15min":0.45900000000000002},"min":{"1min":0.308,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.65300000000000002,"5min":0.748,"15min":0.748},"last":0.496}]},{"osd":3,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.45900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":1.101,"15min":1.101},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.55000000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.626,"15min":0.61399999999999999},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.91000000000000003,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64300000000000002,"15min":0.64100000000000001},"min":{"1min":0.38500000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85999999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.66600000000000004}]},{"osd":5,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.35499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69499999999999995,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.40500000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.58599999999999997}]},{"osd":6,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.68899999999999995,"15min":0.69199999999999995},"min":{"1min":0.48699999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.56899999999999995}]},{"osd":7,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.878,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94699999999999995,"5min":1.276,"15min":1.276},"last":0.75700000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72299999999999998,"15min":0.71299999999999997},"min":{"1min":0.56799999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73899999999999999,"15min":0.73399999999999999},"min":{"1min":0.60499999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.68999999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.73799999999999999,"15min":0.72099999999999997},"min":{"1min":0.66600000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.73999999999999999,"15min":0.72399999999999998},"min":{"1min":0.66900000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.77500000000000002}]},{"osd":11,"last update":"Fri Apr 5 23:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.74299999999999999}]}]},{"osd":7,"up_from":48,"seq":206158430250,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32604,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712548,"statfs":{"total":95995035648,"available":95961649152,"internally_reserved":0,"allocated":1523712,"data_stored":1033930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.76400000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64800000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.434,"15min":0.42199999999999999},"min":{"1min":0.36199999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.34499999999999997}]},{"osd":2,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.63300000000000001,"5min":0.879,"15min":0.879},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.58299999999999996},"min":{"1min":0.438,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.69099999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52300000000000002}]},{"osd":3,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58999999999999997,"15min":0.58399999999999996},"min":{"1min":0.42699999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.621,"15min":0.60999999999999999},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.872,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.497}]},{"osd":4,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.51300000000000001},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.68999999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.505,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.64000000000000001,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.39000000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.61699999999999999,"15min":0.621},"min":{"1min":0.438,"5min":0.378,"15min":0.378},"max":{"1min":0.73699999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65400000000000003,"15min":0.65000000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.83799999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.56799999999999995,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67900000000000005,"15min":0.67800000000000005},"min":{"1min":0.47999999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.73999999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.47599999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.49299999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.81000000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72999999999999998,"15min":0.72899999999999998},"min":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94899999999999995,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.65200000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.77200000000000002,"15min":0.77400000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.96999999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.79400000000000004,"15min":0.79600000000000004},"min":{"1min":0.67400000000000004,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.875,"5min":1.216,"15min":1.216},"last":0.68200000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.52400000000000002,"5min":0.498,"15min":0.498},"max":{"1min":0.95199999999999996,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.623},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.755,"15min":0.751},"min":{"1min":0.50600000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.92700000000000005,"5min":1.099,"15min":1.099},"last":0.55200000000000005}]},{"osd":11,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.67300000000000004}]}]},{"osd":6,"up_from":42,"seq":180388626478,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":909312,"data_stored":439480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.34200000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.39300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.92500000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.65800000000000003,"15min":0.63800000000000001},"min":{"1min":0.50900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.85999999999999999}]},{"osd":2,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63,"15min":0.628},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.004,"5min":0.73599999999999999,"15min":0.69699999999999995},"min":{"1min":0.47499999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":6.867,"5min":6.867,"15min":6.867},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.44700000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.095,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.68999999999999995,"15min":0.67400000000000004},"min":{"1min":0.46600000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.169,"5min":1.246,"15min":1.246},"last":0.89800000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.47399999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.45700000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.76500000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.79400000000000004,"15min":0.753},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":6.9189999999999996,"5min":6.9189999999999996,"15min":6.9189999999999996},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.3180000000000001,"5min":0.86199999999999999,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":6.9009999999999998,"5min":6.9009999999999998,"15min":6.9009999999999998},"last":0.97999999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.58199999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":6.883,"5min":6.883,"15min":6.883},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.84099999999999997,"15min":0.80200000000000005},"min":{"1min":0.61799999999999999,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":4.8559999999999999,"5min":4.8559999999999999,"15min":4.8559999999999999},"last":0.94999999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":0.83899999999999997,"15min":0.80900000000000005},"min":{"1min":0.64900000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.81200000000000006,"15min":0.79600000000000004},"min":{"1min":0.58199999999999996,"5min":0.504,"15min":0.504},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":1}]},{"osd":9,"last update":"Fri Apr 5 23:32:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91200000000000003,"15min":0.91600000000000004},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.63400000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.123,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.93500000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.877,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.91800000000000004,"15min":0.91600000000000004},"min":{"1min":0.74199999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":1.0209999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607618,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36160,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708992,"statfs":{"total":95995035648,"available":95958007808,"internally_reserved":0,"allocated":974848,"data_stored":505878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52100000000000002,"15min":0.504},"min":{"1min":0.45000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68799999999999994,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61499999999999999,"15min":0.67600000000000005},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.79900000000000004,"5min":4.4199999999999999,"15min":4.4199999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Apr 5 23:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55200000000000005,"15min":0.52000000000000002},"min":{"1min":0.34699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.57299999999999995,"15min":0.52100000000000002},"min":{"1min":0.53400000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59499999999999997,"15min":0.54500000000000004},"min":{"1min":0.44900000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56999999999999995,"15min":0.54400000000000004},"min":{"1min":0.45100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89200000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Apr 5 23:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.42799999999999999,"15min":0.40500000000000003},"min":{"1min":0.33700000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.495},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.46800000000000003,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.877,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.52100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91600000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.71399999999999997,"15min":0.67200000000000004},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.67900000000000005}]},{"osd":6,"last update":"Fri Apr 5 23:32:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70999999999999996,"15min":0.68600000000000005},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.438},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66200000000000003,"15min":0.65200000000000002},"min":{"1min":0.441,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57599999999999996}]},{"osd":7,"last update":"Fri Apr 5 23:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.63700000000000001,"15min":0.61099999999999999},"min":{"1min":0.40400000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90300000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.57499999999999996},"min":{"1min":0.38,"5min":0.312,"15min":0.312},"max":{"1min":0.81899999999999995,"5min":1.002,"15min":1.002},"last":0.436}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.64200000000000002,"15min":0.62},"min":{"1min":0.54100000000000004,"5min":0.378,"15min":0.378},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.754,"15min":0.71799999999999997},"min":{"1min":0.71399999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.71399999999999997}]},{"osd":9,"last update":"Fri Apr 5 23:32:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.72299999999999998,"15min":0.71099999999999997},"min":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.69899999999999995,"15min":0.68999999999999995},"min":{"1min":0.60699999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.627}]},{"osd":10,"last update":"Fri Apr 5 23:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86699999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.505},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.80500000000000005,"15min":0.79500000000000004},"min":{"1min":0.59499999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.56299999999999994}]},{"osd":11,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.748}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36124,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709028,"statfs":{"total":95995035648,"available":95958044672,"internally_reserved":0,"allocated":933888,"data_stored":441092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 23:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57299999999999995,"15min":0.51500000000000001},"min":{"1min":0.40300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.97199999999999998,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56100000000000005,"15min":0.50700000000000001},"min":{"1min":0.38100000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85999999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.63200000000000001,"15min":0.58799999999999997},"min":{"1min":0.47999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0049999999999999,"5min":7.2060000000000004,"15min":7.2060000000000004},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:33:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.91600000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.78900000000000003,"15min":0.91500000000000004},"min":{"1min":0.51800000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92500000000000004,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":1.258},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.66000000000000003},"min":{"1min":0.57999999999999996,"5min":0.436,"15min":0.436},"max":{"1min":0.88200000000000001,"5min":1.395,"15min":1.395},"last":0.79900000000000004}]},{"osd":5,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68899999999999995,"15min":0.66800000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.2250000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.70999999999999996},"min":{"1min":0.55900000000000005,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":1.339}]},{"osd":6,"last update":"Fri Apr 5 23:32:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.83399999999999996,"15min":0.87},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.5489999999999999,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.2390000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78400000000000003,"15min":0.77700000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":1.294}]},{"osd":7,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.70699999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.89200000000000002,"15min":0.93000000000000005},"min":{"1min":0.499,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.2789999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.306},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.93700000000000006,"15min":0.995},"min":{"1min":0.629,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4610000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.2010000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:32:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.94999999999999996,"15min":0.96499999999999997},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.88100000000000001,"15min":0.876},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.3919999999999999}]},{"osd":10,"last update":"Fri Apr 5 23:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.91300000000000003,"15min":0.92600000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":1.379},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.94199999999999995,"15min":0.95799999999999996},"min":{"1min":0.66900000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3080000000000001,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.2789999999999999}]},{"osd":11,"last update":"Fri Apr 5 23:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.89800000000000002,"15min":0.90200000000000002},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.92000000000000004,"15min":0.92900000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":1.415}]}]},{"osd":2,"up_from":17,"seq":73014444095,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":987136,"data_stored":504410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76400000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.751,"5min":0.874,"15min":0.874},"last":0.44}]},{"osd":3,"last update":"Fri Apr 5 23:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.311,"15min":0.311},"max":{"1min":0.82299999999999995,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.62}]},{"osd":4,"last update":"Fri Apr 5 23:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.503,"15min":0.47799999999999998},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86699999999999999,"5min":1.359,"15min":1.359},"last":0.60499999999999998}]},{"osd":6,"last update":"Fri Apr 5 23:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.45100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.58999999999999997},"min":{"1min":0.443,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.76100000000000001,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.49399999999999999}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73199999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.48399999999999999,"15min":0.47299999999999998},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70499999999999996,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.63400000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.874,"5min":1.532,"15min":1.532},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85399999999999998,"5min":1.429,"15min":1.429},"last":0.66800000000000004}]},{"osd":10,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.82099999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72499999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.88700000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.65200000000000002}]},{"osd":11,"last update":"Fri Apr 5 23:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.69599999999999995}]}]},{"osd":3,"up_from":23,"seq":98784247866,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36628,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708524,"statfs":{"total":95995035648,"available":95957528576,"internally_reserved":0,"allocated":1454080,"data_stored":963383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.45600000000000002},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.995,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52700000000000002,"15min":0.46899999999999997},"min":{"1min":0.34200000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.996,"5min":4.96,"15min":4.96},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Apr 5 23:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66200000000000003,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2390000000000001,"5min":4.5590000000000002,"15min":4.5590000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.59899999999999998},"min":{"1min":0.35899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":1.0149999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Apr 5 23:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.68200000000000005},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.107,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72599999999999998,"15min":0.67800000000000005},"min":{"1min":0.5,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.167,"5min":4.6710000000000003,"15min":4.6710000000000003},"last":0.76600000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73499999999999999,"15min":0.67600000000000005},"min":{"1min":0.58099999999999996,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1519999999999999,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.753,"15min":0.67900000000000005},"min":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.278,"5min":4.6289999999999996,"15min":4.6289999999999996},"last":0.73199999999999998}]},{"osd":5,"last update":"Fri Apr 5 23:33:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76100000000000001,"15min":0.70099999999999996},"min":{"1min":0.52800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.095,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.78600000000000003,"15min":0.72799999999999998},"min":{"1min":0.47999999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.96699999999999997,"5min":4.8929999999999998,"15min":4.8929999999999998},"last":0.82799999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:32:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.84399999999999997,"15min":0.872},"min":{"1min":0.497,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.29,"5min":4.8159999999999998,"15min":4.8159999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.747,"15min":0.80100000000000005},"min":{"1min":0.433,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0569999999999999,"5min":5.0979999999999999,"15min":5.0979999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.96999999999999997,"15min":0.998},"min":{"1min":0.50800000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":4.7610000000000001,"5min":4.9980000000000002,"15min":4.9980000000000002},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.044,"5min":0.94899999999999995,"15min":0.96699999999999997},"min":{"1min":0.58999999999999997,"5min":0.502,"15min":0.502},"max":{"1min":4.7759999999999998,"5min":4.7759999999999998,"15min":4.7759999999999998},"last":1.149}]},{"osd":8,"last update":"Fri Apr 5 23:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":4.7569999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.0300000000000002,"5min":5.0300000000000002,"15min":5.0300000000000002},"last":0.72199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.94399999999999995,"15min":0.96599999999999997},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.081,"5min":1.349,"15min":1.349},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.80800000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.95699999999999996,"15min":0.97199999999999998},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.224,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.876,"15min":0.878},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":1.1890000000000001}]},{"osd":11,"last update":"Fri Apr 5 23:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":1.0920000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":909312,"data_stored":439647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.40799999999999997,"15min":0.374},"min":{"1min":0.35899999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.65200000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.39100000000000001},"min":{"1min":0.32500000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.71299999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.39800000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76900000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58199999999999996,"15min":0.55400000000000005},"min":{"1min":0.45900000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.76500000000000001,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.56999999999999995},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76700000000000002,"5min":1.117,"15min":1.117},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71899999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Apr 5 23:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63800000000000001,"15min":0.60899999999999999},"min":{"1min":0.53300000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46800000000000003}]},{"osd":6,"last update":"Fri Apr 5 23:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92900000000000005,"5min":1.137,"15min":1.137},"last":0.505},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.627,"15min":0.59899999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89700000000000002,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.55800000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95299999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86299999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.34799999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.48899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.874,"5min":1.087,"15min":1.087},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.755,"15min":0.747},"min":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.434}]},{"osd":9,"last update":"Fri Apr 5 23:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80900000000000005,"15min":0.81299999999999994},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.071,"5min":1.28,"15min":1.28},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.53800000000000003}]},{"osd":10,"last update":"Fri Apr 5 23:33:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.752,"15min":0.75},"min":{"1min":0.55100000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95299999999999996,"5min":1.181,"15min":1.181},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0249999999999999,"5min":1.371,"15min":1.371},"last":0.54900000000000004}]},{"osd":11,"last update":"Fri Apr 5 23:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.58299999999999996}]}]},{"osd":5,"up_from":35,"seq":150323855410,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1536000,"data_stored":1031433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54100000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80200000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.34200000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71999999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68799999999999994,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.30299999999999999}]},{"osd":2,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.28499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.35799999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.58599999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.78200000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.317},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58399999999999996,"15min":0.57199999999999995},"min":{"1min":0.439,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.79700000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56899999999999995,"15min":0.55700000000000005},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60099999999999998,"15min":0.58899999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83299999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.38300000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:32:57 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.45600000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82199999999999995,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.441},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.48199999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84599999999999997,"5min":1.153,"15min":1.153},"last":0.51800000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.61699999999999999,"15min":0.61199999999999999},"min":{"1min":0.46700000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.48899999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53000000000000003,"15min":0.51500000000000001},"min":{"1min":0.34699999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.371,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70099999999999996,"15min":0.69199999999999995},"min":{"1min":0.59999999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.67100000000000004,"15min":0.65700000000000003},"min":{"1min":0.60199999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.89200000000000002,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.58899999999999997}]},{"osd":10,"last update":"Fri Apr 5 23:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.55400000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.67600000000000005,"15min":0.66500000000000004},"min":{"1min":0.53400000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":199865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T23:33:41.481 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-05T23:33:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.723+0000 7f1e8b000700 1 -- 172.21.15.16:0/3620145582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84101940 msgr2=0x7f1e84105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.723+0000 7f1e8b000700 1 --2- 172.21.15.16:0/3620145582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84101940 0x7f1e84105980 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009990 tx=0x7f1e6c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.723+0000 7f1e8b000700 1 -- 172.21.15.16:0/3620145582 shutdown_connections 2024-04-05T23:33:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.723+0000 7f1e8b000700 1 --2- 172.21.15.16:0/3620145582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e841060b0 0x7f1e8410dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.723+0000 7f1e8b000700 1 --2- 172.21.15.16:0/3620145582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e84100f90 0x7f1e84101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.723+0000 7f1e8b000700 1 --2- 172.21.15.16:0/3620145582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84101940 0x7f1e84105980 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.723+0000 7f1e8b000700 1 -- 172.21.15.16:0/3620145582 >> 172.21.15.16:0/3620145582 conn(0x7f1e840fc830 msgr2=0x7f1e840fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.723+0000 7f1e8b000700 1 -- 172.21.15.16:0/3620145582 shutdown_connections 2024-04-05T23:33:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.723+0000 7f1e8b000700 1 -- 172.21.15.16:0/3620145582 wait complete. 2024-04-05T23:33:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.725+0000 7f1e8b000700 1 Processor -- start 2024-04-05T23:33:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.725+0000 7f1e8b000700 1 -- start start 2024-04-05T23:33:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.725+0000 7f1e8b000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84100f90 0x7f1e84199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.725+0000 7f1e8b000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e841060b0 0x7f1e84199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.725+0000 7f1e8b000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8419dd30 0x7f1e8419e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.725+0000 7f1e8b000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8419e7b0 con 0x7f1e84100f90 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.725+0000 7f1e8b000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8419e920 con 0x7f1e841060b0 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.725+0000 7f1e8b000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8419eae0 con 0x7f1e8419dd30 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e83e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84100f90 0x7f1e84199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e83e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84100f90 0x7f1e84199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48466/0 (socket says 172.21.15.16:48466) 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e83e00700 1 -- 172.21.15.16:0/2795082192 learned_addr learned my addr 172.21.15.16:0/2795082192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e7be00700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e841060b0 0x7f1e84199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e89000700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8419dd30 0x7f1e8419e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e83e00700 1 -- 172.21.15.16:0/2795082192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8419dd30 msgr2=0x7f1e8419e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e83e00700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8419dd30 0x7f1e8419e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e83e00700 1 -- 172.21.15.16:0/2795082192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e841060b0 msgr2=0x7f1e84199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e83e00700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e841060b0 0x7f1e84199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e83e00700 1 -- 172.21.15.16:0/2795082192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c0095f0 con 0x7f1e84100f90 2024-04-05T23:33:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e89000700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8419dd30 0x7f1e8419e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.726+0000 7f1e83e00700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84100f90 0x7f1e84199450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400d980 tx=0x7f1e7400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.727+0000 7f1e81600700 1 -- 172.21.15.16:0/2795082192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74004220 con 0x7f1e84100f90 2024-04-05T23:33:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.727+0000 7f1e81600700 1 -- 172.21.15.16:0/2795082192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e74007640 con 0x7f1e84100f90 2024-04-05T23:33:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.727+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e841a5fe0 con 0x7f1e84100f90 2024-04-05T23:33:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.727+0000 7f1e81600700 1 -- 172.21.15.16:0/2795082192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74005610 con 0x7f1e84100f90 2024-04-05T23:33:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.727+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e841a64c0 con 0x7f1e84100f90 2024-04-05T23:33:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.728+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e84102460 con 0x7f1e84100f90 2024-04-05T23:33:41.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.730+0000 7f1e81600700 1 -- 172.21.15.16:0/2795082192 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f1e74004750 con 0x7f1e84100f90 2024-04-05T23:33:41.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.731+0000 7f1e81600700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1e680a1320 0x7f1e680a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.731+0000 7f1e81600700 1 -- 172.21.15.16:0/2795082192 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f1e740cccf0 con 0x7f1e84100f90 2024-04-05T23:33:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.731+0000 7f1e7be00700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1e680a1320 0x7f1e680a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.732+0000 7f1e7be00700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1e680a1320 0x7f1e680a37e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e8419a9a0 tx=0x7f1e6c005fd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.735+0000 7f1e81600700 1 -- 172.21.15.16:0/2795082192 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1e74095fa0 con 0x7f1e84100f90 2024-04-05T23:33:41.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:41 smithi016 ceph-mon[38248]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:33:41.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:41 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1832099073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:41.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:41 smithi016 ceph-mon[38248]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:33:41.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.945+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 --> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1e841019b0 con 0x7f1e680a1320 2024-04-05T23:33:41.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.956+0000 7f1e81600700 1 -- 172.21.15.16:0/2795082192 <== mgr.15240 v2:172.21.15.16:6800/3895420073 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343358 (secure 0 0 0) 0x7f1e841019b0 con 0x7f1e680a1320 2024-04-05T23:33:41.953 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.960+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1e680a1320 msgr2=0x7f1e680a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.960+0000 7f1e8b000700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1e680a1320 0x7f1e680a37e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e8419a9a0 tx=0x7f1e6c005fd0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.960+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84100f90 msgr2=0x7f1e84199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.960+0000 7f1e8b000700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84100f90 0x7f1e84199450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400d980 tx=0x7f1e7400dc90 comp rx=0 tx=0).stop 2024-04-05T23:33:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.961+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 shutdown_connections 2024-04-05T23:33:41.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.961+0000 7f1e8b000700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1e680a1320 0x7f1e680a37e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.961+0000 7f1e8b000700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e84100f90 0x7f1e84199450 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.961+0000 7f1e8b000700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8419dd30 0x7f1e8419e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.961+0000 7f1e8b000700 1 --2- 172.21.15.16:0/2795082192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e841060b0 0x7f1e84199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.961+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 >> 172.21.15.16:0/2795082192 conn(0x7f1e840fc830 msgr2=0x7f1e840feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.961+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 shutdown_connections 2024-04-05T23:33:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:41.961+0000 7f1e8b000700 1 -- 172.21.15.16:0/2795082192 wait complete. 2024-04-05T23:33:41.957 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-05T23:33:41.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:41 smithi027 ceph-mon[78597]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:33:41.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:41 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1832099073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:41.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:41 smithi027 ceph-mon[78597]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:33:41.973 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-05T23:33:40.613366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190190,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1133,"num_read_kb":731,"num_write":543,"num_write_kb":4398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":932,"ondisk_log_size":932,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370856,"kb_used_data":13800,"kb_used_omap":18,"kb_used_meta":356845,"kb_avail":1124570968,"statfs":{"total":1151940427776,"available":1151560671232,"internally_reserved":0,"allocated":14131200,"data_stored":8299437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":365409605},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.976347"},"pg_stats":[{"pgid":"3.3f","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673349+0000","last_change":"2024-04-05T23:32:04.536879+0000","last_active":"2024-04-05T23:33:34.673349+0000","last_peered":"2024-04-05T23:33:34.673349+0000","last_clean":"2024-04-05T23:33:34.673349+0000","last_became_active":"2024-04-05T23:32:04.536183+0000","last_became_peered":"2024-04-05T23:32:04.536183+0000","last_unstale":"2024-04-05T23:33:34.673349+0000","last_undegraded":"2024-04-05T23:33:34.673349+0000","last_fullsized":"2024-04-05T23:33:34.673349+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:13:22.589464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"82'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670374+0000","last_change":"2024-04-05T23:32:04.540791+0000","last_active":"2024-04-05T23:33:34.670374+0000","last_peered":"2024-04-05T23:33:34.670374+0000","last_clean":"2024-04-05T23:33:34.670374+0000","last_became_active":"2024-04-05T23:32:04.540455+0000","last_became_peered":"2024-04-05T23:32:04.540455+0000","last_unstale":"2024-04-05T23:33:34.670374+0000","last_undegraded":"2024-04-05T23:33:34.670374+0000","last_fullsized":"2024-04-05T23:33:34.670374+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:06:19.090672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.671697+0000","last_change":"2024-04-05T23:32:04.543428+0000","last_active":"2024-04-05T23:33:34.671697+0000","last_peered":"2024-04-05T23:33:34.671697+0000","last_clean":"2024-04-05T23:33:34.671697+0000","last_became_active":"2024-04-05T23:32:04.543071+0000","last_became_peered":"2024-04-05T23:32:04.543071+0000","last_unstale":"2024-04-05T23:33:34.671697+0000","last_undegraded":"2024-04-05T23:33:34.671697+0000","last_fullsized":"2024-04-05T23:33:34.671697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:35:44.911749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651922+0000","last_change":"2024-04-05T23:32:04.531444+0000","last_active":"2024-04-05T23:33:34.651922+0000","last_peered":"2024-04-05T23:33:34.651922+0000","last_clean":"2024-04-05T23:33:34.651922+0000","last_became_active":"2024-04-05T23:32:04.529556+0000","last_became_peered":"2024-04-05T23:32:04.529556+0000","last_unstale":"2024-04-05T23:33:34.651922+0000","last_undegraded":"2024-04-05T23:33:34.651922+0000","last_fullsized":"2024-04-05T23:33:34.651922+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:48.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"82'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669652+0000","last_change":"2024-04-05T23:32:04.529565+0000","last_active":"2024-04-05T23:33:34.669652+0000","last_peered":"2024-04-05T23:33:34.669652+0000","last_clean":"2024-04-05T23:33:34.669652+0000","last_became_active":"2024-04-05T23:32:04.529226+0000","last_became_peered":"2024-04-05T23:32:04.529226+0000","last_unstale":"2024-04-05T23:33:34.669652+0000","last_undegraded":"2024-04-05T23:33:34.669652+0000","last_fullsized":"2024-04-05T23:33:34.669652+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:44:25.208996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.655487+0000","last_change":"2024-04-05T23:32:04.546260+0000","last_active":"2024-04-05T23:33:34.655487+0000","last_peered":"2024-04-05T23:33:34.655487+0000","last_clean":"2024-04-05T23:33:34.655487+0000","last_became_active":"2024-04-05T23:32:04.545851+0000","last_became_peered":"2024-04-05T23:32:04.545851+0000","last_unstale":"2024-04-05T23:33:34.655487+0000","last_undegraded":"2024-04-05T23:33:34.655487+0000","last_fullsized":"2024-04-05T23:33:34.655487+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:59:16.735635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.665495+0000","last_change":"2024-04-05T23:32:04.536348+0000","last_active":"2024-04-05T23:33:34.665495+0000","last_peered":"2024-04-05T23:33:34.665495+0000","last_clean":"2024-04-05T23:33:34.665495+0000","last_became_active":"2024-04-05T23:32:04.536015+0000","last_became_peered":"2024-04-05T23:32:04.536015+0000","last_unstale":"2024-04-05T23:33:34.665495+0000","last_undegraded":"2024-04-05T23:33:34.665495+0000","last_fullsized":"2024-04-05T23:33:34.665495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:02:24.897957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666748+0000","last_change":"2024-04-05T23:32:04.532528+0000","last_active":"2024-04-05T23:33:34.666748+0000","last_peered":"2024-04-05T23:33:34.666748+0000","last_clean":"2024-04-05T23:33:34.666748+0000","last_became_active":"2024-04-05T23:32:04.532104+0000","last_became_peered":"2024-04-05T23:32:04.532104+0000","last_unstale":"2024-04-05T23:33:34.666748+0000","last_undegraded":"2024-04-05T23:33:34.666748+0000","last_fullsized":"2024-04-05T23:33:34.666748+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:06.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"82'11","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651120+0000","last_change":"2024-04-05T23:32:04.531223+0000","last_active":"2024-04-05T23:33:34.651120+0000","last_peered":"2024-04-05T23:33:34.651120+0000","last_clean":"2024-04-05T23:33:34.651120+0000","last_became_active":"2024-04-05T23:32:04.530863+0000","last_became_peered":"2024-04-05T23:32:04.530863+0000","last_unstale":"2024-04-05T23:33:34.651120+0000","last_undegraded":"2024-04-05T23:33:34.651120+0000","last_fullsized":"2024-04-05T23:33:34.651120+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:03:35.912984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"82'16","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.653658+0000","last_change":"2024-04-05T23:32:04.541481+0000","last_active":"2024-04-05T23:33:34.653658+0000","last_peered":"2024-04-05T23:33:34.653658+0000","last_clean":"2024-04-05T23:33:34.653658+0000","last_became_active":"2024-04-05T23:32:04.541133+0000","last_became_peered":"2024-04-05T23:32:04.541133+0000","last_unstale":"2024-04-05T23:33:34.653658+0000","last_undegraded":"2024-04-05T23:33:34.653658+0000","last_fullsized":"2024-04-05T23:33:34.653658+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:01:22.825587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.660781+0000","last_change":"2024-04-05T23:32:04.542390+0000","last_active":"2024-04-05T23:33:34.660781+0000","last_peered":"2024-04-05T23:33:34.660781+0000","last_clean":"2024-04-05T23:33:34.660781+0000","last_became_active":"2024-04-05T23:32:04.542026+0000","last_became_peered":"2024-04-05T23:32:04.542026+0000","last_unstale":"2024-04-05T23:33:34.660781+0000","last_undegraded":"2024-04-05T23:33:34.660781+0000","last_fullsized":"2024-04-05T23:33:34.660781+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:15:42.243807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651490+0000","last_change":"2024-04-05T23:32:04.540096+0000","last_active":"2024-04-05T23:33:34.651490+0000","last_peered":"2024-04-05T23:33:34.651490+0000","last_clean":"2024-04-05T23:33:34.651490+0000","last_became_active":"2024-04-05T23:32:04.539764+0000","last_became_peered":"2024-04-05T23:32:04.539764+0000","last_unstale":"2024-04-05T23:33:34.651490+0000","last_undegraded":"2024-04-05T23:33:34.651490+0000","last_fullsized":"2024-04-05T23:33:34.651490+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:57:36.442393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.665139+0000","last_change":"2024-04-05T23:32:04.540839+0000","last_active":"2024-04-05T23:33:34.665139+0000","last_peered":"2024-04-05T23:33:34.665139+0000","last_clean":"2024-04-05T23:33:34.665139+0000","last_became_active":"2024-04-05T23:32:04.540511+0000","last_became_peered":"2024-04-05T23:32:04.540511+0000","last_unstale":"2024-04-05T23:33:34.665139+0000","last_undegraded":"2024-04-05T23:33:34.665139+0000","last_fullsized":"2024-04-05T23:33:34.665139+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:58:24.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.667960+0000","last_change":"2024-04-05T23:32:04.539222+0000","last_active":"2024-04-05T23:33:34.667960+0000","last_peered":"2024-04-05T23:33:34.667960+0000","last_clean":"2024-04-05T23:33:34.667960+0000","last_became_active":"2024-04-05T23:32:04.538906+0000","last_became_peered":"2024-04-05T23:32:04.538906+0000","last_unstale":"2024-04-05T23:33:34.667960+0000","last_undegraded":"2024-04-05T23:33:34.667960+0000","last_fullsized":"2024-04-05T23:33:34.667960+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:05:24.429153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666925+0000","last_change":"2024-04-05T23:32:04.544155+0000","last_active":"2024-04-05T23:33:34.666925+0000","last_peered":"2024-04-05T23:33:34.666925+0000","last_clean":"2024-04-05T23:33:34.666925+0000","last_became_active":"2024-04-05T23:32:04.543852+0000","last_became_peered":"2024-04-05T23:32:04.543852+0000","last_unstale":"2024-04-05T23:33:34.666925+0000","last_undegraded":"2024-04-05T23:33:34.666925+0000","last_fullsized":"2024-04-05T23:33:34.666925+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:18:34.353542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.672831+0000","last_change":"2024-04-05T23:32:04.533772+0000","last_active":"2024-04-05T23:33:34.672831+0000","last_peered":"2024-04-05T23:33:34.672831+0000","last_clean":"2024-04-05T23:33:34.672831+0000","last_became_active":"2024-04-05T23:32:04.533416+0000","last_became_peered":"2024-04-05T23:32:04.533416+0000","last_unstale":"2024-04-05T23:33:34.672831+0000","last_undegraded":"2024-04-05T23:33:34.672831+0000","last_fullsized":"2024-04-05T23:33:34.672831+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:36:55.072349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673455+0000","last_change":"2024-04-05T23:32:04.551902+0000","last_active":"2024-04-05T23:33:34.673455+0000","last_peered":"2024-04-05T23:33:34.673455+0000","last_clean":"2024-04-05T23:33:34.673455+0000","last_became_active":"2024-04-05T23:32:04.551595+0000","last_became_peered":"2024-04-05T23:32:04.551595+0000","last_unstale":"2024-04-05T23:33:34.673455+0000","last_undegraded":"2024-04-05T23:33:34.673455+0000","last_fullsized":"2024-04-05T23:33:34.673455+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:14:46.648789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651142+0000","last_change":"2024-04-05T23:32:04.542967+0000","last_active":"2024-04-05T23:33:34.651142+0000","last_peered":"2024-04-05T23:33:34.651142+0000","last_clean":"2024-04-05T23:33:34.651142+0000","last_became_active":"2024-04-05T23:32:04.541767+0000","last_became_peered":"2024-04-05T23:32:04.541767+0000","last_unstale":"2024-04-05T23:33:34.651142+0000","last_undegraded":"2024-04-05T23:33:34.651142+0000","last_fullsized":"2024-04-05T23:33:34.651142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:27:46.150249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666772+0000","last_change":"2024-04-05T23:32:04.549512+0000","last_active":"2024-04-05T23:33:34.666772+0000","last_peered":"2024-04-05T23:33:34.666772+0000","last_clean":"2024-04-05T23:33:34.666772+0000","last_became_active":"2024-04-05T23:32:04.549184+0000","last_became_peered":"2024-04-05T23:32:04.549184+0000","last_unstale":"2024-04-05T23:33:34.666772+0000","last_undegraded":"2024-04-05T23:33:34.666772+0000","last_fullsized":"2024-04-05T23:33:34.666772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:48:24.186408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'14","reported_seq":115,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.941928+0000","last_change":"2024-04-05T23:32:04.005188+0000","last_active":"2024-04-05T23:33:34.941928+0000","last_peered":"2024-04-05T23:33:34.941928+0000","last_clean":"2024-04-05T23:33:34.941928+0000","last_became_active":"2024-04-05T23:32:04.004818+0000","last_became_peered":"2024-04-05T23:32:04.004818+0000","last_unstale":"2024-04-05T23:33:34.941928+0000","last_undegraded":"2024-04-05T23:33:34.941928+0000","last_fullsized":"2024-04-05T23:33:34.941928+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:49:41.146761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661564+0000","last_change":"2024-04-05T23:32:04.544881+0000","last_active":"2024-04-05T23:33:34.661564+0000","last_peered":"2024-04-05T23:33:34.661564+0000","last_clean":"2024-04-05T23:33:34.661564+0000","last_became_active":"2024-04-05T23:32:04.542149+0000","last_became_peered":"2024-04-05T23:32:04.542149+0000","last_unstale":"2024-04-05T23:33:34.661564+0000","last_undegraded":"2024-04-05T23:33:34.661564+0000","last_fullsized":"2024-04-05T23:33:34.661564+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:28:51.713053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.374976+0000","last_change":"2024-04-05T23:32:04.011450+0000","last_active":"2024-04-05T23:32:38.374976+0000","last_peered":"2024-04-05T23:32:38.374976+0000","last_clean":"2024-04-05T23:32:38.374976+0000","last_became_active":"2024-04-05T23:32:04.009607+0000","last_became_peered":"2024-04-05T23:32:04.009607+0000","last_unstale":"2024-04-05T23:32:38.374976+0000","last_undegraded":"2024-04-05T23:32:38.374976+0000","last_fullsized":"2024-04-05T23:32:38.374976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:16:45.483108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673025+0000","last_change":"2024-04-05T23:32:04.550370+0000","last_active":"2024-04-05T23:33:34.673025+0000","last_peered":"2024-04-05T23:33:34.673025+0000","last_clean":"2024-04-05T23:33:34.673025+0000","last_became_active":"2024-04-05T23:32:04.549980+0000","last_became_peered":"2024-04-05T23:32:04.549980+0000","last_unstale":"2024-04-05T23:33:34.673025+0000","last_undegraded":"2024-04-05T23:33:34.673025+0000","last_fullsized":"2024-04-05T23:33:34.673025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:44:30.704366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379630+0000","last_change":"2024-04-05T23:32:04.004169+0000","last_active":"2024-04-05T23:32:38.379630+0000","last_peered":"2024-04-05T23:32:38.379630+0000","last_clean":"2024-04-05T23:32:38.379630+0000","last_became_active":"2024-04-05T23:32:04.003646+0000","last_became_peered":"2024-04-05T23:32:04.003646+0000","last_unstale":"2024-04-05T23:32:38.379630+0000","last_undegraded":"2024-04-05T23:32:38.379630+0000","last_fullsized":"2024-04-05T23:32:38.379630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:57:02.781815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.662342+0000","last_change":"2024-04-05T23:32:04.542066+0000","last_active":"2024-04-05T23:33:34.662342+0000","last_peered":"2024-04-05T23:33:34.662342+0000","last_clean":"2024-04-05T23:33:34.662342+0000","last_became_active":"2024-04-05T23:32:04.540735+0000","last_became_peered":"2024-04-05T23:32:04.540735+0000","last_unstale":"2024-04-05T23:33:34.662342+0000","last_undegraded":"2024-04-05T23:33:34.662342+0000","last_fullsized":"2024-04-05T23:33:34.662342+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:02:03.595888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:41.225108+0000","last_change":"2024-04-05T23:32:38.396641+0000","last_active":"2024-04-05T23:32:41.225108+0000","last_peered":"2024-04-05T23:32:41.225108+0000","last_clean":"2024-04-05T23:32:41.225108+0000","last_became_active":"2024-04-05T23:32:38.379708+0000","last_became_peered":"2024-04-05T23:32:38.379708+0000","last_unstale":"2024-04-05T23:32:41.225108+0000","last_undegraded":"2024-04-05T23:32:41.225108+0000","last_fullsized":"2024-04-05T23:32:41.225108+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:00.370725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"82'15","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652724+0000","last_change":"2024-04-05T23:32:04.545780+0000","last_active":"2024-04-05T23:33:34.652724+0000","last_peered":"2024-04-05T23:33:34.652724+0000","last_clean":"2024-04-05T23:33:34.652724+0000","last_became_active":"2024-04-05T23:32:04.545279+0000","last_became_peered":"2024-04-05T23:32:04.545279+0000","last_unstale":"2024-04-05T23:33:34.652724+0000","last_undegraded":"2024-04-05T23:33:34.652724+0000","last_fullsized":"2024-04-05T23:33:34.652724+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:10.577280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":111,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.537192+0000","last_change":"2024-04-05T23:32:04.004189+0000","last_active":"2024-04-05T23:33:35.537192+0000","last_peered":"2024-04-05T23:33:35.537192+0000","last_clean":"2024-04-05T23:33:35.537192+0000","last_became_active":"2024-04-05T23:32:04.003841+0000","last_became_peered":"2024-04-05T23:32:04.003841+0000","last_unstale":"2024-04-05T23:33:35.537192+0000","last_undegraded":"2024-04-05T23:33:35.537192+0000","last_fullsized":"2024-04-05T23:33:35.537192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:33:26.898657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.649038+0000","last_change":"2024-04-05T23:32:04.529538+0000","last_active":"2024-04-05T23:33:34.649038+0000","last_peered":"2024-04-05T23:33:34.649038+0000","last_clean":"2024-04-05T23:33:34.649038+0000","last_became_active":"2024-04-05T23:32:04.529231+0000","last_became_peered":"2024-04-05T23:32:04.529231+0000","last_unstale":"2024-04-05T23:33:34.649038+0000","last_undegraded":"2024-04-05T23:33:34.649038+0000","last_fullsized":"2024-04-05T23:33:34.649038+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:49:15.608966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377155+0000","last_change":"2024-04-05T23:32:04.002763+0000","last_active":"2024-04-05T23:32:38.377155+0000","last_peered":"2024-04-05T23:32:38.377155+0000","last_clean":"2024-04-05T23:32:38.377155+0000","last_became_active":"2024-04-05T23:32:04.002410+0000","last_became_peered":"2024-04-05T23:32:04.002410+0000","last_unstale":"2024-04-05T23:32:38.377155+0000","last_undegraded":"2024-04-05T23:32:38.377155+0000","last_fullsized":"2024-04-05T23:32:38.377155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:15.193813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.667881+0000","last_change":"2024-04-05T23:32:38.461447+0000","last_active":"2024-04-05T23:33:34.667881+0000","last_peered":"2024-04-05T23:33:34.667881+0000","last_clean":"2024-04-05T23:33:34.667881+0000","last_became_active":"2024-04-05T23:32:38.460968+0000","last_became_peered":"2024-04-05T23:32:38.460968+0000","last_unstale":"2024-04-05T23:33:34.667881+0000","last_undegraded":"2024-04-05T23:33:34.667881+0000","last_fullsized":"2024-04-05T23:33:34.667881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:42:21.093881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377383+0000","last_change":"2024-04-05T23:32:04.001937+0000","last_active":"2024-04-05T23:32:38.377383+0000","last_peered":"2024-04-05T23:32:38.377383+0000","last_clean":"2024-04-05T23:32:38.377383+0000","last_became_active":"2024-04-05T23:32:04.001508+0000","last_became_peered":"2024-04-05T23:32:04.001508+0000","last_unstale":"2024-04-05T23:32:38.377383+0000","last_undegraded":"2024-04-05T23:32:38.377383+0000","last_fullsized":"2024-04-05T23:32:38.377383+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:09:20.682084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.668565+0000","last_change":"2024-04-05T23:32:04.531576+0000","last_active":"2024-04-05T23:33:34.668565+0000","last_peered":"2024-04-05T23:33:34.668565+0000","last_clean":"2024-04-05T23:33:34.668565+0000","last_became_active":"2024-04-05T23:32:04.531234+0000","last_became_peered":"2024-04-05T23:32:04.531234+0000","last_unstale":"2024-04-05T23:33:34.668565+0000","last_undegraded":"2024-04-05T23:33:34.668565+0000","last_fullsized":"2024-04-05T23:33:34.668565+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:21:57.769861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.376961+0000","last_change":"2024-04-05T23:32:04.001570+0000","last_active":"2024-04-05T23:32:38.376961+0000","last_peered":"2024-04-05T23:32:38.376961+0000","last_clean":"2024-04-05T23:32:38.376961+0000","last_became_active":"2024-04-05T23:32:04.001224+0000","last_became_peered":"2024-04-05T23:32:04.001224+0000","last_unstale":"2024-04-05T23:32:38.376961+0000","last_undegraded":"2024-04-05T23:32:38.376961+0000","last_fullsized":"2024-04-05T23:32:38.376961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:44:41.070205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"82'11","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.660070+0000","last_change":"2024-04-05T23:32:04.542029+0000","last_active":"2024-04-05T23:33:34.660070+0000","last_peered":"2024-04-05T23:33:34.660070+0000","last_clean":"2024-04-05T23:33:34.660070+0000","last_became_active":"2024-04-05T23:32:04.541684+0000","last_became_peered":"2024-04-05T23:32:04.541684+0000","last_unstale":"2024-04-05T23:33:34.660070+0000","last_undegraded":"2024-04-05T23:33:34.660070+0000","last_fullsized":"2024-04-05T23:33:34.660070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:07:06.456917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'6","reported_seq":193,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.536712+0000","last_change":"2024-04-05T23:32:04.004986+0000","last_active":"2024-04-05T23:33:35.536712+0000","last_peered":"2024-04-05T23:33:35.536712+0000","last_clean":"2024-04-05T23:33:35.536712+0000","last_became_active":"2024-04-05T23:32:04.004606+0000","last_became_peered":"2024-04-05T23:32:04.004606+0000","last_unstale":"2024-04-05T23:33:35.536712+0000","last_undegraded":"2024-04-05T23:33:35.536712+0000","last_fullsized":"2024-04-05T23:33:35.536712+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:23:41.060500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.654797+0000","last_change":"2024-04-05T23:32:04.541334+0000","last_active":"2024-04-05T23:33:34.654797+0000","last_peered":"2024-04-05T23:33:34.654797+0000","last_clean":"2024-04-05T23:33:34.654797+0000","last_became_active":"2024-04-05T23:32:04.540967+0000","last_became_peered":"2024-04-05T23:32:04.540967+0000","last_unstale":"2024-04-05T23:33:34.654797+0000","last_undegraded":"2024-04-05T23:33:34.654797+0000","last_fullsized":"2024-04-05T23:33:34.654797+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:16:33.045172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.466958+0000","last_change":"2024-04-05T23:32:38.466417+0000","last_active":"2024-04-05T23:32:38.466958+0000","last_peered":"2024-04-05T23:32:38.466958+0000","last_clean":"2024-04-05T23:32:38.466958+0000","last_became_active":"2024-04-05T23:32:38.462293+0000","last_became_peered":"2024-04-05T23:32:38.462293+0000","last_unstale":"2024-04-05T23:32:38.466958+0000","last_undegraded":"2024-04-05T23:32:38.466958+0000","last_fullsized":"2024-04-05T23:32:38.466958+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:31.118636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"82'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669698+0000","last_change":"2024-04-05T23:32:38.385818+0000","last_active":"2024-04-05T23:33:34.669698+0000","last_peered":"2024-04-05T23:33:34.669698+0000","last_clean":"2024-04-05T23:33:34.669698+0000","last_became_active":"2024-04-05T23:32:38.385423+0000","last_became_peered":"2024-04-05T23:32:38.385423+0000","last_unstale":"2024-04-05T23:33:34.669698+0000","last_undegraded":"2024-04-05T23:33:34.669698+0000","last_fullsized":"2024-04-05T23:33:34.669698+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:07:11.374783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377265+0000","last_change":"2024-04-05T23:32:04.000697+0000","last_active":"2024-04-05T23:32:38.377265+0000","last_peered":"2024-04-05T23:32:38.377265+0000","last_clean":"2024-04-05T23:32:38.377265+0000","last_became_active":"2024-04-05T23:32:04.000377+0000","last_became_peered":"2024-04-05T23:32:04.000377+0000","last_unstale":"2024-04-05T23:32:38.377265+0000","last_undegraded":"2024-04-05T23:32:38.377265+0000","last_fullsized":"2024-04-05T23:32:38.377265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:55:11.227637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.662559+0000","last_change":"2024-04-05T23:32:38.383367+0000","last_active":"2024-04-05T23:33:34.662559+0000","last_peered":"2024-04-05T23:33:34.662559+0000","last_clean":"2024-04-05T23:33:34.662559+0000","last_became_active":"2024-04-05T23:32:38.383002+0000","last_became_peered":"2024-04-05T23:32:38.383002+0000","last_unstale":"2024-04-05T23:33:34.662559+0000","last_undegraded":"2024-04-05T23:33:34.662559+0000","last_fullsized":"2024-04-05T23:33:34.662559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:30:59.947387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'36","reported_seq":234,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.547756+0000","last_change":"2024-04-05T23:32:04.007322+0000","last_active":"2024-04-05T23:33:35.547756+0000","last_peered":"2024-04-05T23:33:35.547756+0000","last_clean":"2024-04-05T23:33:35.547756+0000","last_became_active":"2024-04-05T23:32:04.006971+0000","last_became_peered":"2024-04-05T23:32:04.006971+0000","last_unstale":"2024-04-05T23:33:35.547756+0000","last_undegraded":"2024-04-05T23:33:35.547756+0000","last_fullsized":"2024-04-05T23:33:35.547756+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:04:38.413482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65485,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":183,"num_read_kb":155,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661408+0000","last_change":"2024-04-05T23:32:04.535342+0000","last_active":"2024-04-05T23:33:34.661408+0000","last_peered":"2024-04-05T23:33:34.661408+0000","last_clean":"2024-04-05T23:33:34.661408+0000","last_became_active":"2024-04-05T23:32:04.535053+0000","last_became_peered":"2024-04-05T23:32:04.535053+0000","last_unstale":"2024-04-05T23:33:34.661408+0000","last_undegraded":"2024-04-05T23:33:34.661408+0000","last_fullsized":"2024-04-05T23:33:34.661408+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:19:20.683836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:41.225091+0000","last_change":"2024-04-05T23:32:38.391418+0000","last_active":"2024-04-05T23:32:41.225091+0000","last_peered":"2024-04-05T23:32:41.225091+0000","last_clean":"2024-04-05T23:32:41.225091+0000","last_became_active":"2024-04-05T23:32:38.379643+0000","last_became_peered":"2024-04-05T23:32:38.379643+0000","last_unstale":"2024-04-05T23:32:41.225091+0000","last_undegraded":"2024-04-05T23:32:41.225091+0000","last_fullsized":"2024-04-05T23:32:41.225091+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:37:12.343153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.668971+0000","last_change":"2024-04-05T23:32:04.536455+0000","last_active":"2024-04-05T23:33:34.668971+0000","last_peered":"2024-04-05T23:33:34.668971+0000","last_clean":"2024-04-05T23:33:34.668971+0000","last_became_active":"2024-04-05T23:32:04.533994+0000","last_became_peered":"2024-04-05T23:32:04.533994+0000","last_unstale":"2024-04-05T23:33:34.668971+0000","last_undegraded":"2024-04-05T23:33:34.668971+0000","last_fullsized":"2024-04-05T23:33:34.668971+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:38:18.952823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.376958+0000","last_change":"2024-04-05T23:32:04.000708+0000","last_active":"2024-04-05T23:32:38.376958+0000","last_peered":"2024-04-05T23:32:38.376958+0000","last_clean":"2024-04-05T23:32:38.376958+0000","last_became_active":"2024-04-05T23:32:04.000354+0000","last_became_peered":"2024-04-05T23:32:04.000354+0000","last_unstale":"2024-04-05T23:32:38.376958+0000","last_undegraded":"2024-04-05T23:32:38.376958+0000","last_fullsized":"2024-04-05T23:32:38.376958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:05:34.291607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"82'15","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666642+0000","last_change":"2024-04-05T23:32:04.532146+0000","last_active":"2024-04-05T23:33:34.666642+0000","last_peered":"2024-04-05T23:33:34.666642+0000","last_clean":"2024-04-05T23:33:34.666642+0000","last_became_active":"2024-04-05T23:32:04.531735+0000","last_became_peered":"2024-04-05T23:32:04.531735+0000","last_unstale":"2024-04-05T23:33:34.666642+0000","last_undegraded":"2024-04-05T23:33:34.666642+0000","last_fullsized":"2024-04-05T23:33:34.666642+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:52:44.898871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'8","reported_seq":114,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.540783+0000","last_change":"2024-04-05T23:32:04.000144+0000","last_active":"2024-04-05T23:33:35.540783+0000","last_peered":"2024-04-05T23:33:35.540783+0000","last_clean":"2024-04-05T23:33:35.540783+0000","last_became_active":"2024-04-05T23:32:03.999712+0000","last_became_peered":"2024-04-05T23:32:03.999712+0000","last_unstale":"2024-04-05T23:33:35.540783+0000","last_undegraded":"2024-04-05T23:33:35.540783+0000","last_fullsized":"2024-04-05T23:33:35.540783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:43.775018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1632,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":92,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.658416+0000","last_change":"2024-04-05T23:32:04.537623+0000","last_active":"2024-04-05T23:33:34.658416+0000","last_peered":"2024-04-05T23:33:34.658416+0000","last_clean":"2024-04-05T23:33:34.658416+0000","last_became_active":"2024-04-05T23:32:04.537399+0000","last_became_peered":"2024-04-05T23:32:04.537399+0000","last_unstale":"2024-04-05T23:33:34.658416+0000","last_undegraded":"2024-04-05T23:33:34.658416+0000","last_fullsized":"2024-04-05T23:33:34.658416+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:01:01.643133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378658+0000","last_change":"2024-04-05T23:32:04.006806+0000","last_active":"2024-04-05T23:32:38.378658+0000","last_peered":"2024-04-05T23:32:38.378658+0000","last_clean":"2024-04-05T23:32:38.378658+0000","last_became_active":"2024-04-05T23:32:04.006397+0000","last_became_peered":"2024-04-05T23:32:04.006397+0000","last_unstale":"2024-04-05T23:32:38.378658+0000","last_undegraded":"2024-04-05T23:32:38.378658+0000","last_fullsized":"2024-04-05T23:32:38.378658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:42:01.758863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.653257+0000","last_change":"2024-04-05T23:32:04.545952+0000","last_active":"2024-04-05T23:33:34.653257+0000","last_peered":"2024-04-05T23:33:34.653257+0000","last_clean":"2024-04-05T23:33:34.653257+0000","last_became_active":"2024-04-05T23:32:04.545528+0000","last_became_peered":"2024-04-05T23:32:04.545528+0000","last_unstale":"2024-04-05T23:33:34.653257+0000","last_undegraded":"2024-04-05T23:33:34.653257+0000","last_fullsized":"2024-04-05T23:33:34.653257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:33:06.074173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:33:32.204346+0000","last_change":"2024-04-05T23:32:03.999841+0000","last_active":"2024-04-05T23:33:32.204346+0000","last_peered":"2024-04-05T23:33:32.204346+0000","last_clean":"2024-04-05T23:33:32.204346+0000","last_became_active":"2024-04-05T23:32:03.999418+0000","last_became_peered":"2024-04-05T23:32:03.999418+0000","last_unstale":"2024-04-05T23:33:32.204346+0000","last_undegraded":"2024-04-05T23:33:32.204346+0000","last_fullsized":"2024-04-05T23:33:32.204346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:00:41.452167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.650643+0000","last_change":"2024-04-05T23:32:04.530012+0000","last_active":"2024-04-05T23:33:34.650643+0000","last_peered":"2024-04-05T23:33:34.650643+0000","last_clean":"2024-04-05T23:33:34.650643+0000","last_became_active":"2024-04-05T23:32:04.529628+0000","last_became_peered":"2024-04-05T23:32:04.529628+0000","last_unstale":"2024-04-05T23:33:34.650643+0000","last_undegraded":"2024-04-05T23:33:34.650643+0000","last_fullsized":"2024-04-05T23:33:34.650643+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:40:22.036170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379485+0000","last_change":"2024-04-05T23:32:04.006157+0000","last_active":"2024-04-05T23:32:38.379485+0000","last_peered":"2024-04-05T23:32:38.379485+0000","last_clean":"2024-04-05T23:32:38.379485+0000","last_became_active":"2024-04-05T23:32:04.005853+0000","last_became_peered":"2024-04-05T23:32:04.005853+0000","last_unstale":"2024-04-05T23:32:38.379485+0000","last_undegraded":"2024-04-05T23:32:38.379485+0000","last_fullsized":"2024-04-05T23:32:38.379485+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:48:39.849306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.658363+0000","last_change":"2024-04-05T23:32:04.547367+0000","last_active":"2024-04-05T23:33:34.658363+0000","last_peered":"2024-04-05T23:33:34.658363+0000","last_clean":"2024-04-05T23:33:34.658363+0000","last_became_active":"2024-04-05T23:32:04.547040+0000","last_became_peered":"2024-04-05T23:32:04.547040+0000","last_unstale":"2024-04-05T23:33:34.658363+0000","last_undegraded":"2024-04-05T23:33:34.658363+0000","last_fullsized":"2024-04-05T23:33:34.658363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:41:41.998984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'7","reported_seq":109,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.559448+0000","last_change":"2024-04-05T23:32:04.009851+0000","last_active":"2024-04-05T23:33:35.559448+0000","last_peered":"2024-04-05T23:33:35.559448+0000","last_clean":"2024-04-05T23:33:35.559448+0000","last_became_active":"2024-04-05T23:32:04.009460+0000","last_became_peered":"2024-04-05T23:32:04.009460+0000","last_unstale":"2024-04-05T23:33:35.559448+0000","last_undegraded":"2024-04-05T23:33:35.559448+0000","last_fullsized":"2024-04-05T23:33:35.559448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:39:41.000439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1326,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652783+0000","last_change":"2024-04-05T23:32:04.549621+0000","last_active":"2024-04-05T23:33:34.652783+0000","last_peered":"2024-04-05T23:33:34.652783+0000","last_clean":"2024-04-05T23:33:34.652783+0000","last_became_active":"2024-04-05T23:32:04.549325+0000","last_became_peered":"2024-04-05T23:32:04.549325+0000","last_unstale":"2024-04-05T23:33:34.652783+0000","last_undegraded":"2024-04-05T23:33:34.652783+0000","last_fullsized":"2024-04-05T23:33:34.652783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:11:07.101463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652570+0000","last_change":"2024-04-05T23:32:04.546768+0000","last_active":"2024-04-05T23:33:34.652570+0000","last_peered":"2024-04-05T23:33:34.652570+0000","last_clean":"2024-04-05T23:33:34.652570+0000","last_became_active":"2024-04-05T23:32:04.546471+0000","last_became_peered":"2024-04-05T23:32:04.546471+0000","last_unstale":"2024-04-05T23:33:34.652570+0000","last_undegraded":"2024-04-05T23:33:34.652570+0000","last_fullsized":"2024-04-05T23:33:34.652570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:48:22.728976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":109,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.621543+0000","last_change":"2024-04-05T23:32:04.007093+0000","last_active":"2024-04-05T23:33:34.621543+0000","last_peered":"2024-04-05T23:33:34.621543+0000","last_clean":"2024-04-05T23:33:34.621543+0000","last_became_active":"2024-04-05T23:32:04.006836+0000","last_became_peered":"2024-04-05T23:32:04.006836+0000","last_unstale":"2024-04-05T23:33:34.621543+0000","last_undegraded":"2024-04-05T23:33:34.621543+0000","last_fullsized":"2024-04-05T23:33:34.621543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:20:42.489368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669973+0000","last_change":"2024-04-05T23:32:04.538204+0000","last_active":"2024-04-05T23:33:34.669973+0000","last_peered":"2024-04-05T23:33:34.669973+0000","last_clean":"2024-04-05T23:33:34.669973+0000","last_became_active":"2024-04-05T23:32:04.535254+0000","last_became_peered":"2024-04-05T23:32:04.535254+0000","last_unstale":"2024-04-05T23:33:34.669973+0000","last_undegraded":"2024-04-05T23:33:34.669973+0000","last_fullsized":"2024-04-05T23:33:34.669973+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:06:01.059339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.460974+0000","last_change":"2024-04-05T23:32:03.997704+0000","last_active":"2024-04-05T23:32:38.460974+0000","last_peered":"2024-04-05T23:32:38.460974+0000","last_clean":"2024-04-05T23:32:38.460974+0000","last_became_active":"2024-04-05T23:32:03.997275+0000","last_became_peered":"2024-04-05T23:32:03.997275+0000","last_unstale":"2024-04-05T23:32:38.460974+0000","last_undegraded":"2024-04-05T23:32:38.460974+0000","last_fullsized":"2024-04-05T23:32:38.460974+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:58.355104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":337,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378621+0000","last_change":"2024-04-05T23:31:10.395041+0000","last_active":"2024-04-05T23:32:38.378621+0000","last_peered":"2024-04-05T23:32:38.378621+0000","last_clean":"2024-04-05T23:32:38.378621+0000","last_became_active":"2024-04-05T23:31:10.394479+0000","last_became_peered":"2024-04-05T23:31:10.394479+0000","last_unstale":"2024-04-05T23:32:38.378621+0000","last_undegraded":"2024-04-05T23:32:38.378621+0000","last_fullsized":"2024-04-05T23:32:38.378621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_clean_scrub_stamp":"2024-04-05T23:28:33.945697+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:54:10.683526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.649757+0000","last_change":"2024-04-05T23:32:38.385626+0000","last_active":"2024-04-05T23:33:34.649757+0000","last_peered":"2024-04-05T23:33:34.649757+0000","last_clean":"2024-04-05T23:33:34.649757+0000","last_became_active":"2024-04-05T23:32:38.384654+0000","last_became_peered":"2024-04-05T23:32:38.384654+0000","last_unstale":"2024-04-05T23:33:34.649757+0000","last_undegraded":"2024-04-05T23:33:34.649757+0000","last_fullsized":"2024-04-05T23:33:34.649757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:55:42.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378826+0000","last_change":"2024-04-05T23:32:04.005978+0000","last_active":"2024-04-05T23:32:38.378826+0000","last_peered":"2024-04-05T23:32:38.378826+0000","last_clean":"2024-04-05T23:32:38.378826+0000","last_became_active":"2024-04-05T23:32:04.005638+0000","last_became_peered":"2024-04-05T23:32:04.005638+0000","last_unstale":"2024-04-05T23:32:38.378826+0000","last_undegraded":"2024-04-05T23:32:38.378826+0000","last_fullsized":"2024-04-05T23:32:38.378826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:39:35.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379513+0000","last_change":"2024-04-05T23:32:04.006727+0000","last_active":"2024-04-05T23:32:38.379513+0000","last_peered":"2024-04-05T23:32:38.379513+0000","last_clean":"2024-04-05T23:32:38.379513+0000","last_became_active":"2024-04-05T23:32:04.006414+0000","last_became_peered":"2024-04-05T23:32:04.006414+0000","last_unstale":"2024-04-05T23:32:38.379513+0000","last_undegraded":"2024-04-05T23:32:38.379513+0000","last_fullsized":"2024-04-05T23:32:38.379513+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:41:28.661118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.664992+0000","last_change":"2024-04-05T23:32:04.536893+0000","last_active":"2024-04-05T23:33:34.664992+0000","last_peered":"2024-04-05T23:33:34.664992+0000","last_clean":"2024-04-05T23:33:34.664992+0000","last_became_active":"2024-04-05T23:32:04.536629+0000","last_became_peered":"2024-04-05T23:32:04.536629+0000","last_unstale":"2024-04-05T23:33:34.664992+0000","last_undegraded":"2024-04-05T23:33:34.664992+0000","last_fullsized":"2024-04-05T23:33:34.664992+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:27:58.279768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'44","reported_seq":82,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.876599+0000","last_change":"2024-04-05T23:32:04.010202+0000","last_active":"2024-04-05T23:33:34.876599+0000","last_peered":"2024-04-05T23:33:34.876599+0000","last_clean":"2024-04-05T23:33:34.876599+0000","last_became_active":"2024-04-05T23:32:04.009889+0000","last_became_peered":"2024-04-05T23:32:04.009889+0000","last_unstale":"2024-04-05T23:33:34.876599+0000","last_undegraded":"2024-04-05T23:33:34.876599+0000","last_fullsized":"2024-04-05T23:33:34.876599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:12:13.121449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.667696+0000","last_change":"2024-04-05T23:32:04.533576+0000","last_active":"2024-04-05T23:33:34.667696+0000","last_peered":"2024-04-05T23:33:34.667696+0000","last_clean":"2024-04-05T23:33:34.667696+0000","last_became_active":"2024-04-05T23:32:04.533066+0000","last_became_peered":"2024-04-05T23:32:04.533066+0000","last_unstale":"2024-04-05T23:33:34.667696+0000","last_undegraded":"2024-04-05T23:33:34.667696+0000","last_fullsized":"2024-04-05T23:33:34.667696+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:01:44.019210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.461143+0000","last_change":"2024-04-05T23:32:04.005096+0000","last_active":"2024-04-05T23:32:38.461143+0000","last_peered":"2024-04-05T23:32:38.461143+0000","last_clean":"2024-04-05T23:32:38.461143+0000","last_became_active":"2024-04-05T23:32:04.004702+0000","last_became_peered":"2024-04-05T23:32:04.004702+0000","last_unstale":"2024-04-05T23:32:38.461143+0000","last_undegraded":"2024-04-05T23:32:38.461143+0000","last_fullsized":"2024-04-05T23:32:38.461143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:33:09.934039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670944+0000","last_change":"2024-04-05T23:32:04.535623+0000","last_active":"2024-04-05T23:33:34.670944+0000","last_peered":"2024-04-05T23:33:34.670944+0000","last_clean":"2024-04-05T23:33:34.670944+0000","last_became_active":"2024-04-05T23:32:04.535223+0000","last_became_peered":"2024-04-05T23:32:04.535223+0000","last_unstale":"2024-04-05T23:33:34.670944+0000","last_undegraded":"2024-04-05T23:33:34.670944+0000","last_fullsized":"2024-04-05T23:33:34.670944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:24:43.543119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377010+0000","last_change":"2024-04-05T23:32:04.004097+0000","last_active":"2024-04-05T23:32:38.377010+0000","last_peered":"2024-04-05T23:32:38.377010+0000","last_clean":"2024-04-05T23:32:38.377010+0000","last_became_active":"2024-04-05T23:32:04.003713+0000","last_became_peered":"2024-04-05T23:32:04.003713+0000","last_unstale":"2024-04-05T23:32:38.377010+0000","last_undegraded":"2024-04-05T23:32:38.377010+0000","last_fullsized":"2024-04-05T23:32:38.377010+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:47:43.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"82'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.627575+0000","last_change":"2024-04-05T23:32:04.538843+0000","last_active":"2024-04-05T23:33:34.627575+0000","last_peered":"2024-04-05T23:33:34.627575+0000","last_clean":"2024-04-05T23:33:34.627575+0000","last_became_active":"2024-04-05T23:32:04.538472+0000","last_became_peered":"2024-04-05T23:32:04.538472+0000","last_unstale":"2024-04-05T23:33:34.627575+0000","last_undegraded":"2024-04-05T23:33:34.627575+0000","last_fullsized":"2024-04-05T23:33:34.627575+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:40:40.863815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.373470+0000","last_change":"2024-04-05T23:32:04.013064+0000","last_active":"2024-04-05T23:32:38.373470+0000","last_peered":"2024-04-05T23:32:38.373470+0000","last_clean":"2024-04-05T23:32:38.373470+0000","last_became_active":"2024-04-05T23:32:04.012751+0000","last_became_peered":"2024-04-05T23:32:04.012751+0000","last_unstale":"2024-04-05T23:32:38.373470+0000","last_undegraded":"2024-04-05T23:32:38.373470+0000","last_fullsized":"2024-04-05T23:32:38.373470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:06:10.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673173+0000","last_change":"2024-04-05T23:32:04.529014+0000","last_active":"2024-04-05T23:33:34.673173+0000","last_peered":"2024-04-05T23:33:34.673173+0000","last_clean":"2024-04-05T23:33:34.673173+0000","last_became_active":"2024-04-05T23:32:04.528622+0000","last_became_peered":"2024-04-05T23:32:04.528622+0000","last_unstale":"2024-04-05T23:33:34.673173+0000","last_undegraded":"2024-04-05T23:33:34.673173+0000","last_fullsized":"2024-04-05T23:33:34.673173+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:46:26.390934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377465+0000","last_change":"2024-04-05T23:32:04.002651+0000","last_active":"2024-04-05T23:32:38.377465+0000","last_peered":"2024-04-05T23:32:38.377465+0000","last_clean":"2024-04-05T23:32:38.377465+0000","last_became_active":"2024-04-05T23:32:04.002346+0000","last_became_peered":"2024-04-05T23:32:04.002346+0000","last_unstale":"2024-04-05T23:32:38.377465+0000","last_undegraded":"2024-04-05T23:32:38.377465+0000","last_fullsized":"2024-04-05T23:32:38.377465+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:26:48.176254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669156+0000","last_change":"2024-04-05T23:32:04.538785+0000","last_active":"2024-04-05T23:33:34.669156+0000","last_peered":"2024-04-05T23:33:34.669156+0000","last_clean":"2024-04-05T23:33:34.669156+0000","last_became_active":"2024-04-05T23:32:04.538463+0000","last_became_peered":"2024-04-05T23:32:04.538463+0000","last_unstale":"2024-04-05T23:33:34.669156+0000","last_undegraded":"2024-04-05T23:33:34.669156+0000","last_fullsized":"2024-04-05T23:33:34.669156+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:44:38.941857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379009+0000","last_change":"2024-04-05T23:32:04.003760+0000","last_active":"2024-04-05T23:32:38.379009+0000","last_peered":"2024-04-05T23:32:38.379009+0000","last_clean":"2024-04-05T23:32:38.379009+0000","last_became_active":"2024-04-05T23:32:04.003389+0000","last_became_peered":"2024-04-05T23:32:04.003389+0000","last_unstale":"2024-04-05T23:32:38.379009+0000","last_undegraded":"2024-04-05T23:32:38.379009+0000","last_fullsized":"2024-04-05T23:32:38.379009+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:42:00.353651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661087+0000","last_change":"2024-04-05T23:32:04.540051+0000","last_active":"2024-04-05T23:33:34.661087+0000","last_peered":"2024-04-05T23:33:34.661087+0000","last_clean":"2024-04-05T23:33:34.661087+0000","last_became_active":"2024-04-05T23:32:04.539736+0000","last_became_peered":"2024-04-05T23:32:04.539736+0000","last_unstale":"2024-04-05T23:33:34.661087+0000","last_undegraded":"2024-04-05T23:33:34.661087+0000","last_fullsized":"2024-04-05T23:33:34.661087+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:54:14.086904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'7","reported_seq":110,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.559133+0000","last_change":"2024-04-05T23:32:04.006492+0000","last_active":"2024-04-05T23:33:35.559133+0000","last_peered":"2024-04-05T23:33:35.559133+0000","last_clean":"2024-04-05T23:33:35.559133+0000","last_became_active":"2024-04-05T23:32:04.006154+0000","last_became_peered":"2024-04-05T23:32:04.006154+0000","last_unstale":"2024-04-05T23:33:35.559133+0000","last_undegraded":"2024-04-05T23:33:35.559133+0000","last_fullsized":"2024-04-05T23:33:35.559133+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:53:23.431400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1908,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661003+0000","last_change":"2024-04-05T23:32:04.540757+0000","last_active":"2024-04-05T23:33:34.661003+0000","last_peered":"2024-04-05T23:33:34.661003+0000","last_clean":"2024-04-05T23:33:34.661003+0000","last_became_active":"2024-04-05T23:32:04.540359+0000","last_became_peered":"2024-04-05T23:32:04.540359+0000","last_unstale":"2024-04-05T23:33:34.661003+0000","last_undegraded":"2024-04-05T23:33:34.661003+0000","last_fullsized":"2024-04-05T23:33:34.661003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:43:52.533489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'41","reported_seq":64,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.873377+0000","last_change":"2024-04-05T23:32:04.002736+0000","last_active":"2024-04-05T23:33:34.873377+0000","last_peered":"2024-04-05T23:33:34.873377+0000","last_clean":"2024-04-05T23:33:34.873377+0000","last_became_active":"2024-04-05T23:32:04.002277+0000","last_became_peered":"2024-04-05T23:32:04.002277+0000","last_unstale":"2024-04-05T23:33:34.873377+0000","last_undegraded":"2024-04-05T23:33:34.873377+0000","last_fullsized":"2024-04-05T23:33:34.873377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:56:39.001732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670914+0000","last_change":"2024-04-05T23:32:04.537076+0000","last_active":"2024-04-05T23:33:34.670914+0000","last_peered":"2024-04-05T23:33:34.670914+0000","last_clean":"2024-04-05T23:33:34.670914+0000","last_became_active":"2024-04-05T23:32:04.536707+0000","last_became_peered":"2024-04-05T23:32:04.536707+0000","last_unstale":"2024-04-05T23:33:34.670914+0000","last_undegraded":"2024-04-05T23:33:34.670914+0000","last_fullsized":"2024-04-05T23:33:34.670914+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:51:09.395408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378236+0000","last_change":"2024-04-05T23:32:04.006122+0000","last_active":"2024-04-05T23:32:38.378236+0000","last_peered":"2024-04-05T23:32:38.378236+0000","last_clean":"2024-04-05T23:32:38.378236+0000","last_became_active":"2024-04-05T23:32:04.005683+0000","last_became_peered":"2024-04-05T23:32:04.005683+0000","last_unstale":"2024-04-05T23:32:38.378236+0000","last_undegraded":"2024-04-05T23:32:38.378236+0000","last_fullsized":"2024-04-05T23:32:38.378236+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:21:40.922614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661469+0000","last_change":"2024-04-05T23:32:04.533321+0000","last_active":"2024-04-05T23:33:34.661469+0000","last_peered":"2024-04-05T23:33:34.661469+0000","last_clean":"2024-04-05T23:33:34.661469+0000","last_became_active":"2024-04-05T23:32:04.532949+0000","last_became_peered":"2024-04-05T23:32:04.532949+0000","last_unstale":"2024-04-05T23:33:34.661469+0000","last_undegraded":"2024-04-05T23:33:34.661469+0000","last_fullsized":"2024-04-05T23:33:34.661469+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:31.500132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.648745+0000","last_change":"2024-04-05T23:32:04.536508+0000","last_active":"2024-04-05T23:33:34.648745+0000","last_peered":"2024-04-05T23:33:34.648745+0000","last_clean":"2024-04-05T23:33:34.648745+0000","last_became_active":"2024-04-05T23:32:04.536142+0000","last_became_peered":"2024-04-05T23:32:04.536142+0000","last_unstale":"2024-04-05T23:33:34.648745+0000","last_undegraded":"2024-04-05T23:33:34.648745+0000","last_fullsized":"2024-04-05T23:33:34.648745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:18:28.497807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"82'12","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652638+0000","last_change":"2024-04-05T23:32:04.545511+0000","last_active":"2024-04-05T23:33:34.652638+0000","last_peered":"2024-04-05T23:33:34.652638+0000","last_clean":"2024-04-05T23:33:34.652638+0000","last_became_active":"2024-04-05T23:32:04.545191+0000","last_became_peered":"2024-04-05T23:32:04.545191+0000","last_unstale":"2024-04-05T23:33:34.652638+0000","last_undegraded":"2024-04-05T23:33:34.652638+0000","last_fullsized":"2024-04-05T23:33:34.652638+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:17:06.431429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.668751+0000","last_change":"2024-04-05T23:32:04.549673+0000","last_active":"2024-04-05T23:33:34.668751+0000","last_peered":"2024-04-05T23:33:34.668751+0000","last_clean":"2024-04-05T23:33:34.668751+0000","last_became_active":"2024-04-05T23:32:04.549391+0000","last_became_peered":"2024-04-05T23:32:04.549391+0000","last_unstale":"2024-04-05T23:33:34.668751+0000","last_undegraded":"2024-04-05T23:33:34.668751+0000","last_fullsized":"2024-04-05T23:33:34.668751+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:30:22.689107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666260+0000","last_change":"2024-04-05T23:32:04.535735+0000","last_active":"2024-04-05T23:33:34.666260+0000","last_peered":"2024-04-05T23:33:34.666260+0000","last_clean":"2024-04-05T23:33:34.666260+0000","last_became_active":"2024-04-05T23:32:04.534461+0000","last_became_peered":"2024-04-05T23:32:04.534461+0000","last_unstale":"2024-04-05T23:33:34.666260+0000","last_undegraded":"2024-04-05T23:33:34.666260+0000","last_fullsized":"2024-04-05T23:33:34.666260+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:37:41.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"82'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.671261+0000","last_change":"2024-04-05T23:32:04.532340+0000","last_active":"2024-04-05T23:33:34.671261+0000","last_peered":"2024-04-05T23:33:34.671261+0000","last_clean":"2024-04-05T23:33:34.671261+0000","last_became_active":"2024-04-05T23:32:04.530053+0000","last_became_peered":"2024-04-05T23:32:04.530053+0000","last_unstale":"2024-04-05T23:33:34.671261+0000","last_undegraded":"2024-04-05T23:33:34.671261+0000","last_fullsized":"2024-04-05T23:33:34.671261+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:16:11.168366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.646897+0000","last_change":"2024-04-05T23:32:04.529972+0000","last_active":"2024-04-05T23:33:34.646897+0000","last_peered":"2024-04-05T23:33:34.646897+0000","last_clean":"2024-04-05T23:33:34.646897+0000","last_became_active":"2024-04-05T23:32:04.529551+0000","last_became_peered":"2024-04-05T23:32:04.529551+0000","last_unstale":"2024-04-05T23:33:34.646897+0000","last_undegraded":"2024-04-05T23:33:34.646897+0000","last_fullsized":"2024-04-05T23:33:34.646897+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:13.764971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"82'15","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.648772+0000","last_change":"2024-04-05T23:32:04.529166+0000","last_active":"2024-04-05T23:33:34.648772+0000","last_peered":"2024-04-05T23:33:34.648772+0000","last_clean":"2024-04-05T23:33:34.648772+0000","last_became_active":"2024-04-05T23:32:04.528785+0000","last_became_peered":"2024-04-05T23:32:04.528785+0000","last_unstale":"2024-04-05T23:33:34.648772+0000","last_undegraded":"2024-04-05T23:33:34.648772+0000","last_fullsized":"2024-04-05T23:33:34.648772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:29:43.964265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"82'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.660708+0000","last_change":"2024-04-05T23:32:04.540631+0000","last_active":"2024-04-05T23:33:34.660708+0000","last_peered":"2024-04-05T23:33:34.660708+0000","last_clean":"2024-04-05T23:33:34.660708+0000","last_became_active":"2024-04-05T23:32:04.540327+0000","last_became_peered":"2024-04-05T23:32:04.540327+0000","last_unstale":"2024-04-05T23:33:34.660708+0000","last_undegraded":"2024-04-05T23:33:34.660708+0000","last_fullsized":"2024-04-05T23:33:34.660708+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:22:48.825324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.656475+0000","last_change":"2024-04-05T23:32:04.544845+0000","last_active":"2024-04-05T23:33:34.656475+0000","last_peered":"2024-04-05T23:33:34.656475+0000","last_clean":"2024-04-05T23:33:34.656475+0000","last_became_active":"2024-04-05T23:32:04.544472+0000","last_became_peered":"2024-04-05T23:32:04.544472+0000","last_unstale":"2024-04-05T23:33:34.656475+0000","last_undegraded":"2024-04-05T23:33:34.656475+0000","last_fullsized":"2024-04-05T23:33:34.656475+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:34:09.529644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652655+0000","last_change":"2024-04-05T23:32:38.382577+0000","last_active":"2024-04-05T23:33:34.652655+0000","last_peered":"2024-04-05T23:33:34.652655+0000","last_clean":"2024-04-05T23:33:34.652655+0000","last_became_active":"2024-04-05T23:32:38.380286+0000","last_became_peered":"2024-04-05T23:32:38.380286+0000","last_unstale":"2024-04-05T23:33:34.652655+0000","last_undegraded":"2024-04-05T23:33:34.652655+0000","last_fullsized":"2024-04-05T23:33:34.652655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:42:52.270935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670550+0000","last_change":"2024-04-05T23:32:04.534631+0000","last_active":"2024-04-05T23:33:34.670550+0000","last_peered":"2024-04-05T23:33:34.670550+0000","last_clean":"2024-04-05T23:33:34.670550+0000","last_became_active":"2024-04-05T23:32:04.534330+0000","last_became_peered":"2024-04-05T23:32:04.534330+0000","last_unstale":"2024-04-05T23:33:34.670550+0000","last_undegraded":"2024-04-05T23:33:34.670550+0000","last_fullsized":"2024-04-05T23:33:34.670550+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:20:47.592174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"82'11","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669338+0000","last_change":"2024-04-05T23:32:04.531459+0000","last_active":"2024-04-05T23:33:34.669338+0000","last_peered":"2024-04-05T23:33:34.669338+0000","last_clean":"2024-04-05T23:33:34.669338+0000","last_became_active":"2024-04-05T23:32:04.530828+0000","last_became_peered":"2024-04-05T23:32:04.530828+0000","last_unstale":"2024-04-05T23:33:34.669338+0000","last_undegraded":"2024-04-05T23:33:34.669338+0000","last_fullsized":"2024-04-05T23:33:34.669338+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:51:34.090030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652270+0000","last_change":"2024-04-05T23:32:04.527711+0000","last_active":"2024-04-05T23:33:34.652270+0000","last_peered":"2024-04-05T23:33:34.652270+0000","last_clean":"2024-04-05T23:33:34.652270+0000","last_became_active":"2024-04-05T23:32:04.527374+0000","last_became_peered":"2024-04-05T23:32:04.527374+0000","last_unstale":"2024-04-05T23:33:34.652270+0000","last_undegraded":"2024-04-05T23:33:34.652270+0000","last_fullsized":"2024-04-05T23:33:34.652270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:15:10.764340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599588,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":997,"num_read_kb":612,"num_write":360,"num_write_kb":1458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2162688,"data_stored":1798764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579931,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":440050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.60599999999999998,"15min":0.59599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.968,"5min":3.968,"15min":3.968},"last":0.434}]},{"osd":1,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.64400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80400000000000005,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.64100000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69399999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.48199999999999998,"15min":0.48999999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67400000000000004,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":4,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.79300000000000004,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.75},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80400000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.66400000000000003}]},{"osd":5,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":1.353,"15min":1.353},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79400000000000004,"5min":1.306,"15min":1.306},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.74299999999999999,"15min":0.751},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85399999999999998,"5min":1.109,"15min":1.109},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.751,"15min":0.76600000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.89400000000000002,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.71699999999999997}]},{"osd":8,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64300000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.78500000000000003,"15min":0.80300000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.92800000000000005,"5min":1.71,"15min":1.71},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.74199999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.95699999999999996,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.78300000000000003}]},{"osd":10,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.78500000000000003,"15min":0.79700000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.94299999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.80100000000000005,"15min":0.81799999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.877,"5min":1.667,"15min":1.667},"last":0.73599999999999999}]}]},{"osd":10,"up_from":67,"seq":287762808863,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1744896,"data_stored":1228526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64000000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.35699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.878,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.372}]},{"osd":2,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66700000000000004,"15min":0.66800000000000004},"min":{"1min":0.52900000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65500000000000003,"15min":0.65900000000000003},"min":{"1min":0.435,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.81699999999999995,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.72399999999999998,"15min":0.70399999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.8660000000000001,"5min":3.8660000000000001,"15min":3.8660000000000001},"last":0.43099999999999999}]},{"osd":4,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81499999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.308},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83099999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.39000000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.94299999999999995,"5min":1.145,"15min":1.145},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0269999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.48499999999999999,"5min":0.441,"15min":0.441},"max":{"1min":0.96199999999999997,"5min":1.228,"15min":1.228},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.76700000000000002,"15min":0.77100000000000002},"min":{"1min":0.51200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.92600000000000005,"5min":1.129,"15min":1.129},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79900000000000004,"15min":0.80100000000000005},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0129999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.79200000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.042,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.41999999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.85399999999999998,"15min":0.86699999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0109999999999999,"5min":1.444,"15min":1.444},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.82299999999999995,"15min":0.83299999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.89900000000000002,"5min":1.258,"15min":1.258},"last":0.505}]},{"osd":11,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.82399999999999995,"15min":0.82499999999999996},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0149999999999999,"5min":1.351,"15min":1.351},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.80600000000000005,"15min":0.80900000000000005},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.94999999999999996,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.629}]}]},{"osd":9,"up_from":62,"seq":266287972387,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":439298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.55500000000000005,"15min":0.57099999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67700000000000005,"5min":0.746,"15min":0.746},"last":0.45800000000000002}]},{"osd":1,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.234,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77900000000000003,"15min":0.79700000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1060000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.70599999999999996}]},{"osd":2,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.74299999999999999,"15min":0.76300000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90900000000000003,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.754,"15min":0.77200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.036,"5min":1.256,"15min":1.256},"last":0.72299999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.78700000000000003,"15min":0.81100000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95699999999999996,"5min":1.27,"15min":1.27},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.71899999999999997,"15min":0.74099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.97599999999999998,"5min":1.405,"15min":1.405},"last":0.65300000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0329999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.87,"15min":0.88600000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.86699999999999999,"15min":0.89300000000000002},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.98399999999999999,"5min":1.637,"15min":1.637},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.89600000000000002,"15min":0.91900000000000004},"min":{"1min":0.55800000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1299999999999999,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.745}]},{"osd":6,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.94099999999999995,"15min":0.96899999999999997},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.071,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.94499999999999995,"15min":0.95899999999999996},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1619999999999999,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.68300000000000005}]},{"osd":8,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.0069999999999999,"15min":1.0309999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.173,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.996,"15min":1.02},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.127,"5min":1.778,"15min":1.778},"last":0.84799999999999998}]},{"osd":10,"last update":"Fri Apr 5 23:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.95499999999999996,"15min":0.97699999999999998},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.2110000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.05},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.89500000000000002,"15min":0.91100000000000003},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.111,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.79200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]}]},{"osd":8,"up_from":55,"seq":236223201320,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1314816,"data_stored":832310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56200000000000006,"15min":0.57399999999999995},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68300000000000005,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.63800000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.79400000000000004}]},{"osd":2,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.46200000000000002,"15min":0.44800000000000001},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.46400000000000002,"15min":0.45900000000000002},"min":{"1min":0.308,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.65300000000000002,"5min":0.748,"15min":0.748},"last":0.496}]},{"osd":3,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.45900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":1.101,"15min":1.101},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.55000000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.626,"15min":0.61399999999999999},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.91000000000000003,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64300000000000002,"15min":0.64100000000000001},"min":{"1min":0.38500000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85999999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.66600000000000004}]},{"osd":5,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.35499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69499999999999995,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.40500000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.58599999999999997}]},{"osd":6,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.68899999999999995,"15min":0.69199999999999995},"min":{"1min":0.48699999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.56899999999999995}]},{"osd":7,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.878,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94699999999999995,"5min":1.276,"15min":1.276},"last":0.75700000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72299999999999998,"15min":0.71299999999999997},"min":{"1min":0.56799999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73899999999999999,"15min":0.73399999999999999},"min":{"1min":0.60499999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.68999999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.73799999999999999,"15min":0.72099999999999997},"min":{"1min":0.66600000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.73999999999999999,"15min":0.72399999999999998},"min":{"1min":0.66900000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.77500000000000002}]},{"osd":11,"last update":"Fri Apr 5 23:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.74299999999999999}]}]},{"osd":7,"up_from":48,"seq":206158430250,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32604,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712548,"statfs":{"total":95995035648,"available":95961649152,"internally_reserved":0,"allocated":1523712,"data_stored":1033930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.76400000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64800000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.434,"15min":0.42199999999999999},"min":{"1min":0.36199999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.34499999999999997}]},{"osd":2,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.63300000000000001,"5min":0.879,"15min":0.879},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.58299999999999996},"min":{"1min":0.438,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.69099999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52300000000000002}]},{"osd":3,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58999999999999997,"15min":0.58399999999999996},"min":{"1min":0.42699999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.621,"15min":0.60999999999999999},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.872,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.497}]},{"osd":4,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.51300000000000001},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.68999999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.505,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.64000000000000001,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.39000000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.61699999999999999,"15min":0.621},"min":{"1min":0.438,"5min":0.378,"15min":0.378},"max":{"1min":0.73699999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65400000000000003,"15min":0.65000000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.83799999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.56799999999999995,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67900000000000005,"15min":0.67800000000000005},"min":{"1min":0.47999999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.73999999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.47599999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.49299999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.81000000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72999999999999998,"15min":0.72899999999999998},"min":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94899999999999995,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.65200000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.77200000000000002,"15min":0.77400000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.96999999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.79400000000000004,"15min":0.79600000000000004},"min":{"1min":0.67400000000000004,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.875,"5min":1.216,"15min":1.216},"last":0.68200000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.52400000000000002,"5min":0.498,"15min":0.498},"max":{"1min":0.95199999999999996,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.623},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.755,"15min":0.751},"min":{"1min":0.50600000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.92700000000000005,"5min":1.099,"15min":1.099},"last":0.55200000000000005}]},{"osd":11,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.67300000000000004}]}]},{"osd":6,"up_from":42,"seq":180388626478,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":909312,"data_stored":439480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.34200000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.39300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.92500000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.65800000000000003,"15min":0.63800000000000001},"min":{"1min":0.50900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.85999999999999999}]},{"osd":2,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63,"15min":0.628},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.004,"5min":0.73599999999999999,"15min":0.69699999999999995},"min":{"1min":0.47499999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":6.867,"5min":6.867,"15min":6.867},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.44700000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.095,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.68999999999999995,"15min":0.67400000000000004},"min":{"1min":0.46600000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.169,"5min":1.246,"15min":1.246},"last":0.89800000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.47399999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.45700000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.76500000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.79400000000000004,"15min":0.753},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":6.9189999999999996,"5min":6.9189999999999996,"15min":6.9189999999999996},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.3180000000000001,"5min":0.86199999999999999,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":6.9009999999999998,"5min":6.9009999999999998,"15min":6.9009999999999998},"last":0.97999999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.58199999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":6.883,"5min":6.883,"15min":6.883},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.84099999999999997,"15min":0.80200000000000005},"min":{"1min":0.61799999999999999,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":4.8559999999999999,"5min":4.8559999999999999,"15min":4.8559999999999999},"last":0.94999999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":0.83899999999999997,"15min":0.80900000000000005},"min":{"1min":0.64900000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.81200000000000006,"15min":0.79600000000000004},"min":{"1min":0.58199999999999996,"5min":0.504,"15min":0.504},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":1}]},{"osd":9,"last update":"Fri Apr 5 23:32:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91200000000000003,"15min":0.91600000000000004},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.63400000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.123,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.93500000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.877,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.91800000000000004,"15min":0.91600000000000004},"min":{"1min":0.74199999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":1.0209999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607618,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36160,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708992,"statfs":{"total":95995035648,"available":95958007808,"internally_reserved":0,"allocated":974848,"data_stored":505878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52100000000000002,"15min":0.504},"min":{"1min":0.45000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68799999999999994,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61499999999999999,"15min":0.67600000000000005},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.79900000000000004,"5min":4.4199999999999999,"15min":4.4199999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Apr 5 23:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55200000000000005,"15min":0.52000000000000002},"min":{"1min":0.34699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.57299999999999995,"15min":0.52100000000000002},"min":{"1min":0.53400000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59499999999999997,"15min":0.54500000000000004},"min":{"1min":0.44900000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56999999999999995,"15min":0.54400000000000004},"min":{"1min":0.45100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89200000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Apr 5 23:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.42799999999999999,"15min":0.40500000000000003},"min":{"1min":0.33700000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.495},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.46800000000000003,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.877,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.52100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91600000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.71399999999999997,"15min":0.67200000000000004},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.67900000000000005}]},{"osd":6,"last update":"Fri Apr 5 23:32:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70999999999999996,"15min":0.68600000000000005},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.438},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66200000000000003,"15min":0.65200000000000002},"min":{"1min":0.441,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57599999999999996}]},{"osd":7,"last update":"Fri Apr 5 23:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.63700000000000001,"15min":0.61099999999999999},"min":{"1min":0.40400000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90300000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.57499999999999996},"min":{"1min":0.38,"5min":0.312,"15min":0.312},"max":{"1min":0.81899999999999995,"5min":1.002,"15min":1.002},"last":0.436}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.64200000000000002,"15min":0.62},"min":{"1min":0.54100000000000004,"5min":0.378,"15min":0.378},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.754,"15min":0.71799999999999997},"min":{"1min":0.71399999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.71399999999999997}]},{"osd":9,"last update":"Fri Apr 5 23:32:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.72299999999999998,"15min":0.71099999999999997},"min":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.69899999999999995,"15min":0.68999999999999995},"min":{"1min":0.60699999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.627}]},{"osd":10,"last update":"Fri Apr 5 23:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86699999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.505},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.80500000000000005,"15min":0.79500000000000004},"min":{"1min":0.59499999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.56299999999999994}]},{"osd":11,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.748}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36124,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709028,"statfs":{"total":95995035648,"available":95958044672,"internally_reserved":0,"allocated":933888,"data_stored":441092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 23:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57299999999999995,"15min":0.51500000000000001},"min":{"1min":0.40300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.97199999999999998,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56100000000000005,"15min":0.50700000000000001},"min":{"1min":0.38100000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85999999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.63200000000000001,"15min":0.58799999999999997},"min":{"1min":0.47999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0049999999999999,"5min":7.2060000000000004,"15min":7.2060000000000004},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:33:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.91600000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.78900000000000003,"15min":0.91500000000000004},"min":{"1min":0.51800000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92500000000000004,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":1.258},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.66000000000000003},"min":{"1min":0.57999999999999996,"5min":0.436,"15min":0.436},"max":{"1min":0.88200000000000001,"5min":1.395,"15min":1.395},"last":0.79900000000000004}]},{"osd":5,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68899999999999995,"15min":0.66800000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.2250000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.70999999999999996},"min":{"1min":0.55900000000000005,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":1.339}]},{"osd":6,"last update":"Fri Apr 5 23:32:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.83399999999999996,"15min":0.87},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.5489999999999999,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.2390000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78400000000000003,"15min":0.77700000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":1.294}]},{"osd":7,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.70699999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.89200000000000002,"15min":0.93000000000000005},"min":{"1min":0.499,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.2789999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.306},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.93700000000000006,"15min":0.995},"min":{"1min":0.629,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4610000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.2010000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:32:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.94999999999999996,"15min":0.96499999999999997},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.88100000000000001,"15min":0.876},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.3919999999999999}]},{"osd":10,"last update":"Fri Apr 5 23:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.91300000000000003,"15min":0.92600000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":1.379},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.94199999999999995,"15min":0.95799999999999996},"min":{"1min":0.66900000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3080000000000001,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.2789999999999999}]},{"osd":11,"last update":"Fri Apr 5 23:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.89800000000000002,"15min":0.90200000000000002},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.92000000000000004,"15min":0.92900000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":1.415}]}]},{"osd":2,"up_from":17,"seq":73014444095,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":987136,"data_stored":504410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76400000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.751,"5min":0.874,"15min":0.874},"last":0.44}]},{"osd":3,"last update":"Fri Apr 5 23:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.311,"15min":0.311},"max":{"1min":0.82299999999999995,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.62}]},{"osd":4,"last update":"Fri Apr 5 23:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.503,"15min":0.47799999999999998},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86699999999999999,"5min":1.359,"15min":1.359},"last":0.60499999999999998}]},{"osd":6,"last update":"Fri Apr 5 23:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.45100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.58999999999999997},"min":{"1min":0.443,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.76100000000000001,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.49399999999999999}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73199999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.48399999999999999,"15min":0.47299999999999998},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70499999999999996,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.63400000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.874,"5min":1.532,"15min":1.532},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85399999999999998,"5min":1.429,"15min":1.429},"last":0.66800000000000004}]},{"osd":10,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.82099999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72499999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.88700000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.65200000000000002}]},{"osd":11,"last update":"Fri Apr 5 23:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.69599999999999995}]}]},{"osd":3,"up_from":23,"seq":98784247866,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36628,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708524,"statfs":{"total":95995035648,"available":95957528576,"internally_reserved":0,"allocated":1454080,"data_stored":963383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.45600000000000002},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.995,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52700000000000002,"15min":0.46899999999999997},"min":{"1min":0.34200000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.996,"5min":4.96,"15min":4.96},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Apr 5 23:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66200000000000003,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2390000000000001,"5min":4.5590000000000002,"15min":4.5590000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.59899999999999998},"min":{"1min":0.35899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":1.0149999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Apr 5 23:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.68200000000000005},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.107,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72599999999999998,"15min":0.67800000000000005},"min":{"1min":0.5,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.167,"5min":4.6710000000000003,"15min":4.6710000000000003},"last":0.76600000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73499999999999999,"15min":0.67600000000000005},"min":{"1min":0.58099999999999996,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1519999999999999,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.753,"15min":0.67900000000000005},"min":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.278,"5min":4.6289999999999996,"15min":4.6289999999999996},"last":0.73199999999999998}]},{"osd":5,"last update":"Fri Apr 5 23:33:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76100000000000001,"15min":0.70099999999999996},"min":{"1min":0.52800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.095,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.78600000000000003,"15min":0.72799999999999998},"min":{"1min":0.47999999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.96699999999999997,"5min":4.8929999999999998,"15min":4.8929999999999998},"last":0.82799999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:32:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.84399999999999997,"15min":0.872},"min":{"1min":0.497,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.29,"5min":4.8159999999999998,"15min":4.8159999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.747,"15min":0.80100000000000005},"min":{"1min":0.433,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0569999999999999,"5min":5.0979999999999999,"15min":5.0979999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.96999999999999997,"15min":0.998},"min":{"1min":0.50800000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":4.7610000000000001,"5min":4.9980000000000002,"15min":4.9980000000000002},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.044,"5min":0.94899999999999995,"15min":0.96699999999999997},"min":{"1min":0.58999999999999997,"5min":0.502,"15min":0.502},"max":{"1min":4.7759999999999998,"5min":4.7759999999999998,"15min":4.7759999999999998},"last":1.149}]},{"osd":8,"last update":"Fri Apr 5 23:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":4.7569999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.0300000000000002,"5min":5.0300000000000002,"15min":5.0300000000000002},"last":0.72199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.94399999999999995,"15min":0.96599999999999997},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.081,"5min":1.349,"15min":1.349},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.80800000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.95699999999999996,"15min":0.97199999999999998},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.224,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.876,"15min":0.878},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":1.1890000000000001}]},{"osd":11,"last update":"Fri Apr 5 23:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":1.0920000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":909312,"data_stored":439647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.40799999999999997,"15min":0.374},"min":{"1min":0.35899999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.65200000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.39100000000000001},"min":{"1min":0.32500000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.71299999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.39800000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76900000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58199999999999996,"15min":0.55400000000000005},"min":{"1min":0.45900000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.76500000000000001,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.56999999999999995},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76700000000000002,"5min":1.117,"15min":1.117},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71899999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Apr 5 23:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63800000000000001,"15min":0.60899999999999999},"min":{"1min":0.53300000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46800000000000003}]},{"osd":6,"last update":"Fri Apr 5 23:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92900000000000005,"5min":1.137,"15min":1.137},"last":0.505},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.627,"15min":0.59899999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89700000000000002,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.55800000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95299999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86299999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.34799999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.48899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.874,"5min":1.087,"15min":1.087},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.755,"15min":0.747},"min":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.434}]},{"osd":9,"last update":"Fri Apr 5 23:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80900000000000005,"15min":0.81299999999999994},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.071,"5min":1.28,"15min":1.28},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.53800000000000003}]},{"osd":10,"last update":"Fri Apr 5 23:33:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.752,"15min":0.75},"min":{"1min":0.55100000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95299999999999996,"5min":1.181,"15min":1.181},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0249999999999999,"5min":1.371,"15min":1.371},"last":0.54900000000000004}]},{"osd":11,"last update":"Fri Apr 5 23:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.58299999999999996}]}]},{"osd":5,"up_from":35,"seq":150323855410,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1536000,"data_stored":1031433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54100000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80200000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.34200000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71999999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68799999999999994,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.30299999999999999}]},{"osd":2,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.28499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.35799999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.58599999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.78200000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.317},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58399999999999996,"15min":0.57199999999999995},"min":{"1min":0.439,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.79700000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56899999999999995,"15min":0.55700000000000005},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60099999999999998,"15min":0.58899999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83299999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.38300000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:32:57 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.45600000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82199999999999995,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.441},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.48199999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84599999999999997,"5min":1.153,"15min":1.153},"last":0.51800000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.61699999999999999,"15min":0.61199999999999999},"min":{"1min":0.46700000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.48899999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53000000000000003,"15min":0.51500000000000001},"min":{"1min":0.34699999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.371,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70099999999999996,"15min":0.69199999999999995},"min":{"1min":0.59999999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.67100000000000004,"15min":0.65700000000000003},"min":{"1min":0.60199999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.89200000000000002,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.58899999999999997}]},{"osd":10,"last update":"Fri Apr 5 23:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.55400000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.67600000000000005,"15min":0.66500000000000004},"min":{"1min":0.53400000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":199865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T23:33:41.979 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-05T23:33:41.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-05T23:33:42.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:41 smithi130 ceph-mon[79417]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-05T23:33:42.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:41 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1832099073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:42.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:41 smithi130 ceph-mon[79417]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:33:42.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.220+0000 7f143a000700 1 -- 172.21.15.16:0/3464013915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1434101940 msgr2=0x7f1434105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.220+0000 7f143a000700 1 --2- 172.21.15.16:0/3464013915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1434101940 0x7f1434105980 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1420009990 tx=0x7f1420009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.220+0000 7f143a000700 1 -- 172.21.15.16:0/3464013915 shutdown_connections 2024-04-05T23:33:42.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.220+0000 7f143a000700 1 --2- 172.21.15.16:0/3464013915 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14341060b0 0x7f143410dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.220+0000 7f143a000700 1 --2- 172.21.15.16:0/3464013915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434100f90 0x7f1434101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.220+0000 7f143a000700 1 --2- 172.21.15.16:0/3464013915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1434101940 0x7f1434105980 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.220+0000 7f143a000700 1 -- 172.21.15.16:0/3464013915 >> 172.21.15.16:0/3464013915 conn(0x7f14340fc830 msgr2=0x7f14340fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:42.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.220+0000 7f143a000700 1 -- 172.21.15.16:0/3464013915 shutdown_connections 2024-04-05T23:33:42.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.220+0000 7f143a000700 1 -- 172.21.15.16:0/3464013915 wait complete. 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.221+0000 7f143a000700 1 Processor -- start 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.221+0000 7f143a000700 1 -- start start 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f143a000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1434100f90 0x7f14340731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f143a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14341060b0 0x7f1434073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f143a000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1434073c60 0x7f14341a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f143a000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14341a5ac0 con 0x7f1434073c60 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f143a000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14341a5c30 con 0x7f14341060b0 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f143a000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14341a5df0 con 0x7f1434100f90 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1432a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14341060b0 0x7f1434073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1433e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1434073c60 0x7f14341a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1432a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14341060b0 0x7f1434073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59170/0 (socket says 172.21.15.16:59170) 2024-04-05T23:33:42.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1432a00700 1 -- 172.21.15.16:0/1665588797 learned_addr learned my addr 172.21.15.16:0/1665588797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1433400700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1434100f90 0x7f14340731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1432a00700 1 -- 172.21.15.16:0/1665588797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1434100f90 msgr2=0x7f14340731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1432a00700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1434100f90 0x7f14340731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1432a00700 1 -- 172.21.15.16:0/1665588797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1434073c60 msgr2=0x7f14341a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1432a00700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1434073c60 0x7f14341a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.222+0000 7f1432a00700 1 -- 172.21.15.16:0/1665588797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1424009520 con 0x7f14341060b0 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.223+0000 7f1432a00700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14341060b0 0x7f1434073720 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1420000c00 tx=0x7f1420005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.223+0000 7f142be00700 1 -- 172.21.15.16:0/1665588797 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142001d070 con 0x7f14341060b0 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.223+0000 7f142be00700 1 -- 172.21.15.16:0/1665588797 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1420022470 con 0x7f14341060b0 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.223+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c009c40 con 0x7f14341060b0 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.223+0000 7f142be00700 1 -- 172.21.15.16:0/1665588797 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420017650 con 0x7f14341060b0 2024-04-05T23:33:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.223+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14200095f0 con 0x7f14341060b0 2024-04-05T23:33:42.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.224+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1434109380 con 0x7f14341060b0 2024-04-05T23:33:42.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.226+0000 7f142be00700 1 -- 172.21.15.16:0/1665588797 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f142000fad0 con 0x7f14341060b0 2024-04-05T23:33:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.227+0000 7f142be00700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14180a1320 0x7f14180a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.227+0000 7f142be00700 1 -- 172.21.15.16:0/1665588797 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f14200ce7c0 con 0x7f14341060b0 2024-04-05T23:33:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.228+0000 7f1433400700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14180a1320 0x7f14180a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.228+0000 7f1433400700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14180a1320 0x7f14180a37e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1424009df0 tx=0x7f1424009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.230+0000 7f142be00700 1 -- 172.21.15.16:0/1665588797 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1420097b00 con 0x7f14341060b0 2024-04-05T23:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.442+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 --> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14340746c0 con 0x7f14180a1320 2024-04-05T23:33:42.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.453+0000 7f142be00700 1 -- 172.21.15.16:0/1665588797 <== mgr.15240 v2:172.21.15.16:6800/3895420073 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343358 (secure 0 0 0) 0x7f14340746c0 con 0x7f14180a1320 2024-04-05T23:33:42.451 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:42.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14180a1320 msgr2=0x7f14180a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14180a1320 0x7f14180a37e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1424009df0 tx=0x7f1424009260 comp rx=0 tx=0).stop 2024-04-05T23:33:42.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14341060b0 msgr2=0x7f1434073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14341060b0 0x7f1434073720 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1420000c00 tx=0x7f1420005740 comp rx=0 tx=0).stop 2024-04-05T23:33:42.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 shutdown_connections 2024-04-05T23:33:42.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14180a1320 0x7f14180a37e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1434073c60 0x7f14341a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1434100f90 0x7f14340731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 --2- 172.21.15.16:0/1665588797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14341060b0 0x7f1434073720 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 >> 172.21.15.16:0/1665588797 conn(0x7f14340fc830 msgr2=0x7f14340fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:42.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 shutdown_connections 2024-04-05T23:33:42.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.458+0000 7f143a000700 1 -- 172.21.15.16:0/1665588797 wait complete. 2024-04-05T23:33:42.455 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-05T23:33:42.471 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-05T23:33:40.613366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190190,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1133,"num_read_kb":731,"num_write":543,"num_write_kb":4398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":932,"ondisk_log_size":932,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370856,"kb_used_data":13800,"kb_used_omap":18,"kb_used_meta":356845,"kb_avail":1124570968,"statfs":{"total":1151940427776,"available":1151560671232,"internally_reserved":0,"allocated":14131200,"data_stored":8299437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":365409605},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.976347"},"pg_stats":[{"pgid":"3.3f","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673349+0000","last_change":"2024-04-05T23:32:04.536879+0000","last_active":"2024-04-05T23:33:34.673349+0000","last_peered":"2024-04-05T23:33:34.673349+0000","last_clean":"2024-04-05T23:33:34.673349+0000","last_became_active":"2024-04-05T23:32:04.536183+0000","last_became_peered":"2024-04-05T23:32:04.536183+0000","last_unstale":"2024-04-05T23:33:34.673349+0000","last_undegraded":"2024-04-05T23:33:34.673349+0000","last_fullsized":"2024-04-05T23:33:34.673349+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:13:22.589464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"82'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670374+0000","last_change":"2024-04-05T23:32:04.540791+0000","last_active":"2024-04-05T23:33:34.670374+0000","last_peered":"2024-04-05T23:33:34.670374+0000","last_clean":"2024-04-05T23:33:34.670374+0000","last_became_active":"2024-04-05T23:32:04.540455+0000","last_became_peered":"2024-04-05T23:32:04.540455+0000","last_unstale":"2024-04-05T23:33:34.670374+0000","last_undegraded":"2024-04-05T23:33:34.670374+0000","last_fullsized":"2024-04-05T23:33:34.670374+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:06:19.090672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.671697+0000","last_change":"2024-04-05T23:32:04.543428+0000","last_active":"2024-04-05T23:33:34.671697+0000","last_peered":"2024-04-05T23:33:34.671697+0000","last_clean":"2024-04-05T23:33:34.671697+0000","last_became_active":"2024-04-05T23:32:04.543071+0000","last_became_peered":"2024-04-05T23:32:04.543071+0000","last_unstale":"2024-04-05T23:33:34.671697+0000","last_undegraded":"2024-04-05T23:33:34.671697+0000","last_fullsized":"2024-04-05T23:33:34.671697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:35:44.911749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651922+0000","last_change":"2024-04-05T23:32:04.531444+0000","last_active":"2024-04-05T23:33:34.651922+0000","last_peered":"2024-04-05T23:33:34.651922+0000","last_clean":"2024-04-05T23:33:34.651922+0000","last_became_active":"2024-04-05T23:32:04.529556+0000","last_became_peered":"2024-04-05T23:32:04.529556+0000","last_unstale":"2024-04-05T23:33:34.651922+0000","last_undegraded":"2024-04-05T23:33:34.651922+0000","last_fullsized":"2024-04-05T23:33:34.651922+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:48.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"82'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669652+0000","last_change":"2024-04-05T23:32:04.529565+0000","last_active":"2024-04-05T23:33:34.669652+0000","last_peered":"2024-04-05T23:33:34.669652+0000","last_clean":"2024-04-05T23:33:34.669652+0000","last_became_active":"2024-04-05T23:32:04.529226+0000","last_became_peered":"2024-04-05T23:32:04.529226+0000","last_unstale":"2024-04-05T23:33:34.669652+0000","last_undegraded":"2024-04-05T23:33:34.669652+0000","last_fullsized":"2024-04-05T23:33:34.669652+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:44:25.208996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.655487+0000","last_change":"2024-04-05T23:32:04.546260+0000","last_active":"2024-04-05T23:33:34.655487+0000","last_peered":"2024-04-05T23:33:34.655487+0000","last_clean":"2024-04-05T23:33:34.655487+0000","last_became_active":"2024-04-05T23:32:04.545851+0000","last_became_peered":"2024-04-05T23:32:04.545851+0000","last_unstale":"2024-04-05T23:33:34.655487+0000","last_undegraded":"2024-04-05T23:33:34.655487+0000","last_fullsized":"2024-04-05T23:33:34.655487+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:59:16.735635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.665495+0000","last_change":"2024-04-05T23:32:04.536348+0000","last_active":"2024-04-05T23:33:34.665495+0000","last_peered":"2024-04-05T23:33:34.665495+0000","last_clean":"2024-04-05T23:33:34.665495+0000","last_became_active":"2024-04-05T23:32:04.536015+0000","last_became_peered":"2024-04-05T23:32:04.536015+0000","last_unstale":"2024-04-05T23:33:34.665495+0000","last_undegraded":"2024-04-05T23:33:34.665495+0000","last_fullsized":"2024-04-05T23:33:34.665495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:02:24.897957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666748+0000","last_change":"2024-04-05T23:32:04.532528+0000","last_active":"2024-04-05T23:33:34.666748+0000","last_peered":"2024-04-05T23:33:34.666748+0000","last_clean":"2024-04-05T23:33:34.666748+0000","last_became_active":"2024-04-05T23:32:04.532104+0000","last_became_peered":"2024-04-05T23:32:04.532104+0000","last_unstale":"2024-04-05T23:33:34.666748+0000","last_undegraded":"2024-04-05T23:33:34.666748+0000","last_fullsized":"2024-04-05T23:33:34.666748+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:06.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"82'11","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651120+0000","last_change":"2024-04-05T23:32:04.531223+0000","last_active":"2024-04-05T23:33:34.651120+0000","last_peered":"2024-04-05T23:33:34.651120+0000","last_clean":"2024-04-05T23:33:34.651120+0000","last_became_active":"2024-04-05T23:32:04.530863+0000","last_became_peered":"2024-04-05T23:32:04.530863+0000","last_unstale":"2024-04-05T23:33:34.651120+0000","last_undegraded":"2024-04-05T23:33:34.651120+0000","last_fullsized":"2024-04-05T23:33:34.651120+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:03:35.912984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"82'16","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.653658+0000","last_change":"2024-04-05T23:32:04.541481+0000","last_active":"2024-04-05T23:33:34.653658+0000","last_peered":"2024-04-05T23:33:34.653658+0000","last_clean":"2024-04-05T23:33:34.653658+0000","last_became_active":"2024-04-05T23:32:04.541133+0000","last_became_peered":"2024-04-05T23:32:04.541133+0000","last_unstale":"2024-04-05T23:33:34.653658+0000","last_undegraded":"2024-04-05T23:33:34.653658+0000","last_fullsized":"2024-04-05T23:33:34.653658+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:01:22.825587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.660781+0000","last_change":"2024-04-05T23:32:04.542390+0000","last_active":"2024-04-05T23:33:34.660781+0000","last_peered":"2024-04-05T23:33:34.660781+0000","last_clean":"2024-04-05T23:33:34.660781+0000","last_became_active":"2024-04-05T23:32:04.542026+0000","last_became_peered":"2024-04-05T23:32:04.542026+0000","last_unstale":"2024-04-05T23:33:34.660781+0000","last_undegraded":"2024-04-05T23:33:34.660781+0000","last_fullsized":"2024-04-05T23:33:34.660781+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:15:42.243807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651490+0000","last_change":"2024-04-05T23:32:04.540096+0000","last_active":"2024-04-05T23:33:34.651490+0000","last_peered":"2024-04-05T23:33:34.651490+0000","last_clean":"2024-04-05T23:33:34.651490+0000","last_became_active":"2024-04-05T23:32:04.539764+0000","last_became_peered":"2024-04-05T23:32:04.539764+0000","last_unstale":"2024-04-05T23:33:34.651490+0000","last_undegraded":"2024-04-05T23:33:34.651490+0000","last_fullsized":"2024-04-05T23:33:34.651490+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:57:36.442393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.665139+0000","last_change":"2024-04-05T23:32:04.540839+0000","last_active":"2024-04-05T23:33:34.665139+0000","last_peered":"2024-04-05T23:33:34.665139+0000","last_clean":"2024-04-05T23:33:34.665139+0000","last_became_active":"2024-04-05T23:32:04.540511+0000","last_became_peered":"2024-04-05T23:32:04.540511+0000","last_unstale":"2024-04-05T23:33:34.665139+0000","last_undegraded":"2024-04-05T23:33:34.665139+0000","last_fullsized":"2024-04-05T23:33:34.665139+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:58:24.151326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.667960+0000","last_change":"2024-04-05T23:32:04.539222+0000","last_active":"2024-04-05T23:33:34.667960+0000","last_peered":"2024-04-05T23:33:34.667960+0000","last_clean":"2024-04-05T23:33:34.667960+0000","last_became_active":"2024-04-05T23:32:04.538906+0000","last_became_peered":"2024-04-05T23:32:04.538906+0000","last_unstale":"2024-04-05T23:33:34.667960+0000","last_undegraded":"2024-04-05T23:33:34.667960+0000","last_fullsized":"2024-04-05T23:33:34.667960+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:05:24.429153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666925+0000","last_change":"2024-04-05T23:32:04.544155+0000","last_active":"2024-04-05T23:33:34.666925+0000","last_peered":"2024-04-05T23:33:34.666925+0000","last_clean":"2024-04-05T23:33:34.666925+0000","last_became_active":"2024-04-05T23:32:04.543852+0000","last_became_peered":"2024-04-05T23:32:04.543852+0000","last_unstale":"2024-04-05T23:33:34.666925+0000","last_undegraded":"2024-04-05T23:33:34.666925+0000","last_fullsized":"2024-04-05T23:33:34.666925+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:18:34.353542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.672831+0000","last_change":"2024-04-05T23:32:04.533772+0000","last_active":"2024-04-05T23:33:34.672831+0000","last_peered":"2024-04-05T23:33:34.672831+0000","last_clean":"2024-04-05T23:33:34.672831+0000","last_became_active":"2024-04-05T23:32:04.533416+0000","last_became_peered":"2024-04-05T23:32:04.533416+0000","last_unstale":"2024-04-05T23:33:34.672831+0000","last_undegraded":"2024-04-05T23:33:34.672831+0000","last_fullsized":"2024-04-05T23:33:34.672831+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:36:55.072349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673455+0000","last_change":"2024-04-05T23:32:04.551902+0000","last_active":"2024-04-05T23:33:34.673455+0000","last_peered":"2024-04-05T23:33:34.673455+0000","last_clean":"2024-04-05T23:33:34.673455+0000","last_became_active":"2024-04-05T23:32:04.551595+0000","last_became_peered":"2024-04-05T23:32:04.551595+0000","last_unstale":"2024-04-05T23:33:34.673455+0000","last_undegraded":"2024-04-05T23:33:34.673455+0000","last_fullsized":"2024-04-05T23:33:34.673455+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:14:46.648789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.651142+0000","last_change":"2024-04-05T23:32:04.542967+0000","last_active":"2024-04-05T23:33:34.651142+0000","last_peered":"2024-04-05T23:33:34.651142+0000","last_clean":"2024-04-05T23:33:34.651142+0000","last_became_active":"2024-04-05T23:32:04.541767+0000","last_became_peered":"2024-04-05T23:32:04.541767+0000","last_unstale":"2024-04-05T23:33:34.651142+0000","last_undegraded":"2024-04-05T23:33:34.651142+0000","last_fullsized":"2024-04-05T23:33:34.651142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:27:46.150249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666772+0000","last_change":"2024-04-05T23:32:04.549512+0000","last_active":"2024-04-05T23:33:34.666772+0000","last_peered":"2024-04-05T23:33:34.666772+0000","last_clean":"2024-04-05T23:33:34.666772+0000","last_became_active":"2024-04-05T23:32:04.549184+0000","last_became_peered":"2024-04-05T23:32:04.549184+0000","last_unstale":"2024-04-05T23:33:34.666772+0000","last_undegraded":"2024-04-05T23:33:34.666772+0000","last_fullsized":"2024-04-05T23:33:34.666772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:48:24.186408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'14","reported_seq":115,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.941928+0000","last_change":"2024-04-05T23:32:04.005188+0000","last_active":"2024-04-05T23:33:34.941928+0000","last_peered":"2024-04-05T23:33:34.941928+0000","last_clean":"2024-04-05T23:33:34.941928+0000","last_became_active":"2024-04-05T23:32:04.004818+0000","last_became_peered":"2024-04-05T23:32:04.004818+0000","last_unstale":"2024-04-05T23:33:34.941928+0000","last_undegraded":"2024-04-05T23:33:34.941928+0000","last_fullsized":"2024-04-05T23:33:34.941928+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:49:41.146761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661564+0000","last_change":"2024-04-05T23:32:04.544881+0000","last_active":"2024-04-05T23:33:34.661564+0000","last_peered":"2024-04-05T23:33:34.661564+0000","last_clean":"2024-04-05T23:33:34.661564+0000","last_became_active":"2024-04-05T23:32:04.542149+0000","last_became_peered":"2024-04-05T23:32:04.542149+0000","last_unstale":"2024-04-05T23:33:34.661564+0000","last_undegraded":"2024-04-05T23:33:34.661564+0000","last_fullsized":"2024-04-05T23:33:34.661564+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:28:51.713053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.374976+0000","last_change":"2024-04-05T23:32:04.011450+0000","last_active":"2024-04-05T23:32:38.374976+0000","last_peered":"2024-04-05T23:32:38.374976+0000","last_clean":"2024-04-05T23:32:38.374976+0000","last_became_active":"2024-04-05T23:32:04.009607+0000","last_became_peered":"2024-04-05T23:32:04.009607+0000","last_unstale":"2024-04-05T23:32:38.374976+0000","last_undegraded":"2024-04-05T23:32:38.374976+0000","last_fullsized":"2024-04-05T23:32:38.374976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:16:45.483108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673025+0000","last_change":"2024-04-05T23:32:04.550370+0000","last_active":"2024-04-05T23:33:34.673025+0000","last_peered":"2024-04-05T23:33:34.673025+0000","last_clean":"2024-04-05T23:33:34.673025+0000","last_became_active":"2024-04-05T23:32:04.549980+0000","last_became_peered":"2024-04-05T23:32:04.549980+0000","last_unstale":"2024-04-05T23:33:34.673025+0000","last_undegraded":"2024-04-05T23:33:34.673025+0000","last_fullsized":"2024-04-05T23:33:34.673025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:44:30.704366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379630+0000","last_change":"2024-04-05T23:32:04.004169+0000","last_active":"2024-04-05T23:32:38.379630+0000","last_peered":"2024-04-05T23:32:38.379630+0000","last_clean":"2024-04-05T23:32:38.379630+0000","last_became_active":"2024-04-05T23:32:04.003646+0000","last_became_peered":"2024-04-05T23:32:04.003646+0000","last_unstale":"2024-04-05T23:32:38.379630+0000","last_undegraded":"2024-04-05T23:32:38.379630+0000","last_fullsized":"2024-04-05T23:32:38.379630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:57:02.781815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.662342+0000","last_change":"2024-04-05T23:32:04.542066+0000","last_active":"2024-04-05T23:33:34.662342+0000","last_peered":"2024-04-05T23:33:34.662342+0000","last_clean":"2024-04-05T23:33:34.662342+0000","last_became_active":"2024-04-05T23:32:04.540735+0000","last_became_peered":"2024-04-05T23:32:04.540735+0000","last_unstale":"2024-04-05T23:33:34.662342+0000","last_undegraded":"2024-04-05T23:33:34.662342+0000","last_fullsized":"2024-04-05T23:33:34.662342+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:02:03.595888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:41.225108+0000","last_change":"2024-04-05T23:32:38.396641+0000","last_active":"2024-04-05T23:32:41.225108+0000","last_peered":"2024-04-05T23:32:41.225108+0000","last_clean":"2024-04-05T23:32:41.225108+0000","last_became_active":"2024-04-05T23:32:38.379708+0000","last_became_peered":"2024-04-05T23:32:38.379708+0000","last_unstale":"2024-04-05T23:32:41.225108+0000","last_undegraded":"2024-04-05T23:32:41.225108+0000","last_fullsized":"2024-04-05T23:32:41.225108+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:23:00.370725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"82'15","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652724+0000","last_change":"2024-04-05T23:32:04.545780+0000","last_active":"2024-04-05T23:33:34.652724+0000","last_peered":"2024-04-05T23:33:34.652724+0000","last_clean":"2024-04-05T23:33:34.652724+0000","last_became_active":"2024-04-05T23:32:04.545279+0000","last_became_peered":"2024-04-05T23:32:04.545279+0000","last_unstale":"2024-04-05T23:33:34.652724+0000","last_undegraded":"2024-04-05T23:33:34.652724+0000","last_fullsized":"2024-04-05T23:33:34.652724+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:10.577280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":111,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.537192+0000","last_change":"2024-04-05T23:32:04.004189+0000","last_active":"2024-04-05T23:33:35.537192+0000","last_peered":"2024-04-05T23:33:35.537192+0000","last_clean":"2024-04-05T23:33:35.537192+0000","last_became_active":"2024-04-05T23:32:04.003841+0000","last_became_peered":"2024-04-05T23:32:04.003841+0000","last_unstale":"2024-04-05T23:33:35.537192+0000","last_undegraded":"2024-04-05T23:33:35.537192+0000","last_fullsized":"2024-04-05T23:33:35.537192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:33:26.898657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.649038+0000","last_change":"2024-04-05T23:32:04.529538+0000","last_active":"2024-04-05T23:33:34.649038+0000","last_peered":"2024-04-05T23:33:34.649038+0000","last_clean":"2024-04-05T23:33:34.649038+0000","last_became_active":"2024-04-05T23:32:04.529231+0000","last_became_peered":"2024-04-05T23:32:04.529231+0000","last_unstale":"2024-04-05T23:33:34.649038+0000","last_undegraded":"2024-04-05T23:33:34.649038+0000","last_fullsized":"2024-04-05T23:33:34.649038+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:49:15.608966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377155+0000","last_change":"2024-04-05T23:32:04.002763+0000","last_active":"2024-04-05T23:32:38.377155+0000","last_peered":"2024-04-05T23:32:38.377155+0000","last_clean":"2024-04-05T23:32:38.377155+0000","last_became_active":"2024-04-05T23:32:04.002410+0000","last_became_peered":"2024-04-05T23:32:04.002410+0000","last_unstale":"2024-04-05T23:32:38.377155+0000","last_undegraded":"2024-04-05T23:32:38.377155+0000","last_fullsized":"2024-04-05T23:32:38.377155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:15.193813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.667881+0000","last_change":"2024-04-05T23:32:38.461447+0000","last_active":"2024-04-05T23:33:34.667881+0000","last_peered":"2024-04-05T23:33:34.667881+0000","last_clean":"2024-04-05T23:33:34.667881+0000","last_became_active":"2024-04-05T23:32:38.460968+0000","last_became_peered":"2024-04-05T23:32:38.460968+0000","last_unstale":"2024-04-05T23:33:34.667881+0000","last_undegraded":"2024-04-05T23:33:34.667881+0000","last_fullsized":"2024-04-05T23:33:34.667881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:42:21.093881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377383+0000","last_change":"2024-04-05T23:32:04.001937+0000","last_active":"2024-04-05T23:32:38.377383+0000","last_peered":"2024-04-05T23:32:38.377383+0000","last_clean":"2024-04-05T23:32:38.377383+0000","last_became_active":"2024-04-05T23:32:04.001508+0000","last_became_peered":"2024-04-05T23:32:04.001508+0000","last_unstale":"2024-04-05T23:32:38.377383+0000","last_undegraded":"2024-04-05T23:32:38.377383+0000","last_fullsized":"2024-04-05T23:32:38.377383+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:09:20.682084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.668565+0000","last_change":"2024-04-05T23:32:04.531576+0000","last_active":"2024-04-05T23:33:34.668565+0000","last_peered":"2024-04-05T23:33:34.668565+0000","last_clean":"2024-04-05T23:33:34.668565+0000","last_became_active":"2024-04-05T23:32:04.531234+0000","last_became_peered":"2024-04-05T23:32:04.531234+0000","last_unstale":"2024-04-05T23:33:34.668565+0000","last_undegraded":"2024-04-05T23:33:34.668565+0000","last_fullsized":"2024-04-05T23:33:34.668565+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:21:57.769861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.376961+0000","last_change":"2024-04-05T23:32:04.001570+0000","last_active":"2024-04-05T23:32:38.376961+0000","last_peered":"2024-04-05T23:32:38.376961+0000","last_clean":"2024-04-05T23:32:38.376961+0000","last_became_active":"2024-04-05T23:32:04.001224+0000","last_became_peered":"2024-04-05T23:32:04.001224+0000","last_unstale":"2024-04-05T23:32:38.376961+0000","last_undegraded":"2024-04-05T23:32:38.376961+0000","last_fullsized":"2024-04-05T23:32:38.376961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:44:41.070205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"82'11","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.660070+0000","last_change":"2024-04-05T23:32:04.542029+0000","last_active":"2024-04-05T23:33:34.660070+0000","last_peered":"2024-04-05T23:33:34.660070+0000","last_clean":"2024-04-05T23:33:34.660070+0000","last_became_active":"2024-04-05T23:32:04.541684+0000","last_became_peered":"2024-04-05T23:32:04.541684+0000","last_unstale":"2024-04-05T23:33:34.660070+0000","last_undegraded":"2024-04-05T23:33:34.660070+0000","last_fullsized":"2024-04-05T23:33:34.660070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:07:06.456917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'6","reported_seq":193,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.536712+0000","last_change":"2024-04-05T23:32:04.004986+0000","last_active":"2024-04-05T23:33:35.536712+0000","last_peered":"2024-04-05T23:33:35.536712+0000","last_clean":"2024-04-05T23:33:35.536712+0000","last_became_active":"2024-04-05T23:32:04.004606+0000","last_became_peered":"2024-04-05T23:32:04.004606+0000","last_unstale":"2024-04-05T23:33:35.536712+0000","last_undegraded":"2024-04-05T23:33:35.536712+0000","last_fullsized":"2024-04-05T23:33:35.536712+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:23:41.060500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.654797+0000","last_change":"2024-04-05T23:32:04.541334+0000","last_active":"2024-04-05T23:33:34.654797+0000","last_peered":"2024-04-05T23:33:34.654797+0000","last_clean":"2024-04-05T23:33:34.654797+0000","last_became_active":"2024-04-05T23:32:04.540967+0000","last_became_peered":"2024-04-05T23:32:04.540967+0000","last_unstale":"2024-04-05T23:33:34.654797+0000","last_undegraded":"2024-04-05T23:33:34.654797+0000","last_fullsized":"2024-04-05T23:33:34.654797+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:16:33.045172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.466958+0000","last_change":"2024-04-05T23:32:38.466417+0000","last_active":"2024-04-05T23:32:38.466958+0000","last_peered":"2024-04-05T23:32:38.466958+0000","last_clean":"2024-04-05T23:32:38.466958+0000","last_became_active":"2024-04-05T23:32:38.462293+0000","last_became_peered":"2024-04-05T23:32:38.462293+0000","last_unstale":"2024-04-05T23:32:38.466958+0000","last_undegraded":"2024-04-05T23:32:38.466958+0000","last_fullsized":"2024-04-05T23:32:38.466958+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:03:31.118636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"82'9","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669698+0000","last_change":"2024-04-05T23:32:38.385818+0000","last_active":"2024-04-05T23:33:34.669698+0000","last_peered":"2024-04-05T23:33:34.669698+0000","last_clean":"2024-04-05T23:33:34.669698+0000","last_became_active":"2024-04-05T23:32:38.385423+0000","last_became_peered":"2024-04-05T23:32:38.385423+0000","last_unstale":"2024-04-05T23:33:34.669698+0000","last_undegraded":"2024-04-05T23:33:34.669698+0000","last_fullsized":"2024-04-05T23:33:34.669698+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:07:11.374783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377265+0000","last_change":"2024-04-05T23:32:04.000697+0000","last_active":"2024-04-05T23:32:38.377265+0000","last_peered":"2024-04-05T23:32:38.377265+0000","last_clean":"2024-04-05T23:32:38.377265+0000","last_became_active":"2024-04-05T23:32:04.000377+0000","last_became_peered":"2024-04-05T23:32:04.000377+0000","last_unstale":"2024-04-05T23:32:38.377265+0000","last_undegraded":"2024-04-05T23:32:38.377265+0000","last_fullsized":"2024-04-05T23:32:38.377265+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:55:11.227637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.662559+0000","last_change":"2024-04-05T23:32:38.383367+0000","last_active":"2024-04-05T23:33:34.662559+0000","last_peered":"2024-04-05T23:33:34.662559+0000","last_clean":"2024-04-05T23:33:34.662559+0000","last_became_active":"2024-04-05T23:32:38.383002+0000","last_became_peered":"2024-04-05T23:32:38.383002+0000","last_unstale":"2024-04-05T23:33:34.662559+0000","last_undegraded":"2024-04-05T23:33:34.662559+0000","last_fullsized":"2024-04-05T23:33:34.662559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:30:59.947387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'36","reported_seq":234,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.547756+0000","last_change":"2024-04-05T23:32:04.007322+0000","last_active":"2024-04-05T23:33:35.547756+0000","last_peered":"2024-04-05T23:33:35.547756+0000","last_clean":"2024-04-05T23:33:35.547756+0000","last_became_active":"2024-04-05T23:32:04.006971+0000","last_became_peered":"2024-04-05T23:32:04.006971+0000","last_unstale":"2024-04-05T23:33:35.547756+0000","last_undegraded":"2024-04-05T23:33:35.547756+0000","last_fullsized":"2024-04-05T23:33:35.547756+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:04:38.413482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65485,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":183,"num_read_kb":155,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661408+0000","last_change":"2024-04-05T23:32:04.535342+0000","last_active":"2024-04-05T23:33:34.661408+0000","last_peered":"2024-04-05T23:33:34.661408+0000","last_clean":"2024-04-05T23:33:34.661408+0000","last_became_active":"2024-04-05T23:32:04.535053+0000","last_became_peered":"2024-04-05T23:32:04.535053+0000","last_unstale":"2024-04-05T23:33:34.661408+0000","last_undegraded":"2024-04-05T23:33:34.661408+0000","last_fullsized":"2024-04-05T23:33:34.661408+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:19:20.683836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:41.225091+0000","last_change":"2024-04-05T23:32:38.391418+0000","last_active":"2024-04-05T23:32:41.225091+0000","last_peered":"2024-04-05T23:32:41.225091+0000","last_clean":"2024-04-05T23:32:41.225091+0000","last_became_active":"2024-04-05T23:32:38.379643+0000","last_became_peered":"2024-04-05T23:32:38.379643+0000","last_unstale":"2024-04-05T23:32:41.225091+0000","last_undegraded":"2024-04-05T23:32:41.225091+0000","last_fullsized":"2024-04-05T23:32:41.225091+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:37:12.343153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.668971+0000","last_change":"2024-04-05T23:32:04.536455+0000","last_active":"2024-04-05T23:33:34.668971+0000","last_peered":"2024-04-05T23:33:34.668971+0000","last_clean":"2024-04-05T23:33:34.668971+0000","last_became_active":"2024-04-05T23:32:04.533994+0000","last_became_peered":"2024-04-05T23:32:04.533994+0000","last_unstale":"2024-04-05T23:33:34.668971+0000","last_undegraded":"2024-04-05T23:33:34.668971+0000","last_fullsized":"2024-04-05T23:33:34.668971+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:38:18.952823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.376958+0000","last_change":"2024-04-05T23:32:04.000708+0000","last_active":"2024-04-05T23:32:38.376958+0000","last_peered":"2024-04-05T23:32:38.376958+0000","last_clean":"2024-04-05T23:32:38.376958+0000","last_became_active":"2024-04-05T23:32:04.000354+0000","last_became_peered":"2024-04-05T23:32:04.000354+0000","last_unstale":"2024-04-05T23:32:38.376958+0000","last_undegraded":"2024-04-05T23:32:38.376958+0000","last_fullsized":"2024-04-05T23:32:38.376958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:05:34.291607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"82'15","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666642+0000","last_change":"2024-04-05T23:32:04.532146+0000","last_active":"2024-04-05T23:33:34.666642+0000","last_peered":"2024-04-05T23:33:34.666642+0000","last_clean":"2024-04-05T23:33:34.666642+0000","last_became_active":"2024-04-05T23:32:04.531735+0000","last_became_peered":"2024-04-05T23:32:04.531735+0000","last_unstale":"2024-04-05T23:33:34.666642+0000","last_undegraded":"2024-04-05T23:33:34.666642+0000","last_fullsized":"2024-04-05T23:33:34.666642+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:52:44.898871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'8","reported_seq":114,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.540783+0000","last_change":"2024-04-05T23:32:04.000144+0000","last_active":"2024-04-05T23:33:35.540783+0000","last_peered":"2024-04-05T23:33:35.540783+0000","last_clean":"2024-04-05T23:33:35.540783+0000","last_became_active":"2024-04-05T23:32:03.999712+0000","last_became_peered":"2024-04-05T23:32:03.999712+0000","last_unstale":"2024-04-05T23:33:35.540783+0000","last_undegraded":"2024-04-05T23:33:35.540783+0000","last_fullsized":"2024-04-05T23:33:35.540783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:48:43.775018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1632,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":92,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.658416+0000","last_change":"2024-04-05T23:32:04.537623+0000","last_active":"2024-04-05T23:33:34.658416+0000","last_peered":"2024-04-05T23:33:34.658416+0000","last_clean":"2024-04-05T23:33:34.658416+0000","last_became_active":"2024-04-05T23:32:04.537399+0000","last_became_peered":"2024-04-05T23:32:04.537399+0000","last_unstale":"2024-04-05T23:33:34.658416+0000","last_undegraded":"2024-04-05T23:33:34.658416+0000","last_fullsized":"2024-04-05T23:33:34.658416+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:01:01.643133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378658+0000","last_change":"2024-04-05T23:32:04.006806+0000","last_active":"2024-04-05T23:32:38.378658+0000","last_peered":"2024-04-05T23:32:38.378658+0000","last_clean":"2024-04-05T23:32:38.378658+0000","last_became_active":"2024-04-05T23:32:04.006397+0000","last_became_peered":"2024-04-05T23:32:04.006397+0000","last_unstale":"2024-04-05T23:32:38.378658+0000","last_undegraded":"2024-04-05T23:32:38.378658+0000","last_fullsized":"2024-04-05T23:32:38.378658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:42:01.758863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.653257+0000","last_change":"2024-04-05T23:32:04.545952+0000","last_active":"2024-04-05T23:33:34.653257+0000","last_peered":"2024-04-05T23:33:34.653257+0000","last_clean":"2024-04-05T23:33:34.653257+0000","last_became_active":"2024-04-05T23:32:04.545528+0000","last_became_peered":"2024-04-05T23:32:04.545528+0000","last_unstale":"2024-04-05T23:33:34.653257+0000","last_undegraded":"2024-04-05T23:33:34.653257+0000","last_fullsized":"2024-04-05T23:33:34.653257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:33:06.074173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:33:32.204346+0000","last_change":"2024-04-05T23:32:03.999841+0000","last_active":"2024-04-05T23:33:32.204346+0000","last_peered":"2024-04-05T23:33:32.204346+0000","last_clean":"2024-04-05T23:33:32.204346+0000","last_became_active":"2024-04-05T23:32:03.999418+0000","last_became_peered":"2024-04-05T23:32:03.999418+0000","last_unstale":"2024-04-05T23:33:32.204346+0000","last_undegraded":"2024-04-05T23:33:32.204346+0000","last_fullsized":"2024-04-05T23:33:32.204346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:00:41.452167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.650643+0000","last_change":"2024-04-05T23:32:04.530012+0000","last_active":"2024-04-05T23:33:34.650643+0000","last_peered":"2024-04-05T23:33:34.650643+0000","last_clean":"2024-04-05T23:33:34.650643+0000","last_became_active":"2024-04-05T23:32:04.529628+0000","last_became_peered":"2024-04-05T23:32:04.529628+0000","last_unstale":"2024-04-05T23:33:34.650643+0000","last_undegraded":"2024-04-05T23:33:34.650643+0000","last_fullsized":"2024-04-05T23:33:34.650643+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:40:22.036170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379485+0000","last_change":"2024-04-05T23:32:04.006157+0000","last_active":"2024-04-05T23:32:38.379485+0000","last_peered":"2024-04-05T23:32:38.379485+0000","last_clean":"2024-04-05T23:32:38.379485+0000","last_became_active":"2024-04-05T23:32:04.005853+0000","last_became_peered":"2024-04-05T23:32:04.005853+0000","last_unstale":"2024-04-05T23:32:38.379485+0000","last_undegraded":"2024-04-05T23:32:38.379485+0000","last_fullsized":"2024-04-05T23:32:38.379485+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:48:39.849306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.658363+0000","last_change":"2024-04-05T23:32:04.547367+0000","last_active":"2024-04-05T23:33:34.658363+0000","last_peered":"2024-04-05T23:33:34.658363+0000","last_clean":"2024-04-05T23:33:34.658363+0000","last_became_active":"2024-04-05T23:32:04.547040+0000","last_became_peered":"2024-04-05T23:32:04.547040+0000","last_unstale":"2024-04-05T23:33:34.658363+0000","last_undegraded":"2024-04-05T23:33:34.658363+0000","last_fullsized":"2024-04-05T23:33:34.658363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:41:41.998984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'7","reported_seq":109,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.559448+0000","last_change":"2024-04-05T23:32:04.009851+0000","last_active":"2024-04-05T23:33:35.559448+0000","last_peered":"2024-04-05T23:33:35.559448+0000","last_clean":"2024-04-05T23:33:35.559448+0000","last_became_active":"2024-04-05T23:32:04.009460+0000","last_became_peered":"2024-04-05T23:32:04.009460+0000","last_unstale":"2024-04-05T23:33:35.559448+0000","last_undegraded":"2024-04-05T23:33:35.559448+0000","last_fullsized":"2024-04-05T23:33:35.559448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:39:41.000439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1326,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652783+0000","last_change":"2024-04-05T23:32:04.549621+0000","last_active":"2024-04-05T23:33:34.652783+0000","last_peered":"2024-04-05T23:33:34.652783+0000","last_clean":"2024-04-05T23:33:34.652783+0000","last_became_active":"2024-04-05T23:32:04.549325+0000","last_became_peered":"2024-04-05T23:32:04.549325+0000","last_unstale":"2024-04-05T23:33:34.652783+0000","last_undegraded":"2024-04-05T23:33:34.652783+0000","last_fullsized":"2024-04-05T23:33:34.652783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:11:07.101463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652570+0000","last_change":"2024-04-05T23:32:04.546768+0000","last_active":"2024-04-05T23:33:34.652570+0000","last_peered":"2024-04-05T23:33:34.652570+0000","last_clean":"2024-04-05T23:33:34.652570+0000","last_became_active":"2024-04-05T23:32:04.546471+0000","last_became_peered":"2024-04-05T23:32:04.546471+0000","last_unstale":"2024-04-05T23:33:34.652570+0000","last_undegraded":"2024-04-05T23:33:34.652570+0000","last_fullsized":"2024-04-05T23:33:34.652570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:48:22.728976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'6","reported_seq":109,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.621543+0000","last_change":"2024-04-05T23:32:04.007093+0000","last_active":"2024-04-05T23:33:34.621543+0000","last_peered":"2024-04-05T23:33:34.621543+0000","last_clean":"2024-04-05T23:33:34.621543+0000","last_became_active":"2024-04-05T23:32:04.006836+0000","last_became_peered":"2024-04-05T23:32:04.006836+0000","last_unstale":"2024-04-05T23:33:34.621543+0000","last_undegraded":"2024-04-05T23:33:34.621543+0000","last_fullsized":"2024-04-05T23:33:34.621543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:20:42.489368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669973+0000","last_change":"2024-04-05T23:32:04.538204+0000","last_active":"2024-04-05T23:33:34.669973+0000","last_peered":"2024-04-05T23:33:34.669973+0000","last_clean":"2024-04-05T23:33:34.669973+0000","last_became_active":"2024-04-05T23:32:04.535254+0000","last_became_peered":"2024-04-05T23:32:04.535254+0000","last_unstale":"2024-04-05T23:33:34.669973+0000","last_undegraded":"2024-04-05T23:33:34.669973+0000","last_fullsized":"2024-04-05T23:33:34.669973+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:06:01.059339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.460974+0000","last_change":"2024-04-05T23:32:03.997704+0000","last_active":"2024-04-05T23:32:38.460974+0000","last_peered":"2024-04-05T23:32:38.460974+0000","last_clean":"2024-04-05T23:32:38.460974+0000","last_became_active":"2024-04-05T23:32:03.997275+0000","last_became_peered":"2024-04-05T23:32:03.997275+0000","last_unstale":"2024-04-05T23:32:38.460974+0000","last_undegraded":"2024-04-05T23:32:38.460974+0000","last_fullsized":"2024-04-05T23:32:38.460974+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:58.355104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":337,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378621+0000","last_change":"2024-04-05T23:31:10.395041+0000","last_active":"2024-04-05T23:32:38.378621+0000","last_peered":"2024-04-05T23:32:38.378621+0000","last_clean":"2024-04-05T23:32:38.378621+0000","last_became_active":"2024-04-05T23:31:10.394479+0000","last_became_peered":"2024-04-05T23:31:10.394479+0000","last_unstale":"2024-04-05T23:32:38.378621+0000","last_undegraded":"2024-04-05T23:32:38.378621+0000","last_fullsized":"2024-04-05T23:32:38.378621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:28:33.945697+0000","last_clean_scrub_stamp":"2024-04-05T23:28:33.945697+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:54:10.683526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'5","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.649757+0000","last_change":"2024-04-05T23:32:38.385626+0000","last_active":"2024-04-05T23:33:34.649757+0000","last_peered":"2024-04-05T23:33:34.649757+0000","last_clean":"2024-04-05T23:33:34.649757+0000","last_became_active":"2024-04-05T23:32:38.384654+0000","last_became_peered":"2024-04-05T23:32:38.384654+0000","last_unstale":"2024-04-05T23:33:34.649757+0000","last_undegraded":"2024-04-05T23:33:34.649757+0000","last_fullsized":"2024-04-05T23:33:34.649757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:55:42.833005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378826+0000","last_change":"2024-04-05T23:32:04.005978+0000","last_active":"2024-04-05T23:32:38.378826+0000","last_peered":"2024-04-05T23:32:38.378826+0000","last_clean":"2024-04-05T23:32:38.378826+0000","last_became_active":"2024-04-05T23:32:04.005638+0000","last_became_peered":"2024-04-05T23:32:04.005638+0000","last_unstale":"2024-04-05T23:32:38.378826+0000","last_undegraded":"2024-04-05T23:32:38.378826+0000","last_fullsized":"2024-04-05T23:32:38.378826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:39:35.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379513+0000","last_change":"2024-04-05T23:32:04.006727+0000","last_active":"2024-04-05T23:32:38.379513+0000","last_peered":"2024-04-05T23:32:38.379513+0000","last_clean":"2024-04-05T23:32:38.379513+0000","last_became_active":"2024-04-05T23:32:04.006414+0000","last_became_peered":"2024-04-05T23:32:04.006414+0000","last_unstale":"2024-04-05T23:32:38.379513+0000","last_undegraded":"2024-04-05T23:32:38.379513+0000","last_fullsized":"2024-04-05T23:32:38.379513+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:41:28.661118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.664992+0000","last_change":"2024-04-05T23:32:04.536893+0000","last_active":"2024-04-05T23:33:34.664992+0000","last_peered":"2024-04-05T23:33:34.664992+0000","last_clean":"2024-04-05T23:33:34.664992+0000","last_became_active":"2024-04-05T23:32:04.536629+0000","last_became_peered":"2024-04-05T23:32:04.536629+0000","last_unstale":"2024-04-05T23:33:34.664992+0000","last_undegraded":"2024-04-05T23:33:34.664992+0000","last_fullsized":"2024-04-05T23:33:34.664992+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:27:58.279768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'44","reported_seq":82,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.876599+0000","last_change":"2024-04-05T23:32:04.010202+0000","last_active":"2024-04-05T23:33:34.876599+0000","last_peered":"2024-04-05T23:33:34.876599+0000","last_clean":"2024-04-05T23:33:34.876599+0000","last_became_active":"2024-04-05T23:32:04.009889+0000","last_became_peered":"2024-04-05T23:32:04.009889+0000","last_unstale":"2024-04-05T23:33:34.876599+0000","last_undegraded":"2024-04-05T23:33:34.876599+0000","last_fullsized":"2024-04-05T23:33:34.876599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:12:13.121449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.667696+0000","last_change":"2024-04-05T23:32:04.533576+0000","last_active":"2024-04-05T23:33:34.667696+0000","last_peered":"2024-04-05T23:33:34.667696+0000","last_clean":"2024-04-05T23:33:34.667696+0000","last_became_active":"2024-04-05T23:32:04.533066+0000","last_became_peered":"2024-04-05T23:32:04.533066+0000","last_unstale":"2024-04-05T23:33:34.667696+0000","last_undegraded":"2024-04-05T23:33:34.667696+0000","last_fullsized":"2024-04-05T23:33:34.667696+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:01:44.019210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.461143+0000","last_change":"2024-04-05T23:32:04.005096+0000","last_active":"2024-04-05T23:32:38.461143+0000","last_peered":"2024-04-05T23:32:38.461143+0000","last_clean":"2024-04-05T23:32:38.461143+0000","last_became_active":"2024-04-05T23:32:04.004702+0000","last_became_peered":"2024-04-05T23:32:04.004702+0000","last_unstale":"2024-04-05T23:32:38.461143+0000","last_undegraded":"2024-04-05T23:32:38.461143+0000","last_fullsized":"2024-04-05T23:32:38.461143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:33:09.934039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670944+0000","last_change":"2024-04-05T23:32:04.535623+0000","last_active":"2024-04-05T23:33:34.670944+0000","last_peered":"2024-04-05T23:33:34.670944+0000","last_clean":"2024-04-05T23:33:34.670944+0000","last_became_active":"2024-04-05T23:32:04.535223+0000","last_became_peered":"2024-04-05T23:32:04.535223+0000","last_unstale":"2024-04-05T23:33:34.670944+0000","last_undegraded":"2024-04-05T23:33:34.670944+0000","last_fullsized":"2024-04-05T23:33:34.670944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:24:43.543119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377010+0000","last_change":"2024-04-05T23:32:04.004097+0000","last_active":"2024-04-05T23:32:38.377010+0000","last_peered":"2024-04-05T23:32:38.377010+0000","last_clean":"2024-04-05T23:32:38.377010+0000","last_became_active":"2024-04-05T23:32:04.003713+0000","last_became_peered":"2024-04-05T23:32:04.003713+0000","last_unstale":"2024-04-05T23:32:38.377010+0000","last_undegraded":"2024-04-05T23:32:38.377010+0000","last_fullsized":"2024-04-05T23:32:38.377010+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:47:43.042021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"82'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.627575+0000","last_change":"2024-04-05T23:32:04.538843+0000","last_active":"2024-04-05T23:33:34.627575+0000","last_peered":"2024-04-05T23:33:34.627575+0000","last_clean":"2024-04-05T23:33:34.627575+0000","last_became_active":"2024-04-05T23:32:04.538472+0000","last_became_peered":"2024-04-05T23:32:04.538472+0000","last_unstale":"2024-04-05T23:33:34.627575+0000","last_undegraded":"2024-04-05T23:33:34.627575+0000","last_fullsized":"2024-04-05T23:33:34.627575+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:40:40.863815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.373470+0000","last_change":"2024-04-05T23:32:04.013064+0000","last_active":"2024-04-05T23:32:38.373470+0000","last_peered":"2024-04-05T23:32:38.373470+0000","last_clean":"2024-04-05T23:32:38.373470+0000","last_became_active":"2024-04-05T23:32:04.012751+0000","last_became_peered":"2024-04-05T23:32:04.012751+0000","last_unstale":"2024-04-05T23:32:38.373470+0000","last_undegraded":"2024-04-05T23:32:38.373470+0000","last_fullsized":"2024-04-05T23:32:38.373470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:06:10.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.673173+0000","last_change":"2024-04-05T23:32:04.529014+0000","last_active":"2024-04-05T23:33:34.673173+0000","last_peered":"2024-04-05T23:33:34.673173+0000","last_clean":"2024-04-05T23:33:34.673173+0000","last_became_active":"2024-04-05T23:32:04.528622+0000","last_became_peered":"2024-04-05T23:32:04.528622+0000","last_unstale":"2024-04-05T23:33:34.673173+0000","last_undegraded":"2024-04-05T23:33:34.673173+0000","last_fullsized":"2024-04-05T23:33:34.673173+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:46:26.390934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.377465+0000","last_change":"2024-04-05T23:32:04.002651+0000","last_active":"2024-04-05T23:32:38.377465+0000","last_peered":"2024-04-05T23:32:38.377465+0000","last_clean":"2024-04-05T23:32:38.377465+0000","last_became_active":"2024-04-05T23:32:04.002346+0000","last_became_peered":"2024-04-05T23:32:04.002346+0000","last_unstale":"2024-04-05T23:32:38.377465+0000","last_undegraded":"2024-04-05T23:32:38.377465+0000","last_fullsized":"2024-04-05T23:32:38.377465+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T11:26:48.176254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669156+0000","last_change":"2024-04-05T23:32:04.538785+0000","last_active":"2024-04-05T23:33:34.669156+0000","last_peered":"2024-04-05T23:33:34.669156+0000","last_clean":"2024-04-05T23:33:34.669156+0000","last_became_active":"2024-04-05T23:32:04.538463+0000","last_became_peered":"2024-04-05T23:32:04.538463+0000","last_unstale":"2024-04-05T23:33:34.669156+0000","last_undegraded":"2024-04-05T23:33:34.669156+0000","last_fullsized":"2024-04-05T23:33:34.669156+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:44:38.941857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.379009+0000","last_change":"2024-04-05T23:32:04.003760+0000","last_active":"2024-04-05T23:32:38.379009+0000","last_peered":"2024-04-05T23:32:38.379009+0000","last_clean":"2024-04-05T23:32:38.379009+0000","last_became_active":"2024-04-05T23:32:04.003389+0000","last_became_peered":"2024-04-05T23:32:04.003389+0000","last_unstale":"2024-04-05T23:32:38.379009+0000","last_undegraded":"2024-04-05T23:32:38.379009+0000","last_fullsized":"2024-04-05T23:32:38.379009+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:42:00.353651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"82'5","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661087+0000","last_change":"2024-04-05T23:32:04.540051+0000","last_active":"2024-04-05T23:33:34.661087+0000","last_peered":"2024-04-05T23:33:34.661087+0000","last_clean":"2024-04-05T23:33:34.661087+0000","last_became_active":"2024-04-05T23:32:04.539736+0000","last_became_peered":"2024-04-05T23:32:04.539736+0000","last_unstale":"2024-04-05T23:33:34.661087+0000","last_undegraded":"2024-04-05T23:33:34.661087+0000","last_fullsized":"2024-04-05T23:33:34.661087+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:54:14.086904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'7","reported_seq":110,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:35.559133+0000","last_change":"2024-04-05T23:32:04.006492+0000","last_active":"2024-04-05T23:33:35.559133+0000","last_peered":"2024-04-05T23:33:35.559133+0000","last_clean":"2024-04-05T23:33:35.559133+0000","last_became_active":"2024-04-05T23:32:04.006154+0000","last_became_peered":"2024-04-05T23:32:04.006154+0000","last_unstale":"2024-04-05T23:33:35.559133+0000","last_undegraded":"2024-04-05T23:33:35.559133+0000","last_fullsized":"2024-04-05T23:33:35.559133+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:53:23.431400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1908,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661003+0000","last_change":"2024-04-05T23:32:04.540757+0000","last_active":"2024-04-05T23:33:34.661003+0000","last_peered":"2024-04-05T23:33:34.661003+0000","last_clean":"2024-04-05T23:33:34.661003+0000","last_became_active":"2024-04-05T23:32:04.540359+0000","last_became_peered":"2024-04-05T23:32:04.540359+0000","last_unstale":"2024-04-05T23:33:34.661003+0000","last_undegraded":"2024-04-05T23:33:34.661003+0000","last_fullsized":"2024-04-05T23:33:34.661003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:43:52.533489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'41","reported_seq":64,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.873377+0000","last_change":"2024-04-05T23:32:04.002736+0000","last_active":"2024-04-05T23:33:34.873377+0000","last_peered":"2024-04-05T23:33:34.873377+0000","last_clean":"2024-04-05T23:33:34.873377+0000","last_became_active":"2024-04-05T23:32:04.002277+0000","last_became_peered":"2024-04-05T23:32:04.002277+0000","last_unstale":"2024-04-05T23:33:34.873377+0000","last_undegraded":"2024-04-05T23:33:34.873377+0000","last_fullsized":"2024-04-05T23:33:34.873377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:56:39.001732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670914+0000","last_change":"2024-04-05T23:32:04.537076+0000","last_active":"2024-04-05T23:33:34.670914+0000","last_peered":"2024-04-05T23:33:34.670914+0000","last_clean":"2024-04-05T23:33:34.670914+0000","last_became_active":"2024-04-05T23:32:04.536707+0000","last_became_peered":"2024-04-05T23:32:04.536707+0000","last_unstale":"2024-04-05T23:33:34.670914+0000","last_undegraded":"2024-04-05T23:33:34.670914+0000","last_fullsized":"2024-04-05T23:33:34.670914+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:51:09.395408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-05T23:32:38.378236+0000","last_change":"2024-04-05T23:32:04.006122+0000","last_active":"2024-04-05T23:32:38.378236+0000","last_peered":"2024-04-05T23:32:38.378236+0000","last_clean":"2024-04-05T23:32:38.378236+0000","last_became_active":"2024-04-05T23:32:04.005683+0000","last_became_peered":"2024-04-05T23:32:04.005683+0000","last_unstale":"2024-04-05T23:32:38.378236+0000","last_undegraded":"2024-04-05T23:32:38.378236+0000","last_fullsized":"2024-04-05T23:32:38.378236+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:02.975311+0000","last_clean_scrub_stamp":"2024-04-05T23:32:02.975311+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:21:40.922614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.661469+0000","last_change":"2024-04-05T23:32:04.533321+0000","last_active":"2024-04-05T23:33:34.661469+0000","last_peered":"2024-04-05T23:33:34.661469+0000","last_clean":"2024-04-05T23:33:34.661469+0000","last_became_active":"2024-04-05T23:32:04.532949+0000","last_became_peered":"2024-04-05T23:32:04.532949+0000","last_unstale":"2024-04-05T23:33:34.661469+0000","last_undegraded":"2024-04-05T23:33:34.661469+0000","last_fullsized":"2024-04-05T23:33:34.661469+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T07:45:31.500132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'9","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.648745+0000","last_change":"2024-04-05T23:32:04.536508+0000","last_active":"2024-04-05T23:33:34.648745+0000","last_peered":"2024-04-05T23:33:34.648745+0000","last_clean":"2024-04-05T23:33:34.648745+0000","last_became_active":"2024-04-05T23:32:04.536142+0000","last_became_peered":"2024-04-05T23:32:04.536142+0000","last_unstale":"2024-04-05T23:33:34.648745+0000","last_undegraded":"2024-04-05T23:33:34.648745+0000","last_fullsized":"2024-04-05T23:33:34.648745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:18:28.497807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"82'12","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652638+0000","last_change":"2024-04-05T23:32:04.545511+0000","last_active":"2024-04-05T23:33:34.652638+0000","last_peered":"2024-04-05T23:33:34.652638+0000","last_clean":"2024-04-05T23:33:34.652638+0000","last_became_active":"2024-04-05T23:32:04.545191+0000","last_became_peered":"2024-04-05T23:32:04.545191+0000","last_unstale":"2024-04-05T23:33:34.652638+0000","last_undegraded":"2024-04-05T23:33:34.652638+0000","last_fullsized":"2024-04-05T23:33:34.652638+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T00:17:06.431429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.668751+0000","last_change":"2024-04-05T23:32:04.549673+0000","last_active":"2024-04-05T23:33:34.668751+0000","last_peered":"2024-04-05T23:33:34.668751+0000","last_clean":"2024-04-05T23:33:34.668751+0000","last_became_active":"2024-04-05T23:32:04.549391+0000","last_became_peered":"2024-04-05T23:32:04.549391+0000","last_unstale":"2024-04-05T23:33:34.668751+0000","last_undegraded":"2024-04-05T23:33:34.668751+0000","last_fullsized":"2024-04-05T23:33:34.668751+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:30:22.689107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"82'7","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.666260+0000","last_change":"2024-04-05T23:32:04.535735+0000","last_active":"2024-04-05T23:33:34.666260+0000","last_peered":"2024-04-05T23:33:34.666260+0000","last_clean":"2024-04-05T23:33:34.666260+0000","last_became_active":"2024-04-05T23:32:04.534461+0000","last_became_peered":"2024-04-05T23:32:04.534461+0000","last_unstale":"2024-04-05T23:33:34.666260+0000","last_undegraded":"2024-04-05T23:33:34.666260+0000","last_fullsized":"2024-04-05T23:33:34.666260+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:37:41.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"82'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.671261+0000","last_change":"2024-04-05T23:32:04.532340+0000","last_active":"2024-04-05T23:33:34.671261+0000","last_peered":"2024-04-05T23:33:34.671261+0000","last_clean":"2024-04-05T23:33:34.671261+0000","last_became_active":"2024-04-05T23:32:04.530053+0000","last_became_peered":"2024-04-05T23:32:04.530053+0000","last_unstale":"2024-04-05T23:33:34.671261+0000","last_undegraded":"2024-04-05T23:33:34.671261+0000","last_fullsized":"2024-04-05T23:33:34.671261+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:16:11.168366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.646897+0000","last_change":"2024-04-05T23:32:04.529972+0000","last_active":"2024-04-05T23:33:34.646897+0000","last_peered":"2024-04-05T23:33:34.646897+0000","last_clean":"2024-04-05T23:33:34.646897+0000","last_became_active":"2024-04-05T23:32:04.529551+0000","last_became_peered":"2024-04-05T23:32:04.529551+0000","last_unstale":"2024-04-05T23:33:34.646897+0000","last_undegraded":"2024-04-05T23:33:34.646897+0000","last_fullsized":"2024-04-05T23:33:34.646897+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T01:57:13.764971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"82'15","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.648772+0000","last_change":"2024-04-05T23:32:04.529166+0000","last_active":"2024-04-05T23:33:34.648772+0000","last_peered":"2024-04-05T23:33:34.648772+0000","last_clean":"2024-04-05T23:33:34.648772+0000","last_became_active":"2024-04-05T23:32:04.528785+0000","last_became_peered":"2024-04-05T23:32:04.528785+0000","last_unstale":"2024-04-05T23:33:34.648772+0000","last_undegraded":"2024-04-05T23:33:34.648772+0000","last_fullsized":"2024-04-05T23:33:34.648772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:29:43.964265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"82'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.660708+0000","last_change":"2024-04-05T23:32:04.540631+0000","last_active":"2024-04-05T23:33:34.660708+0000","last_peered":"2024-04-05T23:33:34.660708+0000","last_clean":"2024-04-05T23:33:34.660708+0000","last_became_active":"2024-04-05T23:32:04.540327+0000","last_became_peered":"2024-04-05T23:32:04.540327+0000","last_unstale":"2024-04-05T23:33:34.660708+0000","last_undegraded":"2024-04-05T23:33:34.660708+0000","last_fullsized":"2024-04-05T23:33:34.660708+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:22:48.825324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"82'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.656475+0000","last_change":"2024-04-05T23:32:04.544845+0000","last_active":"2024-04-05T23:33:34.656475+0000","last_peered":"2024-04-05T23:33:34.656475+0000","last_clean":"2024-04-05T23:33:34.656475+0000","last_became_active":"2024-04-05T23:32:04.544472+0000","last_became_peered":"2024-04-05T23:32:04.544472+0000","last_unstale":"2024-04-05T23:33:34.656475+0000","last_undegraded":"2024-04-05T23:33:34.656475+0000","last_fullsized":"2024-04-05T23:33:34.656475+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T05:34:09.529644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652655+0000","last_change":"2024-04-05T23:32:38.382577+0000","last_active":"2024-04-05T23:33:34.652655+0000","last_peered":"2024-04-05T23:33:34.652655+0000","last_clean":"2024-04-05T23:33:34.652655+0000","last_became_active":"2024-04-05T23:32:38.380286+0000","last_became_peered":"2024-04-05T23:32:38.380286+0000","last_unstale":"2024-04-05T23:33:34.652655+0000","last_undegraded":"2024-04-05T23:33:34.652655+0000","last_fullsized":"2024-04-05T23:33:34.652655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T23:42:52.270935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"82'10","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.670550+0000","last_change":"2024-04-05T23:32:04.534631+0000","last_active":"2024-04-05T23:33:34.670550+0000","last_peered":"2024-04-05T23:33:34.670550+0000","last_clean":"2024-04-05T23:33:34.670550+0000","last_became_active":"2024-04-05T23:32:04.534330+0000","last_became_peered":"2024-04-05T23:32:04.534330+0000","last_unstale":"2024-04-05T23:33:34.670550+0000","last_undegraded":"2024-04-05T23:33:34.670550+0000","last_fullsized":"2024-04-05T23:33:34.670550+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T03:20:47.592174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"82'11","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.669338+0000","last_change":"2024-04-05T23:32:04.531459+0000","last_active":"2024-04-05T23:33:34.669338+0000","last_peered":"2024-04-05T23:33:34.669338+0000","last_clean":"2024-04-05T23:33:34.669338+0000","last_became_active":"2024-04-05T23:32:04.530828+0000","last_became_peered":"2024-04-05T23:32:04.530828+0000","last_unstale":"2024-04-05T23:33:34.669338+0000","last_undegraded":"2024-04-05T23:33:34.669338+0000","last_fullsized":"2024-04-05T23:33:34.669338+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T06:51:34.090030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"82'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-05T23:33:34.652270+0000","last_change":"2024-04-05T23:32:04.527711+0000","last_active":"2024-04-05T23:33:34.652270+0000","last_peered":"2024-04-05T23:33:34.652270+0000","last_clean":"2024-04-05T23:33:34.652270+0000","last_became_active":"2024-04-05T23:32:04.527374+0000","last_became_peered":"2024-04-05T23:32:04.527374+0000","last_unstale":"2024-04-05T23:33:34.652270+0000","last_undegraded":"2024-04-05T23:33:34.652270+0000","last_fullsized":"2024-04-05T23:33:34.652270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T23:32:03.978836+0000","last_clean_scrub_stamp":"2024-04-05T23:32:03.978836+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T02:15:10.764340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599588,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":997,"num_read_kb":612,"num_write":360,"num_write_kb":1458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2162688,"data_stored":1798764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579931,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":440050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.60599999999999998,"15min":0.59599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.968,"5min":3.968,"15min":3.968},"last":0.434}]},{"osd":1,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.64400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80400000000000005,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.64100000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69399999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.48199999999999998,"15min":0.48999999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67400000000000004,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":4,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.79300000000000004,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.75},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80400000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.66400000000000003}]},{"osd":5,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":1.353,"15min":1.353},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79400000000000004,"5min":1.306,"15min":1.306},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.74299999999999999,"15min":0.751},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85399999999999998,"5min":1.109,"15min":1.109},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.751,"15min":0.76600000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.89400000000000002,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.71699999999999997}]},{"osd":8,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64300000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.78500000000000003,"15min":0.80300000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.92800000000000005,"5min":1.71,"15min":1.71},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.74199999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.95699999999999996,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.78300000000000003}]},{"osd":10,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.78500000000000003,"15min":0.79700000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.94299999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.80100000000000005,"15min":0.81799999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.877,"5min":1.667,"15min":1.667},"last":0.73599999999999999}]}]},{"osd":10,"up_from":67,"seq":287762808863,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1744896,"data_stored":1228526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64000000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.35699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.878,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.372}]},{"osd":2,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66700000000000004,"15min":0.66800000000000004},"min":{"1min":0.52900000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65500000000000003,"15min":0.65900000000000003},"min":{"1min":0.435,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.81699999999999995,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.72399999999999998,"15min":0.70399999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.8660000000000001,"5min":3.8660000000000001,"15min":3.8660000000000001},"last":0.43099999999999999}]},{"osd":4,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81499999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.308},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83099999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.39000000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.94299999999999995,"5min":1.145,"15min":1.145},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0269999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.48499999999999999,"5min":0.441,"15min":0.441},"max":{"1min":0.96199999999999997,"5min":1.228,"15min":1.228},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.76700000000000002,"15min":0.77100000000000002},"min":{"1min":0.51200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.92600000000000005,"5min":1.129,"15min":1.129},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79900000000000004,"15min":0.80100000000000005},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0129999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.79200000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.042,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.41999999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.85399999999999998,"15min":0.86699999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0109999999999999,"5min":1.444,"15min":1.444},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.82299999999999995,"15min":0.83299999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.89900000000000002,"5min":1.258,"15min":1.258},"last":0.505}]},{"osd":11,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.82399999999999995,"15min":0.82499999999999996},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0149999999999999,"5min":1.351,"15min":1.351},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.80600000000000005,"15min":0.80900000000000005},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.94999999999999996,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.629}]}]},{"osd":9,"up_from":62,"seq":266287972387,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":439298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.55500000000000005,"15min":0.57099999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67700000000000005,"5min":0.746,"15min":0.746},"last":0.45800000000000002}]},{"osd":1,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.234,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77900000000000003,"15min":0.79700000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1060000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.70599999999999996}]},{"osd":2,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.74299999999999999,"15min":0.76300000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90900000000000003,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.754,"15min":0.77200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.036,"5min":1.256,"15min":1.256},"last":0.72299999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.78700000000000003,"15min":0.81100000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95699999999999996,"5min":1.27,"15min":1.27},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.71899999999999997,"15min":0.74099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.97599999999999998,"5min":1.405,"15min":1.405},"last":0.65300000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0329999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.87,"15min":0.88600000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.86699999999999999,"15min":0.89300000000000002},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.98399999999999999,"5min":1.637,"15min":1.637},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.89600000000000002,"15min":0.91900000000000004},"min":{"1min":0.55800000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1299999999999999,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.745}]},{"osd":6,"last update":"Fri Apr 5 23:31:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.94099999999999995,"15min":0.96899999999999997},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.071,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.94499999999999995,"15min":0.95899999999999996},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1619999999999999,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.68300000000000005}]},{"osd":8,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.0069999999999999,"15min":1.0309999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.173,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.996,"15min":1.02},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.127,"5min":1.778,"15min":1.778},"last":0.84799999999999998}]},{"osd":10,"last update":"Fri Apr 5 23:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.95499999999999996,"15min":0.97699999999999998},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.2110000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.05},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.89500000000000002,"15min":0.91100000000000003},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.111,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.79200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]}]},{"osd":8,"up_from":55,"seq":236223201320,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1314816,"data_stored":832310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56200000000000006,"15min":0.57399999999999995},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68300000000000005,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.63800000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.79400000000000004}]},{"osd":2,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.46200000000000002,"15min":0.44800000000000001},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.46400000000000002,"15min":0.45900000000000002},"min":{"1min":0.308,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.65300000000000002,"5min":0.748,"15min":0.748},"last":0.496}]},{"osd":3,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.45900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":1.101,"15min":1.101},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.55000000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.626,"15min":0.61399999999999999},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.91000000000000003,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64300000000000002,"15min":0.64100000000000001},"min":{"1min":0.38500000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85999999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.66600000000000004}]},{"osd":5,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.35499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69499999999999995,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.40500000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.58599999999999997}]},{"osd":6,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.68899999999999995,"15min":0.69199999999999995},"min":{"1min":0.48699999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.56899999999999995}]},{"osd":7,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.878,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94699999999999995,"5min":1.276,"15min":1.276},"last":0.75700000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72299999999999998,"15min":0.71299999999999997},"min":{"1min":0.56799999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73899999999999999,"15min":0.73399999999999999},"min":{"1min":0.60499999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.68999999999999995}]},{"osd":10,"last update":"Fri Apr 5 23:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.73799999999999999,"15min":0.72099999999999997},"min":{"1min":0.66600000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.73999999999999999,"15min":0.72399999999999998},"min":{"1min":0.66900000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.77500000000000002}]},{"osd":11,"last update":"Fri Apr 5 23:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.74299999999999999}]}]},{"osd":7,"up_from":48,"seq":206158430250,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32604,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712548,"statfs":{"total":95995035648,"available":95961649152,"internally_reserved":0,"allocated":1523712,"data_stored":1033930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.76400000000000001}]},{"osd":1,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64800000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.434,"15min":0.42199999999999999},"min":{"1min":0.36199999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.34499999999999997}]},{"osd":2,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.63300000000000001,"5min":0.879,"15min":0.879},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.58299999999999996},"min":{"1min":0.438,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.69099999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52300000000000002}]},{"osd":3,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58999999999999997,"15min":0.58399999999999996},"min":{"1min":0.42699999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.621,"15min":0.60999999999999999},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.872,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.497}]},{"osd":4,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.51300000000000001},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.68999999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.505,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.64000000000000001,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.39000000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.61699999999999999,"15min":0.621},"min":{"1min":0.438,"5min":0.378,"15min":0.378},"max":{"1min":0.73699999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65400000000000003,"15min":0.65000000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.83799999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.56799999999999995,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67900000000000005,"15min":0.67800000000000005},"min":{"1min":0.47999999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.73999999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.47599999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.49299999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.81000000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72999999999999998,"15min":0.72899999999999998},"min":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94899999999999995,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.65200000000000002}]},{"osd":9,"last update":"Fri Apr 5 23:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.77200000000000002,"15min":0.77400000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.96999999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.79400000000000004,"15min":0.79600000000000004},"min":{"1min":0.67400000000000004,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.875,"5min":1.216,"15min":1.216},"last":0.68200000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.52400000000000002,"5min":0.498,"15min":0.498},"max":{"1min":0.95199999999999996,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.623},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.755,"15min":0.751},"min":{"1min":0.50600000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.92700000000000005,"5min":1.099,"15min":1.099},"last":0.55200000000000005}]},{"osd":11,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.67300000000000004}]}]},{"osd":6,"up_from":42,"seq":180388626478,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":909312,"data_stored":439480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.34200000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.39300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.92500000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.65800000000000003,"15min":0.63800000000000001},"min":{"1min":0.50900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.85999999999999999}]},{"osd":2,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63,"15min":0.628},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.004,"5min":0.73599999999999999,"15min":0.69699999999999995},"min":{"1min":0.47499999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":6.867,"5min":6.867,"15min":6.867},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.44700000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.095,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.68999999999999995,"15min":0.67400000000000004},"min":{"1min":0.46600000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.169,"5min":1.246,"15min":1.246},"last":0.89800000000000002}]},{"osd":4,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.47399999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.45700000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.76500000000000001}]},{"osd":5,"last update":"Fri Apr 5 23:33:01 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.79400000000000004,"15min":0.753},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":6.9189999999999996,"5min":6.9189999999999996,"15min":6.9189999999999996},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.3180000000000001,"5min":0.86199999999999999,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":6.9009999999999998,"5min":6.9009999999999998,"15min":6.9009999999999998},"last":0.97999999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.83899999999999997,"15min":0.79900000000000004},"min":{"1min":0.58199999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":6.883,"5min":6.883,"15min":6.883},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.84099999999999997,"15min":0.80200000000000005},"min":{"1min":0.61799999999999999,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":4.8559999999999999,"5min":4.8559999999999999,"15min":4.8559999999999999},"last":0.94999999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":0.83899999999999997,"15min":0.80900000000000005},"min":{"1min":0.64900000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.8719999999999999,"5min":4.8719999999999999,"15min":4.8719999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.81200000000000006,"15min":0.79600000000000004},"min":{"1min":0.58199999999999996,"5min":0.504,"15min":0.504},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":1}]},{"osd":9,"last update":"Fri Apr 5 23:32:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91200000000000003,"15min":0.91600000000000004},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.63400000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.123,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.93500000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.877,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.91800000000000004,"15min":0.91600000000000004},"min":{"1min":0.74199999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":1.0209999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607618,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36160,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708992,"statfs":{"total":95995035648,"available":95958007808,"internally_reserved":0,"allocated":974848,"data_stored":505878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52100000000000002,"15min":0.504},"min":{"1min":0.45000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68799999999999994,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61499999999999999,"15min":0.67600000000000005},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.79900000000000004,"5min":4.4199999999999999,"15min":4.4199999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Apr 5 23:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55200000000000005,"15min":0.52000000000000002},"min":{"1min":0.34699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.57299999999999995,"15min":0.52100000000000002},"min":{"1min":0.53400000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59499999999999997,"15min":0.54500000000000004},"min":{"1min":0.44900000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56999999999999995,"15min":0.54400000000000004},"min":{"1min":0.45100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89200000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Apr 5 23:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.42799999999999999,"15min":0.40500000000000003},"min":{"1min":0.33700000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78700000000000003,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.495},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.46800000000000003,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.877,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.52100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91600000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.71399999999999997,"15min":0.67200000000000004},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.67900000000000005}]},{"osd":6,"last update":"Fri Apr 5 23:32:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70999999999999996,"15min":0.68600000000000005},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.438},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66200000000000003,"15min":0.65200000000000002},"min":{"1min":0.441,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57599999999999996}]},{"osd":7,"last update":"Fri Apr 5 23:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.63700000000000001,"15min":0.61099999999999999},"min":{"1min":0.40400000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90300000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.57499999999999996},"min":{"1min":0.38,"5min":0.312,"15min":0.312},"max":{"1min":0.81899999999999995,"5min":1.002,"15min":1.002},"last":0.436}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.64200000000000002,"15min":0.62},"min":{"1min":0.54100000000000004,"5min":0.378,"15min":0.378},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.754,"15min":0.71799999999999997},"min":{"1min":0.71399999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.71399999999999997}]},{"osd":9,"last update":"Fri Apr 5 23:32:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.72299999999999998,"15min":0.71099999999999997},"min":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.69899999999999995,"15min":0.68999999999999995},"min":{"1min":0.60699999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.627}]},{"osd":10,"last update":"Fri Apr 5 23:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86699999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.505},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.80500000000000005,"15min":0.79500000000000004},"min":{"1min":0.59499999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.56299999999999994}]},{"osd":11,"last update":"Fri Apr 5 23:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.748}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36124,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709028,"statfs":{"total":95995035648,"available":95958044672,"internally_reserved":0,"allocated":933888,"data_stored":441092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 23:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57299999999999995,"15min":0.51500000000000001},"min":{"1min":0.40300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.97199999999999998,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56100000000000005,"15min":0.50700000000000001},"min":{"1min":0.38100000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85999999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.63200000000000001,"15min":0.58799999999999997},"min":{"1min":0.47999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0049999999999999,"5min":7.2060000000000004,"15min":7.2060000000000004},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:33:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.91600000000000004}]},{"osd":4,"last update":"Fri Apr 5 23:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.78900000000000003,"15min":0.91500000000000004},"min":{"1min":0.51800000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92500000000000004,"5min":7.1769999999999996,"15min":7.1769999999999996},"last":1.258},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.66000000000000003},"min":{"1min":0.57999999999999996,"5min":0.436,"15min":0.436},"max":{"1min":0.88200000000000001,"5min":1.395,"15min":1.395},"last":0.79900000000000004}]},{"osd":5,"last update":"Fri Apr 5 23:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68899999999999995,"15min":0.66800000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.2250000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.70999999999999996},"min":{"1min":0.55900000000000005,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":1.339}]},{"osd":6,"last update":"Fri Apr 5 23:32:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.83399999999999996,"15min":0.87},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.5489999999999999,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.2390000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78400000000000003,"15min":0.77700000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":1.294}]},{"osd":7,"last update":"Fri Apr 5 23:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.70699999999999996}]},{"osd":8,"last update":"Fri Apr 5 23:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.89200000000000002,"15min":0.93000000000000005},"min":{"1min":0.499,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.2789999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.306},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.93700000000000006,"15min":0.995},"min":{"1min":0.629,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4610000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.2010000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:32:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.94999999999999996,"15min":0.96499999999999997},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.88100000000000001,"15min":0.876},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.3919999999999999}]},{"osd":10,"last update":"Fri Apr 5 23:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.91300000000000003,"15min":0.92600000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":1.379},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.94199999999999995,"15min":0.95799999999999996},"min":{"1min":0.66900000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3080000000000001,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.2789999999999999}]},{"osd":11,"last update":"Fri Apr 5 23:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.89800000000000002,"15min":0.90200000000000002},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.92000000000000004,"15min":0.92900000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":1.415}]}]},{"osd":2,"up_from":17,"seq":73014444095,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":987136,"data_stored":504410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Fri Apr 5 23:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76400000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.751,"5min":0.874,"15min":0.874},"last":0.44}]},{"osd":3,"last update":"Fri Apr 5 23:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.311,"15min":0.311},"max":{"1min":0.82299999999999995,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.62}]},{"osd":4,"last update":"Fri Apr 5 23:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.503,"15min":0.47799999999999998},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86699999999999999,"5min":1.359,"15min":1.359},"last":0.60499999999999998}]},{"osd":6,"last update":"Fri Apr 5 23:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.45100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.58999999999999997},"min":{"1min":0.443,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.76100000000000001,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.49399999999999999}]},{"osd":7,"last update":"Fri Apr 5 23:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73199999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.48399999999999999,"15min":0.47299999999999998},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70499999999999996,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.63400000000000001}]},{"osd":9,"last update":"Fri Apr 5 23:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.874,"5min":1.532,"15min":1.532},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85399999999999998,"5min":1.429,"15min":1.429},"last":0.66800000000000004}]},{"osd":10,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.82099999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72499999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.88700000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.65200000000000002}]},{"osd":11,"last update":"Fri Apr 5 23:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.69599999999999995}]}]},{"osd":3,"up_from":23,"seq":98784247866,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36628,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708524,"statfs":{"total":95995035648,"available":95957528576,"internally_reserved":0,"allocated":1454080,"data_stored":963383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.45600000000000002},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.995,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52700000000000002,"15min":0.46899999999999997},"min":{"1min":0.34200000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.996,"5min":4.96,"15min":4.96},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Apr 5 23:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66200000000000003,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2390000000000001,"5min":4.5590000000000002,"15min":4.5590000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.59899999999999998},"min":{"1min":0.35899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":1.0149999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Apr 5 23:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.68200000000000005},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.107,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72599999999999998,"15min":0.67800000000000005},"min":{"1min":0.5,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.167,"5min":4.6710000000000003,"15min":4.6710000000000003},"last":0.76600000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73499999999999999,"15min":0.67600000000000005},"min":{"1min":0.58099999999999996,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1519999999999999,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.753,"15min":0.67900000000000005},"min":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.278,"5min":4.6289999999999996,"15min":4.6289999999999996},"last":0.73199999999999998}]},{"osd":5,"last update":"Fri Apr 5 23:33:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76100000000000001,"15min":0.70099999999999996},"min":{"1min":0.52800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.095,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.78600000000000003,"15min":0.72799999999999998},"min":{"1min":0.47999999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.96699999999999997,"5min":4.8929999999999998,"15min":4.8929999999999998},"last":0.82799999999999996}]},{"osd":6,"last update":"Fri Apr 5 23:32:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.84399999999999997,"15min":0.872},"min":{"1min":0.497,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.29,"5min":4.8159999999999998,"15min":4.8159999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.747,"15min":0.80100000000000005},"min":{"1min":0.433,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0569999999999999,"5min":5.0979999999999999,"15min":5.0979999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Apr 5 23:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.96999999999999997,"15min":0.998},"min":{"1min":0.50800000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":4.7610000000000001,"5min":4.9980000000000002,"15min":4.9980000000000002},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.044,"5min":0.94899999999999995,"15min":0.96699999999999997},"min":{"1min":0.58999999999999997,"5min":0.502,"15min":0.502},"max":{"1min":4.7759999999999998,"5min":4.7759999999999998,"15min":4.7759999999999998},"last":1.149}]},{"osd":8,"last update":"Fri Apr 5 23:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":4.7569999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.0300000000000002,"5min":5.0300000000000002,"15min":5.0300000000000002},"last":0.72199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.94399999999999995,"15min":0.96599999999999997},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.081,"5min":1.349,"15min":1.349},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.80800000000000005}]},{"osd":10,"last update":"Fri Apr 5 23:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.95699999999999996,"15min":0.97199999999999998},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.224,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.876,"15min":0.878},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":1.1890000000000001}]},{"osd":11,"last update":"Fri Apr 5 23:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":1.0920000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":909312,"data_stored":439647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.40799999999999997,"15min":0.374},"min":{"1min":0.35899999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.65200000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.39100000000000001},"min":{"1min":0.32500000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.71299999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.39800000000000002}]},{"osd":2,"last update":"Fri Apr 5 23:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76900000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58199999999999996,"15min":0.55400000000000005},"min":{"1min":0.45900000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.76500000000000001,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Apr 5 23:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.56999999999999995},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76700000000000002,"5min":1.117,"15min":1.117},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71899999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Apr 5 23:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63800000000000001,"15min":0.60899999999999999},"min":{"1min":0.53300000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46800000000000003}]},{"osd":6,"last update":"Fri Apr 5 23:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92900000000000005,"5min":1.137,"15min":1.137},"last":0.505},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.627,"15min":0.59899999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89700000000000002,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.55800000000000005}]},{"osd":7,"last update":"Fri Apr 5 23:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95299999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86299999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.34799999999999998}]},{"osd":8,"last update":"Fri Apr 5 23:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.48899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.874,"5min":1.087,"15min":1.087},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.755,"15min":0.747},"min":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.434}]},{"osd":9,"last update":"Fri Apr 5 23:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80900000000000005,"15min":0.81299999999999994},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.071,"5min":1.28,"15min":1.28},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.53800000000000003}]},{"osd":10,"last update":"Fri Apr 5 23:33:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.752,"15min":0.75},"min":{"1min":0.55100000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95299999999999996,"5min":1.181,"15min":1.181},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.59899999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0249999999999999,"5min":1.371,"15min":1.371},"last":0.54900000000000004}]},{"osd":11,"last update":"Fri Apr 5 23:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.58299999999999996}]}]},{"osd":5,"up_from":35,"seq":150323855410,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1536000,"data_stored":1031433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54100000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80200000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.34200000000000003}]},{"osd":1,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71999999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68799999999999994,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.30299999999999999}]},{"osd":2,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.28499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.35799999999999998}]},{"osd":3,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.58599999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.78200000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.317},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58399999999999996,"15min":0.57199999999999995},"min":{"1min":0.439,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.79700000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56899999999999995,"15min":0.55700000000000005},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60099999999999998,"15min":0.58899999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83299999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.38300000000000001}]},{"osd":6,"last update":"Fri Apr 5 23:32:57 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.45600000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82199999999999995,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.441},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.48199999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84599999999999997,"5min":1.153,"15min":1.153},"last":0.51800000000000002}]},{"osd":7,"last update":"Fri Apr 5 23:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.61699999999999999,"15min":0.61199999999999999},"min":{"1min":0.46700000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.48899999999999999}]},{"osd":8,"last update":"Fri Apr 5 23:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53000000000000003,"15min":0.51500000000000001},"min":{"1min":0.34699999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.371,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48199999999999998}]},{"osd":9,"last update":"Fri Apr 5 23:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70099999999999996,"15min":0.69199999999999995},"min":{"1min":0.59999999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.67100000000000004,"15min":0.65700000000000003},"min":{"1min":0.60199999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.89200000000000002,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.58899999999999997}]},{"osd":10,"last update":"Fri Apr 5 23:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.55400000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.67600000000000005,"15min":0.66500000000000004},"min":{"1min":0.53400000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":199865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T23:33:42.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:33:42.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.720+0000 7fa8c5e00700 1 -- 172.21.15.16:0/3321361908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c0103b70 msgr2=0x7fa8c0107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.720+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/3321361908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c0103b70 0x7fa8c0107bb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009990 tx=0x7fa8ac009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.720+0000 7fa8c5e00700 1 -- 172.21.15.16:0/3321361908 shutdown_connections 2024-04-05T23:33:42.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.720+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/3321361908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c01082e0 0x7fa8c010fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.720+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/3321361908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8c01031c0 0x7fa8c01035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.720+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/3321361908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c0103b70 0x7fa8c0107bb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.720+0000 7fa8c5e00700 1 -- 172.21.15.16:0/3321361908 >> 172.21.15.16:0/3321361908 conn(0x7fa8c00fea00 msgr2=0x7fa8c0100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:42.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.720+0000 7fa8c5e00700 1 -- 172.21.15.16:0/3321361908 shutdown_connections 2024-04-05T23:33:42.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.720+0000 7fa8c5e00700 1 -- 172.21.15.16:0/3321361908 wait complete. 2024-04-05T23:33:42.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.721+0000 7fa8c5e00700 1 Processor -- start 2024-04-05T23:33:42.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.721+0000 7fa8c5e00700 1 -- start start 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.721+0000 7fa8c5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c01031c0 0x7fa8c01994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.721+0000 7fa8c5e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8c01082e0 0x7fa8c0199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.721+0000 7fa8c5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c019dda0 0x7fa8c019e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.721+0000 7fa8c5e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c019e820 con 0x7fa8c019dda0 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.721+0000 7fa8c5e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c019e990 con 0x7fa8c01031c0 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.721+0000 7fa8c5e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c019eb50 con 0x7fa8c01082e0 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c019dda0 0x7fa8c019e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c01031c0 0x7fa8c01994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c01031c0 0x7fa8c01994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59182/0 (socket says 172.21.15.16:59182) 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c019dda0 0x7fa8c019e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40786/0 (socket says 172.21.15.16:40786) 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bea00700 1 -- 172.21.15.16:0/2859187764 learned_addr learned my addr 172.21.15.16:0/2859187764 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8be000700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8c01082e0 0x7fa8c0199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bea00700 1 -- 172.21.15.16:0/2859187764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8c01082e0 msgr2=0x7fa8c0199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bea00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8c01082e0 0x7fa8c0199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bea00700 1 -- 172.21.15.16:0/2859187764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c019dda0 msgr2=0x7fa8c019e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bea00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c019dda0 0x7fa8c019e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bea00700 1 -- 172.21.15.16:0/2859187764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac0095f0 con 0x7fa8c01031c0 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8be000700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8c01082e0 0x7fa8c0199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8bea00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c01031c0 0x7fa8c01994c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000e9a0 tx=0x7fa8b000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.722+0000 7fa8c4a00700 1 -- 172.21.15.16:0/2859187764 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0009790 con 0x7fa8c01031c0 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.723+0000 7fa8c4a00700 1 -- 172.21.15.16:0/2859187764 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa8b0010430 con 0x7fa8c01031c0 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.723+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c01a60b0 con 0x7fa8c01031c0 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.723+0000 7fa8c4a00700 1 -- 172.21.15.16:0/2859187764 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0018680 con 0x7fa8c01031c0 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.723+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c01a65e0 con 0x7fa8c01031c0 2024-04-05T23:33:42.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.724+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c010b5b0 con 0x7fa8c01031c0 2024-04-05T23:33:42.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.725+0000 7fa8c4a00700 1 -- 172.21.15.16:0/2859187764 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7fa8b00105a0 con 0x7fa8c01031c0 2024-04-05T23:33:42.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.727+0000 7fa8c4a00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa8a40a13a0 0x7fa8a40a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:42.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.727+0000 7fa8c4a00700 1 -- 172.21.15.16:0/2859187764 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa8b0014070 con 0x7fa8c01031c0 2024-04-05T23:33:42.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.727+0000 7fa8be000700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa8a40a13a0 0x7fa8a40a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:42.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.728+0000 7fa8be000700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa8a40a13a0 0x7fa8a40a3860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa8c019aa10 tx=0x7fa8ac00b260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:42.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.729+0000 7fa8c4a00700 1 -- 172.21.15.16:0/2859187764 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa8b0096310 con 0x7fa8c01031c0 2024-04-05T23:33:42.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:42 smithi016 ceph-mon[38248]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:33:42.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:42 smithi016 ceph-mon[38248]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:33:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.949+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8c019a7c0 con 0x7fa8c01031c0 2024-04-05T23:33:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.951+0000 7fa8c4a00700 1 -- 172.21.15.16:0/2859187764 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22958 (secure 0 0 0) 0x7fa8b0099930 con 0x7fa8c01031c0 2024-04-05T23:33:42.944 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:42.945 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:33:34.593258+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/985211078":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1740984099":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/4157043736":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/1565223355":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2846240830":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/4184690359":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/1565223355":"2024-04-06T23:33:34.593227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:33:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa8a40a13a0 msgr2=0x7fa8a40a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa8a40a13a0 0x7fa8a40a3860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa8c019aa10 tx=0x7fa8ac00b260 comp rx=0 tx=0).stop 2024-04-05T23:33:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c01031c0 msgr2=0x7fa8c01994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c01031c0 0x7fa8c01994c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000e9a0 tx=0x7fa8b000ecb0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 shutdown_connections 2024-04-05T23:33:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa8a40a13a0 0x7fa8a40a3860 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8c019dda0 0x7fa8c019e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c01031c0 0x7fa8c01994c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 --2- 172.21.15.16:0/2859187764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8c01082e0 0x7fa8c0199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.953+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 >> 172.21.15.16:0/2859187764 conn(0x7fa8c00fea00 msgr2=0x7fa8c00ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.954+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 shutdown_connections 2024-04-05T23:33:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:42.954+0000 7fa8c5e00700 1 -- 172.21.15.16:0/2859187764 wait complete. 2024-04-05T23:33:42.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-05T23:32:02.641673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-05T23:32:03.972521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:33:42.967 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:33:42.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:42 smithi027 ceph-mon[78597]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:33:42.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:42 smithi027 ceph-mon[78597]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:33:43.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:42 smithi130 ceph-mon[79417]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:33:43.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:42 smithi130 ceph-mon[79417]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T23:33:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.177+0000 7fc050000700 1 -- 172.21.15.16:0/1120360393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0480732b0 msgr2=0x7fc048073690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.177+0000 7fc050000700 1 --2- 172.21.15.16:0/1120360393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0480732b0 0x7fc048073690 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009990 tx=0x7fc03c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.177+0000 7fc050000700 1 -- 172.21.15.16:0/1120360393 shutdown_connections 2024-04-05T23:33:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.177+0000 7fc050000700 1 --2- 172.21.15.16:0/1120360393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0480732b0 0x7fc048073690 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.177+0000 7fc050000700 1 --2- 172.21.15.16:0/1120360393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc04810d6b0 0x7fc04810fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.177+0000 7fc050000700 1 --2- 172.21.15.16:0/1120360393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc048073bd0 0x7fc04810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.177+0000 7fc050000700 1 -- 172.21.15.16:0/1120360393 >> 172.21.15.16:0/1120360393 conn(0x7fc0480fc920 msgr2=0x7fc0480fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.177+0000 7fc050000700 1 -- 172.21.15.16:0/1120360393 shutdown_connections 2024-04-05T23:33:43.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.177+0000 7fc050000700 1 -- 172.21.15.16:0/1120360393 wait complete. 2024-04-05T23:33:43.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.178+0000 7fc050000700 1 Processor -- start 2024-04-05T23:33:43.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.178+0000 7fc050000700 1 -- start start 2024-04-05T23:33:43.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc050000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480732b0 0x7fc0481995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc050000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc048073bd0 0x7fc048199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc050000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc04810d6b0 0x7fc04819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc050000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04819e4d0 con 0x7fc04810d6b0 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc050000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04819e640 con 0x7fc0480732b0 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc050000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04819e800 con 0x7fc048073bd0 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc04e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc04810d6b0 0x7fc04819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc04d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480732b0 0x7fc0481995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc04e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc04810d6b0 0x7fc04819de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40810/0 (socket says 172.21.15.16:40810) 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc04e000700 1 -- 172.21.15.16:0/2991958445 learned_addr learned my addr 172.21.15.16:0/2991958445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.179+0000 7fc04cc00700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc048073bd0 0x7fc048199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc04d600700 1 -- 172.21.15.16:0/2991958445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc048073bd0 msgr2=0x7fc048199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc04d600700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc048073bd0 0x7fc048199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc04d600700 1 -- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc04810d6b0 msgr2=0x7fc04819de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc04d600700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc04810d6b0 0x7fc04819de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc04d600700 1 -- 172.21.15.16:0/2991958445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c0095f0 con 0x7fc0480732b0 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc04e000700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc04810d6b0 0x7fc04819de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc04d600700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480732b0 0x7fc0481995b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc03c000c00 tx=0x7fc03c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc046000700 1 -- 172.21.15.16:0/2991958445 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c01d070 con 0x7fc0480732b0 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04819ea80 con 0x7fc0480732b0 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc046000700 1 -- 172.21.15.16:0/2991958445 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc03c021410 con 0x7fc0480732b0 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc046000700 1 -- 172.21.15.16:0/2991958445 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c017650 con 0x7fc0480732b0 2024-04-05T23:33:43.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.180+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0481a6320 con 0x7fc0480732b0 2024-04-05T23:33:43.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.181+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0481088c0 con 0x7fc0480732b0 2024-04-05T23:33:43.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.183+0000 7fc046000700 1 -- 172.21.15.16:0/2991958445 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7fc03c021580 con 0x7fc0480732b0 2024-04-05T23:33:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.184+0000 7fc046000700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc0300a1320 0x7fc0300a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.185+0000 7fc046000700 1 -- 172.21.15.16:0/2991958445 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fc03c0ce790 con 0x7fc0480732b0 2024-04-05T23:33:43.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.185+0000 7fc04cc00700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc0300a1320 0x7fc0300a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.185+0000 7fc04cc00700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc0300a1320 0x7fc0300a37e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc04819ab00 tx=0x7fc034009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.187+0000 7fc046000700 1 -- 172.21.15.16:0/2991958445 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc03c097ad0 con 0x7fc0480732b0 2024-04-05T23:33:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.403+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc048068080 con 0x7fc0480732b0 2024-04-05T23:33:43.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.404+0000 7fc046000700 1 -- 172.21.15.16:0/2991958445 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc03c09b0f0 con 0x7fc0480732b0 2024-04-05T23:33:43.396 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:33:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc0300a1320 msgr2=0x7fc0300a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc0300a1320 0x7fc0300a37e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc04819ab00 tx=0x7fc034009260 comp rx=0 tx=0).stop 2024-04-05T23:33:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480732b0 msgr2=0x7fc0481995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480732b0 0x7fc0481995b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc03c000c00 tx=0x7fc03c004730 comp rx=0 tx=0).stop 2024-04-05T23:33:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 shutdown_connections 2024-04-05T23:33:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc0300a1320 0x7fc0300a37e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc04810d6b0 0x7fc04819de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480732b0 0x7fc0481995b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 --2- 172.21.15.16:0/2991958445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc048073bd0 0x7fc048199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 >> 172.21.15.16:0/2991958445 conn(0x7fc0480fc920 msgr2=0x7fc0480fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 shutdown_connections 2024-04-05T23:33:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.406+0000 7fc050000700 1 -- 172.21.15.16:0/2991958445 wait complete. 2024-04-05T23:33:43.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-05T23:33:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.626+0000 7fd1d6800700 1 -- 172.21.15.16:0/2900472666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0073280 msgr2=0x7fd1d0073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.626+0000 7fd1d6800700 1 --2- 172.21.15.16:0/2900472666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0073280 0x7fd1d0073660 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009990 tx=0x7fd1c0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.627+0000 7fd1d6800700 1 -- 172.21.15.16:0/2900472666 shutdown_connections 2024-04-05T23:33:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.627+0000 7fd1d6800700 1 --2- 172.21.15.16:0/2900472666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0073280 0x7fd1d0073660 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.627+0000 7fd1d6800700 1 --2- 172.21.15.16:0/2900472666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d0111e60 0x7fd1d0114250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.627+0000 7fd1d6800700 1 --2- 172.21.15.16:0/2900472666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1d0073ba0 0x7fd1d0111920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.627+0000 7fd1d6800700 1 -- 172.21.15.16:0/2900472666 >> 172.21.15.16:0/2900472666 conn(0x7fd1d00fc920 msgr2=0x7fd1d00fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.627+0000 7fd1d6800700 1 -- 172.21.15.16:0/2900472666 shutdown_connections 2024-04-05T23:33:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.627+0000 7fd1d6800700 1 -- 172.21.15.16:0/2900472666 wait complete. 2024-04-05T23:33:43.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.628+0000 7fd1d6800700 1 Processor -- start 2024-04-05T23:33:43.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.628+0000 7fd1d6800700 1 -- start start 2024-04-05T23:33:43.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.628+0000 7fd1d6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d0073280 0x7fd1d01a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.628+0000 7fd1d6800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0073ba0 0x7fd1d01a3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.628+0000 7fd1d6800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1d0111e60 0x7fd1d019d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.628+0000 7fd1d6800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d019ddf0 con 0x7fd1d0073ba0 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.628+0000 7fd1d6800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d019df60 con 0x7fd1d0073280 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.628+0000 7fd1d6800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d019e120 con 0x7fd1d0111e60 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d0073280 0x7fd1d01a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d0073280 0x7fd1d01a3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59208/0 (socket says 172.21.15.16:59208) 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cf400700 1 -- 172.21.15.16:0/94491505 learned_addr learned my addr 172.21.15.16:0/94491505 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cea00700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0073ba0 0x7fd1d01a3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cfe00700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1d0111e60 0x7fd1d019d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cf400700 1 -- 172.21.15.16:0/94491505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1d0111e60 msgr2=0x7fd1d019d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cf400700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1d0111e60 0x7fd1d019d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cf400700 1 -- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0073ba0 msgr2=0x7fd1d01a3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cf400700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0073ba0 0x7fd1d01a3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cf400700 1 -- 172.21.15.16:0/94491505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c00095f0 con 0x7fd1d0073280 2024-04-05T23:33:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cea00700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0073ba0 0x7fd1d01a3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:43.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.629+0000 7fd1cf400700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d0073280 0x7fd1d01a3650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0000c00 tx=0x7fd1c0005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.632+0000 7fd1c7e00700 1 -- 172.21.15.16:0/94491505 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c001d070 con 0x7fd1d0073280 2024-04-05T23:33:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.633+0000 7fd1c7e00700 1 -- 172.21.15.16:0/94491505 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1c0022470 con 0x7fd1d0073280 2024-04-05T23:33:43.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.633+0000 7fd1c7e00700 1 -- 172.21.15.16:0/94491505 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0017650 con 0x7fd1d0073280 2024-04-05T23:33:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.633+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d019e3a0 con 0x7fd1d0073280 2024-04-05T23:33:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.633+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d01aa640 con 0x7fd1d0073280 2024-04-05T23:33:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.634+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d0104340 con 0x7fd1d0073280 2024-04-05T23:33:43.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.636+0000 7fd1c7e00700 1 -- 172.21.15.16:0/94491505 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7fd1c00225e0 con 0x7fd1d0073280 2024-04-05T23:33:43.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.637+0000 7fd1c7e00700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd1b40a13a0 0x7fd1b40a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:43.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.637+0000 7fd1c7e00700 1 -- 172.21.15.16:0/94491505 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd1c00ce250 con 0x7fd1d0073280 2024-04-05T23:33:43.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.637+0000 7fd1cea00700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd1b40a13a0 0x7fd1b40a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:43.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.638+0000 7fd1cea00700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd1b40a13a0 0x7fd1b40a3860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd1d019eba0 tx=0x7fd1bc009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:43.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.641+0000 7fd1c7e00700 1 -- 172.21.15.16:0/94491505 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd1c00976b0 con 0x7fd1d0073280 2024-04-05T23:33:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:43 smithi016 ceph-mon[38248]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 293 B/s wr, 25 op/s 2024-04-05T23:33:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2859187764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2991958445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:43.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.857+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1d0067c60 con 0x7fd1d0073280 2024-04-05T23:33:43.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.858+0000 7fd1c7e00700 1 -- 172.21.15.16:0/94491505 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd1c0027020 con 0x7fd1d0073280 2024-04-05T23:33:43.850 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-05T23:33:43.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd1b40a13a0 msgr2=0x7fd1b40a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd1b40a13a0 0x7fd1b40a3860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd1d019eba0 tx=0x7fd1bc009260 comp rx=0 tx=0).stop 2024-04-05T23:33:43.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d0073280 msgr2=0x7fd1d01a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d0073280 0x7fd1d01a3650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0000c00 tx=0x7fd1c0005740 comp rx=0 tx=0).stop 2024-04-05T23:33:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 shutdown_connections 2024-04-05T23:33:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd1b40a13a0 0x7fd1b40a3860 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1d0111e60 0x7fd1d019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d0073280 0x7fd1d01a3650 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 --2- 172.21.15.16:0/94491505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d0073ba0 0x7fd1d01a3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 >> 172.21.15.16:0/94491505 conn(0x7fd1d00fc920 msgr2=0x7fd1d00fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.860+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 shutdown_connections 2024-04-05T23:33:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:43.861+0000 7fd1d6800700 1 -- 172.21.15.16:0/94491505 wait complete. 2024-04-05T23:33:43.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-05T23:33:43.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:43 smithi027 ceph-mon[78597]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 293 B/s wr, 25 op/s 2024-04-05T23:33:43.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:43 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2859187764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:43.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:43 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2991958445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.082+0000 7ff89f000700 1 -- 172.21.15.16:0/3022816347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 msgr2=0x7ff898101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.082+0000 7ff89f000700 1 --2- 172.21.15.16:0/3022816347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 0x7ff898101480 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff888009990 tx=0x7ff888009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.082+0000 7ff89f000700 1 -- 172.21.15.16:0/3022816347 shutdown_connections 2024-04-05T23:33:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.082+0000 7ff89f000700 1 --2- 172.21.15.16:0/3022816347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 0x7ff898101480 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.082+0000 7ff89f000700 1 --2- 172.21.15.16:0/3022816347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8981061c0 0x7ff89810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.082+0000 7ff89f000700 1 --2- 172.21.15.16:0/3022816347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff898101a50 0x7ff898105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.082+0000 7ff89f000700 1 -- 172.21.15.16:0/3022816347 >> 172.21.15.16:0/3022816347 conn(0x7ff8980fc900 msgr2=0x7ff8980fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.082+0000 7ff89f000700 1 -- 172.21.15.16:0/3022816347 shutdown_connections 2024-04-05T23:33:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.082+0000 7ff89f000700 1 -- 172.21.15.16:0/3022816347 wait complete. 2024-04-05T23:33:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.083+0000 7ff89f000700 1 Processor -- start 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.083+0000 7ff89f000700 1 -- start start 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.083+0000 7ff89f000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 0x7ff898199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.083+0000 7ff89f000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8981061c0 0x7ff898199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.083+0000 7ff89f000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89819de20 0x7ff89819e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.083+0000 7ff89f000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89819e8a0 con 0x7ff8981010a0 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.083+0000 7ff89f000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89819ea10 con 0x7ff8981061c0 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.083+0000 7ff89f000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89819ebd0 con 0x7ff89819de20 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 0x7ff898199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 0x7ff898199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40836/0 (socket says 172.21.15.16:40836) 2024-04-05T23:33:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897e00700 1 -- 172.21.15.16:0/2695725799 learned_addr learned my addr 172.21.15.16:0/2695725799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897400700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8981061c0 0x7ff898199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff89d000700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89819de20 0x7ff89819e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897e00700 1 -- 172.21.15.16:0/2695725799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89819de20 msgr2=0x7ff89819e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897e00700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89819de20 0x7ff89819e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897e00700 1 -- 172.21.15.16:0/2695725799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8981061c0 msgr2=0x7ff898199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897e00700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8981061c0 0x7ff898199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897e00700 1 -- 172.21.15.16:0/2695725799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8880095f0 con 0x7ff8981010a0 2024-04-05T23:33:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.084+0000 7ff897e00700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 0x7ff898199540 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff888005770 tx=0x7ff888009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.085+0000 7ff894c00700 1 -- 172.21.15.16:0/2695725799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88801d070 con 0x7ff8981010a0 2024-04-05T23:33:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.085+0000 7ff894c00700 1 -- 172.21.15.16:0/2695725799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff88800fb30 con 0x7ff8981010a0 2024-04-05T23:33:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.085+0000 7ff894c00700 1 -- 172.21.15.16:0/2695725799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888022bf0 con 0x7ff8981010a0 2024-04-05T23:33:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.085+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8981a60d0 con 0x7ff8981010a0 2024-04-05T23:33:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.085+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8981a64f0 con 0x7ff8981010a0 2024-04-05T23:33:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.086+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff898102570 con 0x7ff8981010a0 2024-04-05T23:33:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.088+0000 7ff894c00700 1 -- 172.21.15.16:0/2695725799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7ff888022d50 con 0x7ff8981010a0 2024-04-05T23:33:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.089+0000 7ff894c00700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff87c0a13a0 0x7ff87c0a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.089+0000 7ff894c00700 1 -- 172.21.15.16:0/2695725799 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff8880cdff0 con 0x7ff8981010a0 2024-04-05T23:33:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.089+0000 7ff897400700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff87c0a13a0 0x7ff87c0a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.090+0000 7ff897400700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff87c0a13a0 0x7ff87c0a3860 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff89819aa90 tx=0x7ff88400b410 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.091+0000 7ff894c00700 1 -- 172.21.15.16:0/2695725799 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff888097350 con 0x7ff8981010a0 2024-04-05T23:33:44.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:43 smithi130 ceph-mon[79417]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 293 B/s wr, 25 op/s 2024-04-05T23:33:44.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:43 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2859187764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:44.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:43 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2991958445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:33:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.307+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff898067c30 con 0x7ff8981010a0 2024-04-05T23:33:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.307+0000 7ff894c00700 1 -- 172.21.15.16:0/2695725799 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff888031020 con 0x7ff8981010a0 2024-04-05T23:33:44.300 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-05T23:33:44.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff87c0a13a0 msgr2=0x7ff87c0a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff87c0a13a0 0x7ff87c0a3860 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff89819aa90 tx=0x7ff88400b410 comp rx=0 tx=0).stop 2024-04-05T23:33:44.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 msgr2=0x7ff898199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 0x7ff898199540 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff888005770 tx=0x7ff888009580 comp rx=0 tx=0).stop 2024-04-05T23:33:44.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 shutdown_connections 2024-04-05T23:33:44.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff87c0a13a0 0x7ff87c0a3860 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8981010a0 0x7ff898199540 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89819de20 0x7ff89819e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 --2- 172.21.15.16:0/2695725799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8981061c0 0x7ff898199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 >> 172.21.15.16:0/2695725799 conn(0x7ff8980fc900 msgr2=0x7ff8980fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:44.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 shutdown_connections 2024-04-05T23:33:44.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.310+0000 7ff89f000700 1 -- 172.21.15.16:0/2695725799 wait complete. 2024-04-05T23:33:44.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:33:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.529+0000 7f7902200700 1 -- 172.21.15.16:0/36467938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78fc101950 msgr2=0x7f78fc105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.529+0000 7f7902200700 1 --2- 172.21.15.16:0/36467938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78fc101950 0x7f78fc105990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009990 tx=0x7f78e8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.530+0000 7f7902200700 1 -- 172.21.15.16:0/36467938 shutdown_connections 2024-04-05T23:33:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.530+0000 7f7902200700 1 --2- 172.21.15.16:0/36467938 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78fc1060c0 0x7f78fc10dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.530+0000 7f7902200700 1 --2- 172.21.15.16:0/36467938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78fc100fa0 0x7f78fc101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.530+0000 7f7902200700 1 --2- 172.21.15.16:0/36467938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78fc101950 0x7f78fc105990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.530+0000 7f7902200700 1 -- 172.21.15.16:0/36467938 >> 172.21.15.16:0/36467938 conn(0x7f78fc0fc8a0 msgr2=0x7f78fc0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.530+0000 7f7902200700 1 -- 172.21.15.16:0/36467938 shutdown_connections 2024-04-05T23:33:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.530+0000 7f7902200700 1 -- 172.21.15.16:0/36467938 wait complete. 2024-04-05T23:33:44.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f7902200700 1 Processor -- start 2024-04-05T23:33:44.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f7902200700 1 -- start start 2024-04-05T23:33:44.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f7902200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78fc100fa0 0x7f78fc0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:44.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f7902200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78fc1060c0 0x7f78fc073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:44.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f7902200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78fc073c60 0x7f78fc1a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:44.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f7902200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc1a5a60 con 0x7f78fc100fa0 2024-04-05T23:33:44.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f7902200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc1a5bd0 con 0x7f78fc1060c0 2024-04-05T23:33:44.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f7902200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc1a5d90 con 0x7f78fc073c60 2024-04-05T23:33:44.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f78fb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78fc100fa0 0x7f78fc0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:44.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f78faa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78fc1060c0 0x7f78fc073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:44.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f78faa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78fc1060c0 0x7f78fc073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59236/0 (socket says 172.21.15.16:59236) 2024-04-05T23:33:44.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.531+0000 7f78faa00700 1 -- 172.21.15.16:0/3239793829 learned_addr learned my addr 172.21.15.16:0/3239793829 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:44.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.532+0000 7f78fbe00700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78fc073c60 0x7f78fc1a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:44.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.532+0000 7f78faa00700 1 -- 172.21.15.16:0/3239793829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78fc073c60 msgr2=0x7f78fc1a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.532+0000 7f78faa00700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78fc073c60 0x7f78fc1a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.532+0000 7f78faa00700 1 -- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78fc100fa0 msgr2=0x7f78fc0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.532+0000 7f78faa00700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78fc100fa0 0x7f78fc0731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.532+0000 7f78faa00700 1 -- 172.21.15.16:0/3239793829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ec009520 con 0x7f78fc1060c0 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.532+0000 7f78fb400700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78fc100fa0 0x7f78fc0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.532+0000 7f78faa00700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78fc1060c0 0x7f78fc073720 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f78e8005770 tx=0x7f78e80057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.532+0000 7f78f3e00700 1 -- 172.21.15.16:0/3239793829 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e801d070 con 0x7f78fc1060c0 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.533+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e80095f0 con 0x7f78fc1060c0 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.533+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc1a6380 con 0x7f78fc1060c0 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.533+0000 7f78f3e00700 1 -- 172.21.15.16:0/3239793829 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78e8021410 con 0x7f78fc1060c0 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.533+0000 7f78f3e00700 1 -- 172.21.15.16:0/3239793829 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8017650 con 0x7f78fc1060c0 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.534+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78dc0052f0 con 0x7f78fc1060c0 2024-04-05T23:33:44.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.536+0000 7f78f3e00700 1 -- 172.21.15.16:0/3239793829 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f78e800fb40 con 0x7f78fc1060c0 2024-04-05T23:33:44.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.537+0000 7f78f3e00700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f78e00a13a0 0x7f78e00a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:44.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.537+0000 7f78f3e00700 1 -- 172.21.15.16:0/3239793829 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f78e80cde40 con 0x7f78fc1060c0 2024-04-05T23:33:44.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.537+0000 7f78fb400700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f78e00a13a0 0x7f78e00a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:44.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.538+0000 7f78fb400700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f78e00a13a0 0x7f78e00a3860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009fd0 tx=0x7f78ec009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:44.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.540+0000 7f78f3e00700 1 -- 172.21.15.16:0/3239793829 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f78e8097290 con 0x7f78fc1060c0 2024-04-05T23:33:44.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.809+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78dc005160 con 0x7f78fc1060c0 2024-04-05T23:33:44.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.810+0000 7f78f3e00700 1 -- 172.21.15.16:0/3239793829 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10902 (secure 0 0 0) 0x7f78e809a8b0 con 0x7f78fc1060c0 2024-04-05T23:33:44.803 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:44.803 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:33:36.604775+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3125278540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3125278540},{"type":"v1","addr":"172.21.15.130:6835","nonce":3125278540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:33:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f78e00a13a0 msgr2=0x7f78e00a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f78e00a13a0 0x7f78e00a3860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009fd0 tx=0x7f78ec009260 comp rx=0 tx=0).stop 2024-04-05T23:33:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78fc1060c0 msgr2=0x7f78fc073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78fc1060c0 0x7f78fc073720 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f78e8005770 tx=0x7f78e80057a0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 shutdown_connections 2024-04-05T23:33:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f78e00a13a0 0x7f78e00a3860 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78fc100fa0 0x7f78fc0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78fc073c60 0x7f78fc1a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 --2- 172.21.15.16:0/3239793829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78fc1060c0 0x7f78fc073720 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 >> 172.21.15.16:0/3239793829 conn(0x7f78fc0fc8a0 msgr2=0x7f78fc0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 shutdown_connections 2024-04-05T23:33:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:44.812+0000 7f7902200700 1 -- 172.21.15.16:0/3239793829 wait complete. 2024-04-05T23:33:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-04-05T23:33:44.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:33:44.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:44 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/94491505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:44.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:44 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2695725799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:44.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:44 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/94491505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:44.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:44 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2695725799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:45.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.032+0000 7f860ce00700 1 -- 172.21.15.16:0/1390730652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f860810d560 msgr2=0x7f860810f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.032+0000 7f860ce00700 1 --2- 172.21.15.16:0/1390730652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f860810d560 0x7f860810f9d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f860000b0a0 tx=0x7f860000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.032+0000 7f860ce00700 1 -- 172.21.15.16:0/1390730652 shutdown_connections 2024-04-05T23:33:45.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.032+0000 7f860ce00700 1 --2- 172.21.15.16:0/1390730652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f860810d560 0x7f860810f9d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.032+0000 7f860ce00700 1 --2- 172.21.15.16:0/1390730652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8608073310 0x7f86080736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.032+0000 7f860ce00700 1 --2- 172.21.15.16:0/1390730652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8608073c30 0x7f860810d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.032+0000 7f860ce00700 1 -- 172.21.15.16:0/1390730652 >> 172.21.15.16:0/1390730652 conn(0x7f86080fc830 msgr2=0x7f86080fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.032+0000 7f860ce00700 1 -- 172.21.15.16:0/1390730652 shutdown_connections 2024-04-05T23:33:45.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.032+0000 7f860ce00700 1 -- 172.21.15.16:0/1390730652 wait complete. 2024-04-05T23:33:45.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.034+0000 7f860ce00700 1 Processor -- start 2024-04-05T23:33:45.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.034+0000 7f860ce00700 1 -- start start 2024-04-05T23:33:45.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.034+0000 7f860ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8608073310 0x7f8608199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f860ce00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8608073c30 0x7f8608199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f860ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860810d560 0x7f860819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f860ce00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860819e4a0 con 0x7f8608073310 2024-04-05T23:33:45.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f860ce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860819e610 con 0x7f860810d560 2024-04-05T23:33:45.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f860ce00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860819e7d0 con 0x7f8608073c30 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8605e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8608073310 0x7f8608199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8605e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8608073310 0x7f8608199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40888/0 (socket says 172.21.15.16:40888) 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8605e00700 1 -- 172.21.15.16:0/2365950119 learned_addr learned my addr 172.21.15.16:0/2365950119 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8606800700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860810d560 0x7f860819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8605400700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8608073c30 0x7f8608199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8605e00700 1 -- 172.21.15.16:0/2365950119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8608073c30 msgr2=0x7f8608199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8605e00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8608073c30 0x7f8608199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8605e00700 1 -- 172.21.15.16:0/2365950119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860810d560 msgr2=0x7f860819de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8605e00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860810d560 0x7f860819de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.035+0000 7f8605e00700 1 -- 172.21.15.16:0/2365950119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8600009d00 con 0x7f8608073310 2024-04-05T23:33:45.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.036+0000 7f8605e00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8608073310 0x7f8608199580 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f85f800e9a0 tx=0x7f85f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.036+0000 7f85fea00700 1 -- 172.21.15.16:0/2365950119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8009790 con 0x7f8608073310 2024-04-05T23:33:45.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.036+0000 7f85fea00700 1 -- 172.21.15.16:0/2365950119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f85f8010430 con 0x7f8608073310 2024-04-05T23:33:45.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.036+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860819eab0 con 0x7f8608073310 2024-04-05T23:33:45.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.036+0000 7f85fea00700 1 -- 172.21.15.16:0/2365950119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8018680 con 0x7f8608073310 2024-04-05T23:33:45.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.036+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86081a62f0 con 0x7f8608073310 2024-04-05T23:33:45.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.037+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8608108770 con 0x7f8608073310 2024-04-05T23:33:45.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.039+0000 7f85fea00700 1 -- 172.21.15.16:0/2365950119 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f85f80105a0 con 0x7f8608073310 2024-04-05T23:33:45.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.040+0000 7f85fea00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f85ec0a13a0 0x7f85ec0a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.041+0000 7f85fea00700 1 -- 172.21.15.16:0/2365950119 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f85f8014070 con 0x7f8608073310 2024-04-05T23:33:45.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.041+0000 7f8605400700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f85ec0a13a0 0x7f85ec0a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.041+0000 7f8605400700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f85ec0a13a0 0x7f85ec0a3860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f860819aad0 tx=0x7f85f4009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.045+0000 7f85fea00700 1 -- 172.21.15.16:0/2365950119 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f85f8052ab0 con 0x7f8608073310 2024-04-05T23:33:45.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:44 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/94491505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:33:45.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:44 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2695725799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:33:45.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.265+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86080749c0 con 0x7f8608073310 2024-04-05T23:33:45.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.266+0000 7f85fea00700 1 -- 172.21.15.16:0/2365950119 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22958 (secure 0 0 0) 0x7f85f80962b0 con 0x7f8608073310 2024-04-05T23:33:45.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:33:45.259 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:33:34.593258+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/985211078":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1740984099":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/4157043736":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/1565223355":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2846240830":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/4184690359":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/1565223355":"2024-04-06T23:33:34.593227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f85ec0a13a0 msgr2=0x7f85ec0a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f85ec0a13a0 0x7f85ec0a3860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f860819aad0 tx=0x7f85f4009260 comp rx=0 tx=0).stop 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8608073310 msgr2=0x7f8608199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8608073310 0x7f8608199580 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f85f800e9a0 tx=0x7f85f800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 shutdown_connections 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f85ec0a13a0 0x7f85ec0a3860 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8608073310 0x7f8608199580 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860810d560 0x7f860819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 --2- 172.21.15.16:0/2365950119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8608073c30 0x7f8608199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 >> 172.21.15.16:0/2365950119 conn(0x7f86080fc830 msgr2=0x7f86080fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 shutdown_connections 2024-04-05T23:33:45.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.269+0000 7f860ce00700 1 -- 172.21.15.16:0/2365950119 wait complete. 2024-04-05T23:33:45.281 DEBUG:tasks.fwd_scrub:Yielding 2024-04-05T23:33:45.282 INFO:teuthology.run_tasks:Running task exec... 2024-04-05T23:33:45.292 INFO:teuthology.task.exec:Executing custom commands... 2024-04-05T23:33:45.292 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:33:45.292 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-05T23:33:45.293 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-05T23:33:45.293 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-05T23:33:45.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-05T23:33:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.504+0000 7f6e8b000700 1 -- 172.21.15.16:0/798523298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84106f40 msgr2=0x7f6e84107320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.504+0000 7f6e8b000700 1 --2- 172.21.15.16:0/798523298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84106f40 0x7f6e84107320 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009990 tx=0x7f6e74009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.504+0000 7f6e8b000700 1 -- 172.21.15.16:0/798523298 shutdown_connections 2024-04-05T23:33:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.504+0000 7f6e8b000700 1 --2- 172.21.15.16:0/798523298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e84068be0 0x7f6e8410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.504+0000 7f6e8b000700 1 --2- 172.21.15.16:0/798523298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84106f40 0x7f6e84107320 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.504+0000 7f6e8b000700 1 --2- 172.21.15.16:0/798523298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e84068220 0x7f6e840686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.504+0000 7f6e8b000700 1 -- 172.21.15.16:0/798523298 >> 172.21.15.16:0/798523298 conn(0x7f6e84076e50 msgr2=0x7f6e84077260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.504+0000 7f6e8b000700 1 -- 172.21.15.16:0/798523298 shutdown_connections 2024-04-05T23:33:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.504+0000 7f6e8b000700 1 -- 172.21.15.16:0/798523298 wait complete. 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.505+0000 7f6e8b000700 1 Processor -- start 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.505+0000 7f6e8b000700 1 -- start start 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e8b000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84068220 0x7f6e840ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e8b000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e84068be0 0x7f6e84100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e8b000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e84106f40 0x7f6e841007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e8b000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84101d20 con 0x7f6e84068be0 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e8b000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e841a9bd0 con 0x7f6e84106f40 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e8b000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e841a9d90 con 0x7f6e84068220 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e83e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84068220 0x7f6e840ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e83400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e84068be0 0x7f6e84100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e89000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e84106f40 0x7f6e841007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e83e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84068220 0x7f6e840ffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:48880/0 (socket says 172.21.15.16:48880) 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.506+0000 7f6e83e00700 1 -- 172.21.15.16:0/2763931769 learned_addr learned my addr 172.21.15.16:0/2763931769 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e89000700 1 -- 172.21.15.16:0/2763931769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84068220 msgr2=0x7f6e840ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e89000700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84068220 0x7f6e840ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e89000700 1 -- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e84068be0 msgr2=0x7f6e84100290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e89000700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e84068be0 0x7f6e84100290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e89000700 1 -- 172.21.15.16:0/2763931769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e740095f0 con 0x7f6e84106f40 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e83e00700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84068220 0x7f6e840ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e83400700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e84068be0 0x7f6e84100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e89000700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e84106f40 0x7f6e841007d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00efc0 tx=0x7f6e7c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e80c00700 1 -- 172.21.15.16:0/2763931769 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c018070 con 0x7f6e84106f40 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e80c00700 1 -- 172.21.15.16:0/2763931769 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e7c00f040 con 0x7f6e84106f40 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e80c00700 1 -- 172.21.15.16:0/2763931769 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c00cd40 con 0x7f6e84106f40 2024-04-05T23:33:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e841aa010 con 0x7f6e84106f40 2024-04-05T23:33:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.507+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e841aa520 con 0x7f6e84106f40 2024-04-05T23:33:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e841092c0 con 0x7f6e84106f40 2024-04-05T23:33:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f8a79000700 1 -- 172.21.15.16:0/2663632691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a74073c30 msgr2=0x7f8a7410d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f8a79000700 1 --2- 172.21.15.16:0/2663632691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a74073c30 0x7f8a7410d130 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8a600098f0 tx=0x7f8a60009c00 comp rx=0 tx=0).stop 2024-04-05T23:33:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f8a79000700 1 -- 172.21.15.16:0/2663632691 shutdown_connections 2024-04-05T23:33:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f8a79000700 1 --2- 172.21.15.16:0/2663632691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a7410d670 0x7f8a7410fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f8a79000700 1 --2- 172.21.15.16:0/2663632691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a74073310 0x7f8a740736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f8a79000700 1 --2- 172.21.15.16:0/2663632691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a74073c30 0x7f8a7410d130 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f8a79000700 1 -- 172.21.15.16:0/2663632691 >> 172.21.15.16:0/2663632691 conn(0x7f8a740fc900 msgr2=0x7f8a740fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f8a79000700 1 -- 172.21.15.16:0/2663632691 shutdown_connections 2024-04-05T23:33:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.509+0000 7f8a79000700 1 -- 172.21.15.16:0/2663632691 wait complete. 2024-04-05T23:33:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.510+0000 7f8a79000700 1 Processor -- start 2024-04-05T23:33:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a79000700 1 -- start start 2024-04-05T23:33:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f6e80c00700 1 -- 172.21.15.16:0/2763931769 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f6e7c003a00 con 0x7f6e84106f40 2024-04-05T23:33:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a79000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a74073310 0x7f8a741995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a79000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a74073c30 0x7f8a74199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a79000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a7410d670 0x7f8a7419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a79000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7419e4b0 con 0x7f8a7410d670 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a79000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7419e620 con 0x7f8a74073310 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a79000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7419e7e0 con 0x7f8a74073c30 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a71e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a74073310 0x7f8a741995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a71e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a74073310 0x7f8a741995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59292/0 (socket says 172.21.15.16:59292) 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a71e00700 1 -- 172.21.15.16:0/292541312 learned_addr learned my addr 172.21.15.16:0/292541312 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a71400700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a74073c30 0x7f8a74199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.511+0000 7f8a72800700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a7410d670 0x7f8a7419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a71400700 1 -- 172.21.15.16:0/292541312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a74073310 msgr2=0x7f8a741995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a71400700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a74073310 0x7f8a741995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a71400700 1 -- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a7410d670 msgr2=0x7f8a7419de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a71400700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a7410d670 0x7f8a7419de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a71400700 1 -- 172.21.15.16:0/292541312 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a60009520 con 0x7f8a74073c30 2024-04-05T23:33:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a71e00700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a74073310 0x7f8a741995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a71400700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a74073c30 0x7f8a74199b20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009810 tx=0x7f8a6000fa80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a6aa00700 1 -- 172.21.15.16:0/292541312 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6000ba70 con 0x7f8a74073c30 2024-04-05T23:33:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a79000700 1 -- 172.21.15.16:0/292541312 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a7419ea60 con 0x7f8a74073c30 2024-04-05T23:33:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a6aa00700 1 -- 172.21.15.16:0/292541312 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a6000bbd0 con 0x7f8a74073c30 2024-04-05T23:33:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f8a6aa00700 1 -- 172.21.15.16:0/292541312 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60017400 con 0x7f8a74073c30 2024-04-05T23:33:45.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.512+0000 7f6e80c00700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6e680a13a0 0x7f6e680a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.513+0000 7f8a79000700 1 -- 172.21.15.16:0/292541312 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a741a6300 con 0x7f8a74073c30 2024-04-05T23:33:45.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.513+0000 7f6e80c00700 1 -- 172.21.15.16:0/2763931769 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f6e7c0ccab0 con 0x7f6e84106f40 2024-04-05T23:33:45.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.513+0000 7f6e83e00700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6e680a13a0 0x7f6e680a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.513+0000 7f6e83e00700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6e680a13a0 0x7f6e680a3860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6e74006010 tx=0x7f6e7400b280 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.514+0000 7f6e80c00700 1 -- 172.21.15.16:0/2763931769 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6e7c095df0 con 0x7f6e84106f40 2024-04-05T23:33:45.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.516+0000 7f8a6aa00700 1 -- 172.21.15.16:0/292541312 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f8a60003710 con 0x7f8a74073c30 2024-04-05T23:33:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.517+0000 7f8a6aa00700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8a580a1320 0x7f8a580a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.517+0000 7f8a8cb48b80 1 Processor -- start 2024-04-05T23:33:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.517+0000 7f8a6aa00700 1 -- 172.21.15.16:0/292541312 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f8a600cd910 con 0x7f8a74073c30 2024-04-05T23:33:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.517+0000 7f8a71e00700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8a580a1320 0x7f8a580a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a8cb48b80 1 -- start start 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba995e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba937e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fb9f9c60 con 0x5585fba93d20 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fb9eaa10 con 0x5585fba99200 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fb9e83d0 con 0x5585fba93360 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a71e00700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8a580a1320 0x7f8a580a37e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8a64005950 tx=0x7f8a64009310 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a71e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a72800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a71e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59306/0 (socket says 172.21.15.16:59306) 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a71e00700 1 -- 172.21.15.16:0/3006010045 learned_addr learned my addr 172.21.15.16:0/3006010045 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.518+0000 7f8a71400700 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba937e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.519+0000 7f8a71e00700 1 -- 172.21.15.16:0/3006010045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 msgr2=0x5585fba937e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.519+0000 7f8a71e00700 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba937e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.519+0000 7f8a71e00700 1 -- 172.21.15.16:0/3006010045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 msgr2=0x5585fba9ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.519+0000 7f8a71e00700 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.519+0000 7f8a71e00700 1 -- 172.21.15.16:0/3006010045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585fb865430 con 0x5585fba99200 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.519+0000 7f8a71400700 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba937e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.519+0000 7f8a72800700 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.519+0000 7f8a71e00700 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba995e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400e6b0 tx=0x7f8a64014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66f994277af9cfa server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.519+0000 7f8a5f400700 1 -- 172.21.15.16:0/3006010045 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a64009df0 con 0x5585fba99200 2024-04-05T23:33:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a5f400700 1 -- 172.21.15.16:0/3006010045 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a64021070 con 0x5585fba99200 2024-04-05T23:33:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a5f400700 1 -- 172.21.15.16:0/3006010045 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a64025870 con 0x5585fba99200 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/3006010045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 msgr2=0x5585fba995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba995e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400e6b0 tx=0x7f8a64014000 comp rx=0 tx=0).stop 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/3006010045 shutdown_connections 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba995e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/3006010045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba937e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/3006010045 >> 172.21.15.16:0/3006010045 conn(0x5585fba072a0 msgr2=0x5585fba8e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.520+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/3006010045 shutdown_connections 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.521+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/3006010045 wait complete. 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.521+0000 7f8a8cb48b80 1 -- start start 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.521+0000 7f8a8cb48b80 1 Processor -- start 2024-04-05T23:33:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.522+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba9fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.522+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.522+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba9d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.522+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fb9e3cc0 con 0x5585fba93d20 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.522+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fb97fe00 con 0x5585fba99200 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.522+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fb97faa0 con 0x5585fba93360 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.522+0000 7f8a8cb48b80 10 client.0 _ll_register_callbacks cb 0x5585fb7eb420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.522+0000 7f8a71400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a72800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba9d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a71400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40950/0 (socket says 172.21.15.16:40950) 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a71400700 1 -- 172.21.15.16:0/610946383 learned_addr learned my addr 172.21.15.16:0/610946383 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a71e00700 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba9fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a72800700 1 -- 172.21.15.16:0/610946383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 msgr2=0x5585fba9fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a72800700 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba9fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a72800700 1 -- 172.21.15.16:0/610946383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 msgr2=0x5585fba9d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a72800700 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a72800700 1 -- 172.21.15.16:0/610946383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c011230 con 0x5585fba99200 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a72800700 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba9d700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c007c90 tx=0x7f8a6c0113c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d03b1bf870721bd server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a53e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a53e00700 1 -- 172.21.15.16:0/610946383 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c00e040 con 0x5585fba99200 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.523+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/610946383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5585fba86510 con 0x5585fba99200 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a53e00700 1 -- 172.21.15.16:0/610946383 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f8a6c004db0 con 0x5585fba99200 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a53e00700 1 -- 172.21.15.16:0/610946383 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a6c011b70 con 0x5585fba99200 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a8cb48b80 10 client.25445 fetch_fsmap learned FSMap version 28 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/610946383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8a64017070 con 0x5585fba99200 2024-04-05T23:33:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a53e00700 1 -- 172.21.15.16:0/610946383 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5585fba86510 con 0x5585fba99200 2024-04-05T23:33:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a53e00700 1 -- 172.21.15.16:0/610946383 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f8a6c005020 con 0x5585fba99200 2024-04-05T23:33:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a8cb48b80 10 client.25445 fetch_fsmap finished waiting for FSMap version 28 2024-04-05T23:33:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a8cb48b80 10 client.25445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:33:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a8cb48b80 20 client.25445 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5585fbaad4c0 0x5585fbaaf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a8cb48b80 4 client.25445 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.524+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/610946383 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585fb9dc780 con 0x5585fbaad4c0 2024-04-05T23:33:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.525+0000 7f8a71e00700 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5585fbaad4c0 0x5585fbaaf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.525+0000 7f8a71e00700 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5585fbaad4c0 0x5585fbaaf8e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcad8c1258dbca2b server_cookie=7cf0bf8b4f9d893a in_seq=0 out_seq=0 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.525+0000 7f8a53e00700 10 client.25445 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a53e00700 1 -- 172.21.15.16:0/610946383 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5585fb9dc780 con 0x5585fbaad4c0 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a53e00700 10 client.25445 handle_command_reply: tid=0 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a8cb48b80 1 client.25445 shutdown 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/610946383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 msgr2=0x5585fba9d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba9d700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c007c90 tx=0x7f8a6c0113c0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/610946383 shutdown_connections 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5585fbaad4c0 0x5585fbaaf8e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fba9d700 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba9fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/610946383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a53e00700 0 client.25445 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.526+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/610946383 >> 172.21.15.16:0/610946383 conn(0x5585fba072a0 msgr2=0x5585fba884e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.527+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/610946383 shutdown_connections 2024-04-05T23:33:45.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.527+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/610946383 wait complete. 2024-04-05T23:33:45.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.527+0000 7f8a8cb48b80 20 client.25445 trim_cache size 0 max 16384 2024-04-05T23:33:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.544+0000 7f8a8cb48b80 1 Processor -- start 2024-04-05T23:33:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.544+0000 7f8a8cb48b80 1 -- start start 2024-04-05T23:33:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba8e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fbaa93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fb867720 con 0x5585fba93d20 2024-04-05T23:33:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fba86080 con 0x5585fba99200 2024-04-05T23:33:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fba8f980 con 0x5585fba93360 2024-04-05T23:33:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a72800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fbaa93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a71400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a71e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba8e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a72800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fbaa93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59324/0 (socket says 172.21.15.16:59324) 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a72800700 1 -- 172.21.15.16:0/2899521034 learned_addr learned my addr 172.21.15.16:0/2899521034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a72800700 1 -- 172.21.15.16:0/2899521034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 msgr2=0x5585fba8e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a72800700 1 --2- 172.21.15.16:0/2899521034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba8e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a72800700 1 -- 172.21.15.16:0/2899521034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 msgr2=0x5585fba9d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a72800700 1 --2- 172.21.15.16:0/2899521034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a72800700 1 -- 172.21.15.16:0/2899521034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585fb811be0 con 0x5585fba99200 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.545+0000 7f8a71e00700 1 --2- 172.21.15.16:0/2899521034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba8e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:45.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.546+0000 7f8a72800700 1 --2- 172.21.15.16:0/2899521034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fbaa93f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c005380 tx=0x7f8a6c011790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d61fdeb53430ad18 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.546+0000 7f8a5fe00700 1 -- 172.21.15.16:0/2899521034 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c00e070 con 0x5585fba99200 2024-04-05T23:33:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.546+0000 7f8a5fe00700 1 -- 172.21.15.16:0/2899521034 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a6c00f040 con 0x5585fba99200 2024-04-05T23:33:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.546+0000 7f8a5fe00700 1 -- 172.21.15.16:0/2899521034 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c029470 con 0x5585fba99200 2024-04-05T23:33:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/2899521034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 msgr2=0x5585fbaa93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/2899521034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fbaa93f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c005380 tx=0x7f8a6c011790 comp rx=0 tx=0).stop 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/2899521034 shutdown_connections 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/2899521034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba99200 0x5585fbaa93f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/2899521034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fba8e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/2899521034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba93d20 0x5585fba9d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/2899521034 >> 172.21.15.16:0/2899521034 conn(0x5585fba072a0 msgr2=0x5585fbaa8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/2899521034 shutdown_connections 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/2899521034 wait complete. 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 -- start start 2024-04-05T23:33:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.547+0000 7f8a8cb48b80 1 Processor -- start 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fbaa3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba93d20 0x5585fbaa3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a8cb48b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba99200 0x5585fbafb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fbafb840 con 0x5585fba99200 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fbafb9b0 con 0x5585fba93d20 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a8cb48b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585fbafbb70 con 0x5585fba93360 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a8cb48b80 10 client.0 _ll_register_callbacks cb 0x7f8a6c0075e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a71400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba93d20 0x5585fbaa3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a71e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fbaa3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a71400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba93d20 0x5585fbaa3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59328/0 (socket says 172.21.15.16:59328) 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a71e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fbaa3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:48918/0 (socket says 172.21.15.16:48918) 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a71400700 1 -- 172.21.15.16:0/1134668912 learned_addr learned my addr 172.21.15.16:0/1134668912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.548+0000 7f8a72800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba99200 0x5585fbafb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a71400700 1 -- 172.21.15.16:0/1134668912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 msgr2=0x5585fbaa3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a71400700 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fbaa3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a71400700 1 -- 172.21.15.16:0/1134668912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba99200 msgr2=0x5585fbafb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a71400700 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba99200 0x5585fbafb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a71400700 1 -- 172.21.15.16:0/1134668912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6402a040 con 0x5585fba93d20 2024-04-05T23:33:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a71e00700 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fbaa3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a71400700 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba93d20 0x5585fbaa3bd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a600ba010 tx=0x7f8a60003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d986f310ef8e602 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a53e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:33:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a53e00700 1 -- 172.21.15.16:0/1134668912 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6009a270 con 0x5585fba93d20 2024-04-05T23:33:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a53e00700 1 -- 172.21.15.16:0/1134668912 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f8a6009b010 con 0x5585fba93d20 2024-04-05T23:33:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.549+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/1134668912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a64021070 con 0x5585fba93d20 2024-04-05T23:33:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a53e00700 1 -- 172.21.15.16:0/1134668912 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a6009a910 con 0x5585fba93d20 2024-04-05T23:33:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a8cb48b80 10 client.25451 fetch_fsmap learned FSMap version 28 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/1134668912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8a64017070 con 0x5585fba93d20 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a53e00700 1 -- 172.21.15.16:0/1134668912 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a60064020 con 0x5585fba93d20 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a53e00700 1 -- 172.21.15.16:0/1134668912 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f8a60099e00 con 0x5585fba93d20 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a8cb48b80 10 client.25451 fetch_fsmap finished waiting for FSMap version 28 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a8cb48b80 10 client.25451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a8cb48b80 20 client.25451 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5585fbb03130 0x5585fbb05550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a8cb48b80 4 client.25451 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/1134668912 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5585fb832d70 con 0x5585fbb03130 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.550+0000 7f8a71e00700 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5585fbb03130 0x5585fbb05550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.551+0000 7f8a71e00700 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5585fbb03130 0x5585fbb05550 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b490bb27c19e57f8 server_cookie=afee3523400f06c4 in_seq=0 out_seq=0 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.551+0000 7f8a53e00700 10 client.25451 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a53e00700 1 -- 172.21.15.16:0/1134668912 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5585fb832d70 con 0x5585fbb03130 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a53e00700 10 client.25451 handle_command_reply: tid=0 2024-04-05T23:33:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a8cb48b80 1 client.25451 shutdown 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/1134668912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba93d20 msgr2=0x5585fbaa3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba93d20 0x5585fbaa3bd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a600ba010 tx=0x7f8a60003680 comp rx=0 tx=0).stop 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/1134668912 shutdown_connections 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5585fbb03130 0x5585fbb05550 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585fba99200 0x5585fbafb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585fba93360 0x5585fbaa3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a8cb48b80 1 --2- 172.21.15.16:0/1134668912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585fba93d20 0x5585fbaa3bd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a53e00700 0 client.25451 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.552+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/1134668912 >> 172.21.15.16:0/1134668912 conn(0x5585fba072a0 msgr2=0x5585fba92be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.553+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/1134668912 shutdown_connections 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.553+0000 7f8a8cb48b80 1 -- 172.21.15.16:0/1134668912 wait complete. 2024-04-05T23:33:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.553+0000 7f8a8cb48b80 20 client.25451 trim_cache size 0 max 16384 2024-04-05T23:33:45.549 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:45.549 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-04-05T23:33:45.549 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "e409aeb5-ba51-4f56-9a8d-bbd2205182e1", 2024-04-05T23:33:45.549 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-04-05T23:33:45.549 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.553+0000 7f8a79000700 1 -- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8a580a1320 msgr2=0x7f8a580a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.553+0000 7f8a79000700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8a580a1320 0x7f8a580a37e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8a64005950 tx=0x7f8a64009310 comp rx=0 tx=0).stop 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.553+0000 7f8a79000700 1 -- 172.21.15.16:0/292541312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a74073c30 msgr2=0x7f8a74199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.553+0000 7f8a79000700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a74073c30 0x7f8a74199b20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009810 tx=0x7f8a6000fa80 comp rx=0 tx=0).stop 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.554+0000 7f8a79000700 1 -- 172.21.15.16:0/292541312 shutdown_connections 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.554+0000 7f8a79000700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8a580a1320 0x7f8a580a37e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.554+0000 7f8a79000700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a7410d670 0x7f8a7419de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.554+0000 7f8a79000700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a74073310 0x7f8a741995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.554+0000 7f8a79000700 1 --2- 172.21.15.16:0/292541312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a74073c30 0x7f8a74199b20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.554+0000 7f8a79000700 1 -- 172.21.15.16:0/292541312 >> 172.21.15.16:0/292541312 conn(0x7f8a740fc900 msgr2=0x7f8a740fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.554+0000 7f8a79000700 1 -- 172.21.15.16:0/292541312 shutdown_connections 2024-04-05T23:33:45.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.554+0000 7f8a79000700 1 -- 172.21.15.16:0/292541312 wait complete. 2024-04-05T23:33:45.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-05T23:33:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.773+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 --> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e84110d30 con 0x7f6e680a13a0 2024-04-05T23:33:45.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.777+0000 7f0912c00700 1 -- 172.21.15.16:0/3742249100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f090c068220 msgr2=0x7f090c068600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.777+0000 7f0912c00700 1 --2- 172.21.15.16:0/3742249100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f090c068220 0x7f090c068600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009990 tx=0x7f08fc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.777+0000 7f0912c00700 1 -- 172.21.15.16:0/3742249100 shutdown_connections 2024-04-05T23:33:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.777+0000 7f0912c00700 1 --2- 172.21.15.16:0/3742249100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c1061d0 0x7f090c10dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.777+0000 7f0912c00700 1 --2- 172.21.15.16:0/3742249100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f090c068220 0x7f090c068600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.777+0000 7f0912c00700 1 --2- 172.21.15.16:0/3742249100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f090c068bd0 0x7f090c105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.777+0000 7f0912c00700 1 -- 172.21.15.16:0/3742249100 >> 172.21.15.16:0/3742249100 conn(0x7f090c076e80 msgr2=0x7f090c077290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.777+0000 7f0912c00700 1 -- 172.21.15.16:0/3742249100 shutdown_connections 2024-04-05T23:33:45.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.778+0000 7f0912c00700 1 -- 172.21.15.16:0/3742249100 wait complete. 2024-04-05T23:33:45.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.778+0000 7f0912c00700 1 Processor -- start 2024-04-05T23:33:45.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f0912c00700 1 -- start start 2024-04-05T23:33:45.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f0912c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f090c068220 0x7f090c197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f0912c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c1061d0 0x7f090c197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f0912c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f090c19bc00 0x7f090c19c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f0912c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c19c680 con 0x7f090c068220 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f0912c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c19c7f0 con 0x7f090c19bc00 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f0912c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c19c9b0 con 0x7f090c1061d0 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f090be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f090c068220 0x7f090c197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f0910c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f090c19bc00 0x7f090c19c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f090b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c1061d0 0x7f090c197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f090be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f090c068220 0x7f090c197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40980/0 (socket says 172.21.15.16:40980) 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f090be00700 1 -- 172.21.15.16:0/4020753514 learned_addr learned my addr 172.21.15.16:0/4020753514 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.779+0000 7f0910c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f090c19bc00 0x7f090c19c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59350/0 (socket says 172.21.15.16:59350) 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.780+0000 7f090b400700 1 -- 172.21.15.16:0/4020753514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f090c19bc00 msgr2=0x7f090c19c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.780+0000 7f090b400700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f090c19bc00 0x7f090c19c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.780+0000 7f090b400700 1 -- 172.21.15.16:0/4020753514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f090c068220 msgr2=0x7f090c197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.780+0000 7f090b400700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f090c068220 0x7f090c197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.780+0000 7f090b400700 1 -- 172.21.15.16:0/4020753514 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc0095f0 con 0x7f090c1061d0 2024-04-05T23:33:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.780+0000 7f090b400700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c1061d0 0x7f090c197860 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f08f800e9c0 tx=0x7f08f800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.781+0000 7f0908c00700 1 -- 172.21.15.16:0/4020753514 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f800c9c0 con 0x7f090c1061d0 2024-04-05T23:33:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.781+0000 7f0908c00700 1 -- 172.21.15.16:0/4020753514 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08f8010430 con 0x7f090c1061d0 2024-04-05T23:33:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.781+0000 7f0912c00700 1 -- 172.21.15.16:0/4020753514 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090c1a3f10 con 0x7f090c1061d0 2024-04-05T23:33:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.781+0000 7f0908c00700 1 -- 172.21.15.16:0/4020753514 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8018680 con 0x7f090c1061d0 2024-04-05T23:33:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.781+0000 7f0912c00700 1 -- 172.21.15.16:0/4020753514 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f090c1a43e0 con 0x7f090c1061d0 2024-04-05T23:33:45.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.786+0000 7f0908c00700 1 -- 172.21.15.16:0/4020753514 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f08f800cb20 con 0x7f090c1061d0 2024-04-05T23:33:45.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.787+0000 7f09267e0b80 1 Processor -- start 2024-04-05T23:33:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.787+0000 7f09267e0b80 1 -- start start 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56354564b590 0x56354564b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 0x563545645b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x563545652320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f09267e0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635455a7be0 con 0x5635456460b0 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f09267e0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563545598990 con 0x56354564b590 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f09267e0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563545596350 con 0x5635456456f0 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f0910c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56354564b590 0x56354564b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f0910c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56354564b590 0x56354564b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59352/0 (socket says 172.21.15.16:59352) 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f0910c00700 1 -- 172.21.15.16:0/4123134975 learned_addr learned my addr 172.21.15.16:0/4123134975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f090be00700 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 0x563545645b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f090b400700 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x563545652320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f0910c00700 1 -- 172.21.15.16:0/4123134975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 msgr2=0x563545645b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f0910c00700 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 0x563545645b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f0910c00700 1 -- 172.21.15.16:0/4123134975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 msgr2=0x563545652320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f0910c00700 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x563545652320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f0910c00700 1 -- 172.21.15.16:0/4123134975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635454133b0 con 0x56354564b590 2024-04-05T23:33:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f0908c00700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f08f00a13a0 0x7f08f00a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.788+0000 7f090be00700 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 0x563545645b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.789+0000 7f0908c00700 1 -- 172.21.15.16:0/4020753514 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f08f8014070 con 0x7f090c1061d0 2024-04-05T23:33:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.789+0000 7f0910c00700 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56354564b590 0x56354564b970 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0904011680 tx=0x7f0904009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f13d9ea744035c82 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.789+0000 7f090be00700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f08f00a13a0 0x7f08f00a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.789+0000 7f0901200700 1 -- 172.21.15.16:0/4123134975 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090400e040 con 0x56354564b590 2024-04-05T23:33:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.789+0000 7f0901200700 1 -- 172.21.15.16:0/4123134975 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f090400f040 con 0x56354564b590 2024-04-05T23:33:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.789+0000 7f0901200700 1 -- 172.21.15.16:0/4123134975 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09040048c0 con 0x56354564b590 2024-04-05T23:33:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f09267e0b80 1 -- 172.21.15.16:0/4123134975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56354564b590 msgr2=0x56354564b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56354564b590 0x56354564b970 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0904011680 tx=0x7f0904009920 comp rx=0 tx=0).stop 2024-04-05T23:33:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f090be00700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f08f00a13a0 0x7f08f00a3860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x563545636ca0 tx=0x7f08fc005bc0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f09267e0b80 1 -- 172.21.15.16:0/4123134975 shutdown_connections 2024-04-05T23:33:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x563545652320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56354564b590 0x56354564b970 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4123134975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 0x563545645b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f09267e0b80 1 -- 172.21.15.16:0/4123134975 >> 172.21.15.16:0/4123134975 conn(0x5635455b5220 msgr2=0x56354563c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f09267e0b80 1 -- 172.21.15.16:0/4123134975 shutdown_connections 2024-04-05T23:33:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.790+0000 7f09267e0b80 1 -- 172.21.15.16:0/4123134975 wait complete. 2024-04-05T23:33:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.791+0000 7f09267e0b80 1 -- start start 2024-04-05T23:33:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.791+0000 7f09267e0b80 1 Processor -- start 2024-04-05T23:33:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x56354564c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x56354564c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56354564b590 0x56354564cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f09267e0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563545591c40 con 0x5635456460b0 2024-04-05T23:33:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f09267e0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56354552dd80 con 0x5635456456f0 2024-04-05T23:33:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f09267e0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56354552da20 con 0x56354564b590 2024-04-05T23:33:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f09267e0b80 10 client.0 _ll_register_callbacks cb 0x5635453993a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f090b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x56354564c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f090be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x56354564c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.792+0000 7f090be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x56354564c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59356/0 (socket says 172.21.15.16:59356) 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f090be00700 1 -- 172.21.15.16:0/4245949815 learned_addr learned my addr 172.21.15.16:0/4245949815 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f090b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x56354564c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40988/0 (socket says 172.21.15.16:40988) 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f0910c00700 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56354564b590 0x56354564cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f0910c00700 1 -- 172.21.15.16:0/4245949815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 msgr2=0x56354564c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f0910c00700 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x56354564c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f0910c00700 1 -- 172.21.15.16:0/4245949815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 msgr2=0x56354564c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f0910c00700 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x56354564c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f0910c00700 1 -- 172.21.15.16:0/4245949815 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0904011330 con 0x56354564b590 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f090be00700 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x56354564c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f0910c00700 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56354564b590 0x56354564cce0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0904012010 tx=0x7f0904003c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a23af1330e9adbb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.793+0000 7f08f5600700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:33:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.794+0000 7f08f5600700 1 -- 172.21.15.16:0/4245949815 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090400e030 con 0x56354564b590 2024-04-05T23:33:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.794+0000 7f08f5600700 1 -- 172.21.15.16:0/4245949815 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f09040093a0 con 0x56354564b590 2024-04-05T23:33:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.794+0000 7f09267e0b80 1 -- 172.21.15.16:0/4245949815 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563545634490 con 0x56354564b590 2024-04-05T23:33:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.795+0000 7f08f5600700 1 -- 172.21.15.16:0/4245949815 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09040041e0 con 0x56354564b590 2024-04-05T23:33:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.795+0000 7f09267e0b80 10 client.25483 fetch_fsmap learned FSMap version 28 2024-04-05T23:33:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.795+0000 7f09267e0b80 1 -- 172.21.15.16:0/4245949815 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5635453bfb60 con 0x56354564b590 2024-04-05T23:33:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.795+0000 7f08f5600700 1 -- 172.21.15.16:0/4245949815 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563545634490 con 0x56354564b590 2024-04-05T23:33:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.796+0000 7f08f5600700 1 -- 172.21.15.16:0/4245949815 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f0904024020 con 0x56354564b590 2024-04-05T23:33:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.796+0000 7f09267e0b80 10 client.25483 fetch_fsmap finished waiting for FSMap version 28 2024-04-05T23:33:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.796+0000 7f09267e0b80 10 client.25483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:33:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.796+0000 7f09267e0b80 20 client.25483 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.796+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56354565b3c0 0x56354565d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.796+0000 7f09267e0b80 4 client.25483 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:33:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.796+0000 7f09267e0b80 1 -- 172.21.15.16:0/4245949815 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56354558a700 con 0x56354565b3c0 2024-04-05T23:33:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.797+0000 7f090be00700 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56354565b3c0 0x56354565d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.798+0000 7f08f5600700 10 client.25483 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:45.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.798+0000 7f090be00700 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56354565b3c0 0x56354565d7b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e844000eefb371b8 server_cookie=50715e0cc3e72fdf in_seq=0 out_seq=0 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f08f5600700 1 -- 172.21.15.16:0/4245949815 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56354558a700 con 0x56354565b3c0 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f08f5600700 10 client.25483 handle_command_reply: tid=0 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 client.25483 shutdown 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 -- 172.21.15.16:0/4245949815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56354564b590 msgr2=0x56354564cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56354564b590 0x56354564cce0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0904012010 tx=0x7f0904003c30 comp rx=0 tx=0).stop 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 -- 172.21.15.16:0/4245949815 shutdown_connections 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56354565b3c0 0x56354565d7b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56354564b590 0x56354564cce0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x56354564c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 --2- 172.21.15.16:0/4245949815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x56354564c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f08f5600700 0 client.25483 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 -- 172.21.15.16:0/4245949815 >> 172.21.15.16:0/4245949815 conn(0x5635455b5220 msgr2=0x563545636460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 -- 172.21.15.16:0/4245949815 shutdown_connections 2024-04-05T23:33:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 1 -- 172.21.15.16:0/4245949815 wait complete. 2024-04-05T23:33:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.799+0000 7f09267e0b80 20 client.25483 trim_cache size 0 max 16384 2024-04-05T23:33:45.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.825+0000 7f09267e0b80 1 Processor -- start 2024-04-05T23:33:45.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.825+0000 7f09267e0b80 1 -- start start 2024-04-05T23:33:45.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635455b5220 0x563545650290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 0x563545636560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x56354563c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f09267e0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635454156a0 con 0x5635456460b0 2024-04-05T23:33:45.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f09267e0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635456572b0 con 0x5635455b5220 2024-04-05T23:33:45.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f09267e0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563545658040 con 0x5635456456f0 2024-04-05T23:33:45.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f090be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635455b5220 0x563545650290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f090be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635455b5220 0x563545650290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59362/0 (socket says 172.21.15.16:59362) 2024-04-05T23:33:45.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f090be00700 1 -- 172.21.15.16:0/3645953658 learned_addr learned my addr 172.21.15.16:0/3645953658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.826+0000 7f0910c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x56354563c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f090b400700 1 --2- 172.21.15.16:0/3645953658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 0x563545636560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f090be00700 1 -- 172.21.15.16:0/3645953658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 msgr2=0x563545636560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f090be00700 1 --2- 172.21.15.16:0/3645953658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 0x563545636560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f090be00700 1 -- 172.21.15.16:0/3645953658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 msgr2=0x56354563c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f090be00700 1 --2- 172.21.15.16:0/3645953658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x56354563c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f090be00700 1 -- 172.21.15.16:0/3645953658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563545557ea0 con 0x5635455b5220 2024-04-05T23:33:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f090be00700 1 --2- 172.21.15.16:0/3645953658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635455b5220 0x563545650290 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00b2e0 tx=0x7f08fc022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e87d537355f1774 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f0901c00700 1 -- 172.21.15.16:0/3645953658 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc01d050 con 0x5635455b5220 2024-04-05T23:33:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f0901c00700 1 -- 172.21.15.16:0/3645953658 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08fc01e040 con 0x5635455b5220 2024-04-05T23:33:45.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.827+0000 7f0901c00700 1 -- 172.21.15.16:0/3645953658 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc004b20 con 0x5635455b5220 2024-04-05T23:33:45.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.828+0000 7f09267e0b80 1 -- 172.21.15.16:0/3645953658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635455b5220 msgr2=0x563545650290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.828+0000 7f09267e0b80 1 --2- 172.21.15.16:0/3645953658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635455b5220 0x563545650290 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00b2e0 tx=0x7f08fc022000 comp rx=0 tx=0).stop 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.828+0000 7f09267e0b80 1 -- 172.21.15.16:0/3645953658 shutdown_connections 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.828+0000 7f09267e0b80 1 --2- 172.21.15.16:0/3645953658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x56354563c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.828+0000 7f09267e0b80 1 --2- 172.21.15.16:0/3645953658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635455b5220 0x563545650290 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.828+0000 7f09267e0b80 1 --2- 172.21.15.16:0/3645953658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635456456f0 0x563545636560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.828+0000 7f09267e0b80 1 -- 172.21.15.16:0/3645953658 >> 172.21.15.16:0/3645953658 conn(0x563545643690 msgr2=0x563545643d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.828+0000 7f09267e0b80 1 -- 172.21.15.16:0/3645953658 shutdown_connections 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.828+0000 7f09267e0b80 1 -- 172.21.15.16:0/3645953658 wait complete. 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.829+0000 7f09267e0b80 1 -- start start 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.829+0000 7f09267e0b80 1 Processor -- start 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.829+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635455b5220 0x5635456a8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x5635456a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f09267e0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x5635456a3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f09267e0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635456a37f0 con 0x5635456460b0 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f09267e0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635456a3960 con 0x5635456456f0 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f09267e0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635456a3b20 con 0x5635455b5220 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f09267e0b80 10 client.0 _ll_register_callbacks cb 0x7f0904011550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635455b5220 0x5635456a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635455b5220 0x5635456a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:48994/0 (socket says 172.21.15.16:48994) 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090be00700 1 -- 172.21.15.16:0/2468728821 learned_addr learned my addr 172.21.15.16:0/2468728821 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f0910c00700 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x5635456a3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090b400700 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x5635456a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090be00700 1 -- 172.21.15.16:0/2468728821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 msgr2=0x5635456a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090be00700 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x5635456a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090be00700 1 -- 172.21.15.16:0/2468728821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 msgr2=0x5635456a3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090be00700 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x5635456a3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090be00700 1 -- 172.21.15.16:0/2468728821 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f090c10e590 con 0x5635455b5220 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.830+0000 7f090b400700 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x5635456a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.831+0000 7f090be00700 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635455b5220 0x5635456a8710 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00b6f0 tx=0x7f08fc036140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a2b54313c9d0de6 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.831+0000 7f08f5600700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.831+0000 7f08f5600700 1 -- 172.21.15.16:0/2468728821 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc01d030 con 0x5635455b5220 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.831+0000 7f09267e0b80 1 -- 172.21.15.16:0/2468728821 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f090400f040 con 0x5635455b5220 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.831+0000 7f08f5600700 1 -- 172.21.15.16:0/2468728821 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f08fc036550 con 0x5635455b5220 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.831+0000 7f08f5600700 1 -- 172.21.15.16:0/2468728821 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08fc004710 con 0x5635455b5220 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.831+0000 7f09267e0b80 10 client.25486 fetch_fsmap learned FSMap version 28 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.831+0000 7f09267e0b80 1 -- 172.21.15.16:0/2468728821 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563545417cd0 con 0x5635455b5220 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.832+0000 7f08f5600700 1 -- 172.21.15.16:0/2468728821 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08fc04e020 con 0x5635455b5220 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.832+0000 7f08f5600700 1 -- 172.21.15.16:0/2468728821 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f08fc03fa60 con 0x5635455b5220 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.832+0000 7f09267e0b80 10 client.25486 fetch_fsmap finished waiting for FSMap version 28 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.832+0000 7f09267e0b80 10 client.25486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:33:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.832+0000 7f09267e0b80 20 client.25486 populate_metadata read hostname 'smithi016' 2024-04-05T23:33:45.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.832+0000 7f09267e0b80 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56354564b590 0x5635456b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:45.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.832+0000 7f09267e0b80 4 client.25486 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-05T23:33:45.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.832+0000 7f09267e0b80 1 -- 172.21.15.16:0/2468728821 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5635453e0cf0 con 0x56354564b590 2024-04-05T23:33:45.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.832+0000 7f090b400700 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56354564b590 0x5635456b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f090b400700 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56354564b590 0x5635456b20e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=484cf8d3c9957865 server_cookie=34cc43f649d38236 in_seq=0 out_seq=0 2024-04-05T23:33:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f08f5600700 10 client.25486 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f08f5600700 1 -- 172.21.15.16:0/2468728821 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5635453e0cf0 con 0x56354564b590 2024-04-05T23:33:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f08f5600700 10 client.25486 handle_command_reply: tid=0 2024-04-05T23:33:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f09267e0b80 1 client.25486 shutdown 2024-04-05T23:33:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f09267e0b80 1 -- 172.21.15.16:0/2468728821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635455b5220 msgr2=0x5635456a8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f09267e0b80 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635455b5220 0x5635456a8710 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00b6f0 tx=0x7f08fc036140 comp rx=0 tx=0).stop 2024-04-05T23:33:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f09267e0b80 1 -- 172.21.15.16:0/2468728821 shutdown_connections 2024-04-05T23:33:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f09267e0b80 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56354564b590 0x5635456b20e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f09267e0b80 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635456460b0 0x5635456a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f09267e0b80 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635455b5220 0x5635456a8710 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f08f5600700 0 client.25486 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:33:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f09267e0b80 1 --2- 172.21.15.16:0/2468728821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635456456f0 0x5635456a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.833+0000 7f09267e0b80 1 -- 172.21.15.16:0/2468728821 >> 172.21.15.16:0/2468728821 conn(0x563545643690 msgr2=0x56354563c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.834+0000 7f09267e0b80 1 -- 172.21.15.16:0/2468728821 shutdown_connections 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.834+0000 7f09267e0b80 1 -- 172.21.15.16:0/2468728821 wait complete. 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.834+0000 7f09267e0b80 20 client.25486 trim_cache size 0 max 16384 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "e409aeb5-ba51-4f56-9a8d-bbd2205182e1": { 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "e409aeb5-ba51-4f56-9a8d-bbd2205182e1", 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-04-05T23:33:45.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:33:45.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:33:45.832 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 -- 172.21.15.16:0/4020753514 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f08f00a13a0 msgr2=0x7f08f00a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f08f00a13a0 0x7f08f00a3860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x563545636ca0 tx=0x7f08fc005bc0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 -- 172.21.15.16:0/4020753514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c1061d0 msgr2=0x7f090c197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c1061d0 0x7f090c197860 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f08f800e9c0 tx=0x7f08f800ecd0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 -- 172.21.15.16:0/4020753514 shutdown_connections 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f08f00a13a0 0x7f08f00a3860 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f090c068220 0x7f090c197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f090c19bc00 0x7f090c19c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 --2- 172.21.15.16:0/4020753514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c1061d0 0x7f090c197860 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 -- 172.21.15.16:0/4020753514 >> 172.21.15.16:0/4020753514 conn(0x7f090c076e80 msgr2=0x7f090c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 -- 172.21.15.16:0/4020753514 shutdown_connections 2024-04-05T23:33:45.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.835+0000 7f0912c00700 1 -- 172.21.15.16:0/4020753514 wait complete. 2024-04-05T23:33:45.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:45 smithi016 ceph-mon[38248]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 228 B/s wr, 19 op/s 2024-04-05T23:33:45.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:45 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3239793829' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:45.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:45 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2365950119' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:45.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:45 smithi016 ceph-mon[38248]: scrub queued for path: / 2024-04-05T23:33:45.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:45 smithi016 ceph-mon[38248]: scrub summary: idle+waiting paths [/] 2024-04-05T23:33:45.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:45 smithi016 ceph-mon[38248]: scrub summary: active paths [/] 2024-04-05T23:33:45.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:45 smithi016 ceph-mon[38248]: scrub summary: idle+waiting paths [/] 2024-04-05T23:33:45.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.849+0000 7f6e80c00700 1 -- 172.21.15.16:0/2763931769 <== mgr.15240 v2:172.21.15.16:6800/3895420073 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6e84110d30 con 0x7f6e680a13a0 2024-04-05T23:33:45.842 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path / 2024-04-05T23:33:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6e680a13a0 msgr2=0x7f6e680a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6e680a13a0 0x7f6e680a3860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6e74006010 tx=0x7f6e7400b280 comp rx=0 tx=0).stop 2024-04-05T23:33:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e84106f40 msgr2=0x7f6e841007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e84106f40 0x7f6e841007d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00efc0 tx=0x7f6e7c00c3a0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 shutdown_connections 2024-04-05T23:33:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6e680a13a0 0x7f6e680a3860 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e84106f40 0x7f6e841007d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e84068220 0x7f6e840ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 --2- 172.21.15.16:0/2763931769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e84068be0 0x7f6e84100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 >> 172.21.15.16:0/2763931769 conn(0x7f6e84076e50 msgr2=0x7f6e840febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 shutdown_connections 2024-04-05T23:33:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:45.852+0000 7f6e8b000700 1 -- 172.21.15.16:0/2763931769 wait complete. 2024-04-05T23:33:45.848 INFO:tasks.cephfs.filesystem:scrub status for tag:e409aeb5-ba51-4f56-9a8d-bbd2205182e1 - {'path': '/', 'tag': 'e409aeb5-ba51-4f56-9a8d-bbd2205182e1', 'options': 'recursive,force'} 2024-04-05T23:33:45.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-05T23:33:45.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:45 smithi027 ceph-mon[78597]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 228 B/s wr, 19 op/s 2024-04-05T23:33:45.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:45 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3239793829' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:45.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:45 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2365950119' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:45.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:45 smithi027 ceph-mon[78597]: scrub queued for path: / 2024-04-05T23:33:45.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:45 smithi027 ceph-mon[78597]: scrub summary: idle+waiting paths [/] 2024-04-05T23:33:45.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:45 smithi027 ceph-mon[78597]: scrub summary: active paths [/] 2024-04-05T23:33:45.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:45 smithi027 ceph-mon[78597]: scrub summary: idle+waiting paths [/] 2024-04-05T23:33:46.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:45 smithi130 ceph-mon[79417]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 228 B/s wr, 19 op/s 2024-04-05T23:33:46.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:45 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3239793829' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:33:46.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:45 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2365950119' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:33:46.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:45 smithi130 ceph-mon[79417]: scrub queued for path: / 2024-04-05T23:33:46.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:45 smithi130 ceph-mon[79417]: scrub summary: idle+waiting paths [/] 2024-04-05T23:33:46.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:45 smithi130 ceph-mon[79417]: scrub summary: active paths [/] 2024-04-05T23:33:46.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:45 smithi130 ceph-mon[79417]: scrub summary: idle+waiting paths [/] 2024-04-05T23:33:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.108+0000 7f4555600700 1 -- 172.21.15.16:0/965919501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4550068be0 msgr2=0x7f455010dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.108+0000 7f4555600700 1 --2- 172.21.15.16:0/965919501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4550068be0 0x7f455010dbe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f454800b0a0 tx=0x7f454800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.108+0000 7f4555600700 1 -- 172.21.15.16:0/965919501 shutdown_connections 2024-04-05T23:33:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.108+0000 7f4555600700 1 --2- 172.21.15.16:0/965919501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550106f40 0x7f4550107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.108+0000 7f4555600700 1 --2- 172.21.15.16:0/965919501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4550068be0 0x7f455010dbe0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.108+0000 7f4555600700 1 --2- 172.21.15.16:0/965919501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4550068220 0x7f45500686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.108+0000 7f4555600700 1 -- 172.21.15.16:0/965919501 >> 172.21.15.16:0/965919501 conn(0x7f4550076e50 msgr2=0x7f4550077260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.109+0000 7f4555600700 1 -- 172.21.15.16:0/965919501 shutdown_connections 2024-04-05T23:33:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.109+0000 7f4555600700 1 -- 172.21.15.16:0/965919501 wait complete. 2024-04-05T23:33:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.109+0000 7f4555600700 1 Processor -- start 2024-04-05T23:33:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f4555600700 1 -- start start 2024-04-05T23:33:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f4555600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4550068220 0x7f45500ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f4555600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4550068be0 0x7f4550100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f4555600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550106f40 0x7f45501007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f4555600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501a5760 con 0x7f4550106f40 2024-04-05T23:33:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f4555600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501a58d0 con 0x7f4550068be0 2024-04-05T23:33:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f4555600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501a5a90 con 0x7f4550068220 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f454f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550106f40 0x7f45501007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f454ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4550068220 0x7f45500ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f4547e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4550068be0 0x7f4550100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f454ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4550068220 0x7f45500ffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:49006/0 (socket says 172.21.15.16:49006) 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.110+0000 7f454ea00700 1 -- 172.21.15.16:0/1027213295 learned_addr learned my addr 172.21.15.16:0/1027213295 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f454f400700 1 -- 172.21.15.16:0/1027213295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4550068220 msgr2=0x7f45500ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f454f400700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4550068220 0x7f45500ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f454f400700 1 -- 172.21.15.16:0/1027213295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4550068be0 msgr2=0x7f4550100290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f454f400700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4550068be0 0x7f4550100290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f454f400700 1 -- 172.21.15.16:0/1027213295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4548009d00 con 0x7f4550106f40 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f454f400700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550106f40 0x7f45501007d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f454800b070 tx=0x7f4548007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f4547400700 1 -- 172.21.15.16:0/1027213295 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454800e040 con 0x7f4550106f40 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f4547400700 1 -- 172.21.15.16:0/1027213295 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f454801d470 con 0x7f4550106f40 2024-04-05T23:33:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45501a5d10 con 0x7f4550106f40 2024-04-05T23:33:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f4547400700 1 -- 172.21.15.16:0/1027213295 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4548005280 con 0x7f4550106f40 2024-04-05T23:33:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.111+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45501a6190 con 0x7f4550106f40 2024-04-05T23:33:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.113+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45501092c0 con 0x7f4550106f40 2024-04-05T23:33:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.114+0000 7f4547400700 1 -- 172.21.15.16:0/1027213295 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f454801d5e0 con 0x7f4550106f40 2024-04-05T23:33:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.116+0000 7f4547400700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f45340a13a0 0x7f45340a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.116+0000 7f4547400700 1 -- 172.21.15.16:0/1027213295 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f45480ce110 con 0x7f4550106f40 2024-04-05T23:33:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.116+0000 7f454ea00700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f45340a13a0 0x7f45340a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.117+0000 7f454ea00700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f45340a13a0 0x7f45340a3860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4540009ca0 tx=0x7f4540009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.119+0000 7f4547400700 1 -- 172.21.15.16:0/1027213295 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4548097490 con 0x7f4550106f40 2024-04-05T23:33:46.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.374+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 --> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f4550110d50 con 0x7f45340a13a0 2024-04-05T23:33:46.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.445+0000 7f4547400700 1 -- 172.21.15.16:0/1027213295 <== mgr.15240 v2:172.21.15.16:6800/3895420073 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f4550110d50 con 0x7f45340a13a0 2024-04-05T23:33:46.438 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path / 2024-04-05T23:33:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f45340a13a0 msgr2=0x7f45340a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f45340a13a0 0x7f45340a3860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4540009ca0 tx=0x7f4540009260 comp rx=0 tx=0).stop 2024-04-05T23:33:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550106f40 msgr2=0x7f45501007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550106f40 0x7f45501007d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f454800b070 tx=0x7f4548007860 comp rx=0 tx=0).stop 2024-04-05T23:33:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 shutdown_connections 2024-04-05T23:33:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f45340a13a0 0x7f45340a3860 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550106f40 0x7f45501007d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4550068220 0x7f45500ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 --2- 172.21.15.16:0/1027213295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4550068be0 0x7f4550100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 >> 172.21.15.16:0/1027213295 conn(0x7f4550076e50 msgr2=0x7f45500fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:46.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 shutdown_connections 2024-04-05T23:33:46.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.448+0000 7f4555600700 1 -- 172.21.15.16:0/1027213295 wait complete. 2024-04-05T23:33:46.460 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-05T23:33:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.664+0000 7fada0600700 1 -- 172.21.15.16:0/1052939074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 msgr2=0x7fad98101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.664+0000 7fada0600700 1 --2- 172.21.15.16:0/1052939074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 0x7fad98101410 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009990 tx=0x7fad8c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.664+0000 7fada0600700 1 -- 172.21.15.16:0/1052939074 shutdown_connections 2024-04-05T23:33:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.664+0000 7fada0600700 1 --2- 172.21.15.16:0/1052939074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 0x7fad98101410 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.664+0000 7fada0600700 1 --2- 172.21.15.16:0/1052939074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad98106150 0x7fad9810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.664+0000 7fada0600700 1 --2- 172.21.15.16:0/1052939074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad981019e0 0x7fad98105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.664+0000 7fada0600700 1 -- 172.21.15.16:0/1052939074 >> 172.21.15.16:0/1052939074 conn(0x7fad980fc8b0 msgr2=0x7fad980fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.664+0000 7fada0600700 1 -- 172.21.15.16:0/1052939074 shutdown_connections 2024-04-05T23:33:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.664+0000 7fada0600700 1 -- 172.21.15.16:0/1052939074 wait complete. 2024-04-05T23:33:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.666+0000 7fada0600700 1 Processor -- start 2024-04-05T23:33:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.666+0000 7fada0600700 1 -- start start 2024-04-05T23:33:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.666+0000 7fada0600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 0x7fad981972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.666+0000 7fada0600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad98106150 0x7fad98197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.666+0000 7fada0600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad9819bba0 0x7fad9819c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.666+0000 7fada0600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9819c620 con 0x7fad98101030 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.666+0000 7fada0600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9819c790 con 0x7fad98106150 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.666+0000 7fada0600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9819c950 con 0x7fad9819bba0 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9dc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 0x7fad981972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9dc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 0x7fad981972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41060/0 (socket says 172.21.15.16:41060) 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9dc00700 1 -- 172.21.15.16:0/4283572691 learned_addr learned my addr 172.21.15.16:0/4283572691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9d200700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad98106150 0x7fad98197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9e600700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad9819bba0 0x7fad9819c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9dc00700 1 -- 172.21.15.16:0/4283572691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad9819bba0 msgr2=0x7fad9819c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9dc00700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad9819bba0 0x7fad9819c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9dc00700 1 -- 172.21.15.16:0/4283572691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad98106150 msgr2=0x7fad98197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9dc00700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad98106150 0x7fad98197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9dc00700 1 -- 172.21.15.16:0/4283572691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c0095f0 con 0x7fad98101030 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.667+0000 7fad9dc00700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 0x7fad981972c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fad8c005910 tx=0x7fad8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.668+0000 7fad96000700 1 -- 172.21.15.16:0/4283572691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c01d070 con 0x7fad98101030 2024-04-05T23:33:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.668+0000 7fad96000700 1 -- 172.21.15.16:0/4283572691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad8c022470 con 0x7fad98101030 2024-04-05T23:33:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.668+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad981a3e50 con 0x7fad98101030 2024-04-05T23:33:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.668+0000 7fad96000700 1 -- 172.21.15.16:0/4283572691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c017650 con 0x7fad98101030 2024-04-05T23:33:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.668+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad981a4210 con 0x7fad98101030 2024-04-05T23:33:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.669+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad98109370 con 0x7fad98101030 2024-04-05T23:33:46.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.671+0000 7fad96000700 1 -- 172.21.15.16:0/4283572691 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7fad8c0225e0 con 0x7fad98101030 2024-04-05T23:33:46.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.672+0000 7fad96000700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fad800a1320 0x7fad800a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:46.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.672+0000 7fad96000700 1 -- 172.21.15.16:0/4283572691 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fad8c0ce110 con 0x7fad98101030 2024-04-05T23:33:46.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.672+0000 7fad9d200700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fad800a1320 0x7fad800a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:46.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.673+0000 7fad9d200700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fad800a1320 0x7fad800a37e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fad98198810 tx=0x7fad84009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:46.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.676+0000 7fad96000700 1 -- 172.21.15.16:0/4283572691 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fad8c097570 con 0x7fad98101030 2024-04-05T23:33:46.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:46 smithi016 ceph-mon[38248]: from='client.25433 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:46.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:46 smithi016 ceph-mon[38248]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.922+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 --> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fad981985c0 con 0x7fad800a1320 2024-04-05T23:33:46.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.930+0000 7fad96000700 1 -- 172.21.15.16:0/4283572691 <== mgr.15240 v2:172.21.15.16:6800/3895420073 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7fad981985c0 con 0x7fad800a1320 2024-04-05T23:33:46.923 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-05T00:00:00", "created": "2024-04-05T23:33:45", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-05T23:33:46.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fad800a1320 msgr2=0x7fad800a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fad800a1320 0x7fad800a37e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fad98198810 tx=0x7fad84009260 comp rx=0 tx=0).stop 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 msgr2=0x7fad981972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 0x7fad981972c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fad8c005910 tx=0x7fad8c004730 comp rx=0 tx=0).stop 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 shutdown_connections 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fad800a1320 0x7fad800a37e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad98101030 0x7fad981972c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad9819bba0 0x7fad9819c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 --2- 172.21.15.16:0/4283572691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad98106150 0x7fad98197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 >> 172.21.15.16:0/4283572691 conn(0x7fad980fc8b0 msgr2=0x7fad980fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 shutdown_connections 2024-04-05T23:33:46.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:46.933+0000 7fada0600700 1 -- 172.21.15.16:0/4283572691 wait complete. 2024-04-05T23:33:46.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-05T23:33:46.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:46 smithi027 ceph-mon[78597]: from='client.25433 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:46.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:46 smithi027 ceph-mon[78597]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:47.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:46 smithi130 ceph-mon[79417]: from='client.25433 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:47.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:46 smithi130 ceph-mon[79417]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.149+0000 7f8785200700 1 -- 172.21.15.16:0/919216361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87801019e0 msgr2=0x7f8780105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.149+0000 7f8785200700 1 --2- 172.21.15.16:0/919216361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87801019e0 0x7f8780105a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8768009990 tx=0x7f8768009ca0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.149+0000 7f8785200700 1 -- 172.21.15.16:0/919216361 shutdown_connections 2024-04-05T23:33:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.149+0000 7f8785200700 1 --2- 172.21.15.16:0/919216361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8780106150 0x7f878010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.149+0000 7f8785200700 1 --2- 172.21.15.16:0/919216361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8780101030 0x7f8780101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.149+0000 7f8785200700 1 --2- 172.21.15.16:0/919216361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87801019e0 0x7f8780105a20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.149+0000 7f8785200700 1 -- 172.21.15.16:0/919216361 >> 172.21.15.16:0/919216361 conn(0x7f87800fc8b0 msgr2=0x7f87800fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:47.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.149+0000 7f8785200700 1 -- 172.21.15.16:0/919216361 shutdown_connections 2024-04-05T23:33:47.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.149+0000 7f8785200700 1 -- 172.21.15.16:0/919216361 wait complete. 2024-04-05T23:33:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.150+0000 7f8785200700 1 Processor -- start 2024-04-05T23:33:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.150+0000 7f8785200700 1 -- start start 2024-04-05T23:33:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.150+0000 7f8785200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8780101030 0x7f8780199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.150+0000 7f8785200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8780106150 0x7f8780199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.150+0000 7f8785200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f878019de00 0x7f878019e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.150+0000 7f8785200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878019e880 con 0x7f8780106150 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.150+0000 7f8785200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878019e9f0 con 0x7f878019de00 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.150+0000 7f8785200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878019ebb0 con 0x7f8780101030 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.150+0000 7f8777e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8780106150 0x7f8780199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8777e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8780106150 0x7f8780199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41078/0 (socket says 172.21.15.16:41078) 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8777e00700 1 -- 172.21.15.16:0/4004510803 learned_addr learned my addr 172.21.15.16:0/4004510803 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f877de00700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8780101030 0x7f8780199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f877e800700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f878019de00 0x7f878019e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8777e00700 1 -- 172.21.15.16:0/4004510803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8780101030 msgr2=0x7f8780199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8777e00700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8780101030 0x7f8780199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8777e00700 1 -- 172.21.15.16:0/4004510803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f878019de00 msgr2=0x7f878019e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8777e00700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f878019de00 0x7f878019e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8777e00700 1 -- 172.21.15.16:0/4004510803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680095f0 con 0x7f8780106150 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f877e800700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f878019de00 0x7f878019e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8777e00700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8780106150 0x7f8780199a60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8768000c00 tx=0x7f8768009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8776a00700 1 -- 172.21.15.16:0/4004510803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876801d070 con 0x7f8780106150 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.151+0000 7f8776a00700 1 -- 172.21.15.16:0/4004510803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f876800fb30 con 0x7f8780106150 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.152+0000 7f8776a00700 1 -- 172.21.15.16:0/4004510803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8768022bb0 con 0x7f8780106150 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.152+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87801a60b0 con 0x7f8780106150 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.152+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87801a64d0 con 0x7f8780106150 2024-04-05T23:33:47.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.153+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8780102500 con 0x7f8780106150 2024-04-05T23:33:47.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.155+0000 7f8776a00700 1 -- 172.21.15.16:0/4004510803 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 136049+0+0 (secure 0 0 0) 0x7f876800fca0 con 0x7f8780106150 2024-04-05T23:33:47.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.156+0000 7f8776a00700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87640a13a0 0x7f87640a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:33:47.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.156+0000 7f8776a00700 1 -- 172.21.15.16:0/4004510803 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f87680ce200 con 0x7f8780106150 2024-04-05T23:33:47.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.156+0000 7f877de00700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87640a13a0 0x7f87640a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:33:47.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.157+0000 7f877de00700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87640a13a0 0x7f87640a3860 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8770005950 tx=0x7f877000b410 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:33:47.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.160+0000 7f8776a00700 1 -- 172.21.15.16:0/4004510803 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8768097580 con 0x7f8780106150 2024-04-05T23:33:47.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.403+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 --> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f87801019e0 con 0x7f87640a13a0 2024-04-05T23:33:47.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.410+0000 7f8776a00700 1 -- 172.21.15.16:0/4004510803 <== mgr.15240 v2:172.21.15.16:6800/3895420073 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f87801019e0 con 0x7f87640a13a0 2024-04-05T23:33:47.403 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-04-05T23:33:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.412+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87640a13a0 msgr2=0x7f87640a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:47.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.412+0000 7f8785200700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87640a13a0 0x7f87640a3860 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8770005950 tx=0x7f877000b410 comp rx=0 tx=0).stop 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.412+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8780106150 msgr2=0x7f8780199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.412+0000 7f8785200700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8780106150 0x7f8780199a60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8768000c00 tx=0x7f8768009580 comp rx=0 tx=0).stop 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.413+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 shutdown_connections 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.413+0000 7f8785200700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87640a13a0 0x7f87640a3860 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.413+0000 7f8785200700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f878019de00 0x7f878019e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.413+0000 7f8785200700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8780101030 0x7f8780199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.413+0000 7f8785200700 1 --2- 172.21.15.16:0/4004510803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8780106150 0x7f8780199a60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.413+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 >> 172.21.15.16:0/4004510803 conn(0x7f87800fc8b0 msgr2=0x7f87800fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.413+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 shutdown_connections 2024-04-05T23:33:47.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:33:47.413+0000 7f8785200700 1 -- 172.21.15.16:0/4004510803 wait complete. 2024-04-05T23:33:47.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-05T23:33:47.504 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-05T23:33:47.516 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-05T23:33:47.543 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:33:47.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-05T23:33:47.547 INFO:teuthology.run_tasks:Running task workunit... 2024-04-05T23:33:47.579 INFO:tasks.workunit:Pulling workunits from ref 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:33:47.579 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-05T23:33:47.579 INFO:tasks.workunit:timeout=3h 2024-04-05T23:33:47.579 INFO:tasks.workunit:cleanup=True 2024-04-05T23:33:47.580 DEBUG:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-05T23:33:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-05T23:33:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-05T23:33:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 7ah/122d Inode: 1099511628283 Links: 2 2024-04-05T23:33:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T23:33:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:cephfs_t:s0 2024-04-05T23:33:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-05 23:32:12.452677430 +0000 2024-04-05T23:33:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-05 23:32:12.452677430 +0000 2024-04-05T23:33:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-05 23:33:34.631941502 +0000 2024-04-05T23:33:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-05 23:32:12.452677430 +0000 2024-04-05T23:33:47.624 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-05T23:33:47.624 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-05T23:33:47.721 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:47 smithi016 ceph-mon[38248]: pgmap v9: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 373 B/s wr, 20 op/s 2024-04-05T23:33:47.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:47 smithi016 ceph-mon[38248]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:47.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:47 smithi016 ceph-mon[38248]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:47.742 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T23:33:47.786 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-05T23:33:47.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:47 smithi027 ceph-mon[78597]: pgmap v9: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 373 B/s wr, 20 op/s 2024-04-05T23:33:47.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:47 smithi027 ceph-mon[78597]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:47.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:47 smithi027 ceph-mon[78597]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:48.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:47 smithi130 ceph-mon[79417]: pgmap v9: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 373 B/s wr, 20 op/s 2024-04-05T23:33:48.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:47 smithi130 ceph-mon[79417]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:48.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:47 smithi130 ceph-mon[79417]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:33:49.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:49 smithi027 ceph-mon[78597]: pgmap v10: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 341 B/s wr, 19 op/s 2024-04-05T23:33:50.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:49 smithi016 ceph-mon[38248]: pgmap v10: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 341 B/s wr, 19 op/s 2024-04-05T23:33:50.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:49 smithi130 ceph-mon[79417]: pgmap v10: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 341 B/s wr, 19 op/s 2024-04-05T23:33:51.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:51 smithi027 ceph-mon[78597]: pgmap v11: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 341 B/s wr, 19 op/s 2024-04-05T23:33:52.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:51 smithi016 ceph-mon[38248]: pgmap v11: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 341 B/s wr, 19 op/s 2024-04-05T23:33:52.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:51 smithi130 ceph-mon[79417]: pgmap v11: 97 pgs: 97 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 341 B/s wr, 19 op/s 2024-04-05T23:33:53.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:53 smithi027 ceph-mon[78597]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-04-05T23:33:54.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:53 smithi016 ceph-mon[38248]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-04-05T23:33:54.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:53 smithi130 ceph-mon[79417]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-04-05T23:33:55.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:55 smithi027 ceph-mon[78597]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-05T23:33:56.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:55 smithi016 ceph-mon[38248]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-05T23:33:56.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:55 smithi130 ceph-mon[79417]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-05T23:33:57.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:57 smithi027 ceph-mon[78597]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-04-05T23:33:57.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:57 smithi027 ceph-mon[78597]: scrub summary: idle 2024-04-05T23:33:58.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:57 smithi016 ceph-mon[38248]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-04-05T23:33:58.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:57 smithi016 ceph-mon[38248]: scrub summary: idle 2024-04-05T23:33:58.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:57 smithi130 ceph-mon[79417]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-04-05T23:33:58.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:57 smithi130 ceph-mon[79417]: scrub summary: idle 2024-04-05T23:33:59.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:33:59 smithi027 ceph-mon[78597]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-05T23:34:00.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:33:59 smithi016 ceph-mon[38248]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-05T23:34:00.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:33:59 smithi130 ceph-mon[79417]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-05T23:34:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.806+0000 7ff65dc00700 1 -- 172.21.15.16:0/1119597281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff658103c00 msgr2=0x7ff658107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.806+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1119597281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff658103c00 0x7ff658107c40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff644009990 tx=0x7ff644009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.806+0000 7ff65dc00700 1 -- 172.21.15.16:0/1119597281 shutdown_connections 2024-04-05T23:34:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.806+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1119597281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff658103250 0x7ff658103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.806+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1119597281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff658108370 0x7ff65810fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.806+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1119597281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff658103c00 0x7ff658107c40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.806+0000 7ff65dc00700 1 -- 172.21.15.16:0/1119597281 >> 172.21.15.16:0/1119597281 conn(0x7ff6580fea90 msgr2=0x7ff658100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.807+0000 7ff65dc00700 1 -- 172.21.15.16:0/1119597281 shutdown_connections 2024-04-05T23:34:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.807+0000 7ff65dc00700 1 -- 172.21.15.16:0/1119597281 wait complete. 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.808+0000 7ff65dc00700 1 Processor -- start 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.808+0000 7ff65dc00700 1 -- start start 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.808+0000 7ff65dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff658103250 0x7ff6581994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.808+0000 7ff65dc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff658108370 0x7ff658199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.808+0000 7ff65dc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff65819ddb0 0x7ff65819e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.808+0000 7ff65dc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65819e830 con 0x7ff658108370 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.808+0000 7ff65dc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65819e9a0 con 0x7ff658103250 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.808+0000 7ff65dc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65819eb60 con 0x7ff65819ddb0 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff658108370 0x7ff658199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff658108370 0x7ff658199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49848/0 (socket says 172.21.15.16:49848) 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656000700 1 -- 172.21.15.16:0/1514096704 learned_addr learned my addr 172.21.15.16:0/1514096704 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656a00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff658103250 0x7ff6581994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff657400700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff65819ddb0 0x7ff65819e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656a00700 1 -- 172.21.15.16:0/1514096704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff65819ddb0 msgr2=0x7ff65819e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656a00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff65819ddb0 0x7ff65819e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656a00700 1 -- 172.21.15.16:0/1514096704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff658108370 msgr2=0x7ff658199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656a00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff658108370 0x7ff658199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656a00700 1 -- 172.21.15.16:0/1514096704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6440095f0 con 0x7ff658103250 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.809+0000 7ff656a00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff658103250 0x7ff6581994d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff64800e9f0 tx=0x7ff64800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.810+0000 7ff64f400700 1 -- 172.21.15.16:0/1514096704 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64800c9c0 con 0x7ff658103250 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.810+0000 7ff65dc00700 1 -- 172.21.15.16:0/1514096704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6581a60c0 con 0x7ff658103250 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.810+0000 7ff64f400700 1 -- 172.21.15.16:0/1514096704 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff648010430 con 0x7ff658103250 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.810+0000 7ff65dc00700 1 -- 172.21.15.16:0/1514096704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6581a65f0 con 0x7ff658103250 2024-04-05T23:34:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.810+0000 7ff64f400700 1 -- 172.21.15.16:0/1514096704 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff648018620 con 0x7ff658103250 2024-04-05T23:34:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.813+0000 7ff64f400700 1 -- 172.21.15.16:0/1514096704 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7ff648010860 con 0x7ff658103250 2024-04-05T23:34:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.814+0000 7ff64f400700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff63c0a13b0 0x7ff63c0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.814+0000 7ff671636b80 1 Processor -- start 2024-04-05T23:34:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.814+0000 7ff64f400700 1 -- 172.21.15.16:0/1514096704 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff648014070 con 0x7ff658103250 2024-04-05T23:34:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.814+0000 7ff671636b80 1 -- start start 2024-04-05T23:34:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff656000700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff63c0a13b0 0x7ff63c0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff671636b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff671636b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2ed030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff671636b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2eda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff671636b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc253b80 con 0x55c5bc2f34a0 2024-04-05T23:34:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff671636b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc244930 con 0x55c5bc2f2b80 2024-04-05T23:34:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff671636b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc2422f0 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff656a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff656a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:49598/0 (socket says 172.21.15.16:49598) 2024-04-05T23:34:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff656a00700 1 -- 172.21.15.16:0/67673284 learned_addr learned my addr 172.21.15.16:0/67673284 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff656000700 1 --2- 172.21.15.16:0/67673284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2ed030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.815+0000 7ff657400700 1 --2- 172.21.15.16:0/67673284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2eda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.816+0000 7ff656a00700 1 -- 172.21.15.16:0/67673284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 msgr2=0x55c5bc2eda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.816+0000 7ff656a00700 1 --2- 172.21.15.16:0/67673284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2eda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.816+0000 7ff656a00700 1 -- 172.21.15.16:0/67673284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 msgr2=0x55c5bc2ed030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.816+0000 7ff656a00700 1 --2- 172.21.15.16:0/67673284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2ed030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.816+0000 7ff656a00700 1 -- 172.21.15.16:0/67673284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5bc0bf350 con 0x55c5bc2f2b80 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.816+0000 7ff656000700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff63c0a13b0 0x7ff63c0a3870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff65819aa20 tx=0x7ff6440053d0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.816+0000 7ff656a00700 1 --2- 172.21.15.16:0/67673284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f2f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff648099f00 tx=0x7ff648099bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f576fb88d72bac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.816+0000 7ff643e00700 1 -- 172.21.15.16:0/67673284 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff648099f70 con 0x55c5bc2f2b80 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.816+0000 7ff643e00700 1 -- 172.21.15.16:0/67673284 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff64809dda0 con 0x55c5bc2f2b80 2024-04-05T23:34:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff643e00700 1 -- 172.21.15.16:0/67673284 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6480a1f60 con 0x55c5bc2f2b80 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff671636b80 1 -- 172.21.15.16:0/67673284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 msgr2=0x55c5bc2f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff671636b80 1 --2- 172.21.15.16:0/67673284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f2f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff648099f00 tx=0x7ff648099bb0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff671636b80 1 -- 172.21.15.16:0/67673284 shutdown_connections 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff671636b80 1 --2- 172.21.15.16:0/67673284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2eda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff671636b80 1 --2- 172.21.15.16:0/67673284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f2f60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff671636b80 1 --2- 172.21.15.16:0/67673284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2ed030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff671636b80 1 -- 172.21.15.16:0/67673284 >> 172.21.15.16:0/67673284 conn(0x55c5bc2611c0 msgr2=0x55c5bc2e7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff671636b80 1 -- 172.21.15.16:0/67673284 shutdown_connections 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.817+0000 7ff671636b80 1 -- 172.21.15.16:0/67673284 wait complete. 2024-04-05T23:34:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.818+0000 7ff671636b80 1 -- start start 2024-04-05T23:34:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.818+0000 7ff671636b80 1 Processor -- start 2024-04-05T23:34:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff671636b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff671636b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff671636b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff671636b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc23dbe0 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff671636b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc1d9d20 con 0x55c5bc2f2b80 2024-04-05T23:34:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff671636b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc1d99c0 con 0x55c5bc2f34a0 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff671636b80 10 client.0 _ll_register_callbacks cb 0x55c5bc045340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff656a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff656000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff656a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49874/0 (socket says 172.21.15.16:49874) 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff656a00700 1 -- 172.21.15.16:0/1729176746 learned_addr learned my addr 172.21.15.16:0/1729176746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff657400700 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff656a00700 1 -- 172.21.15.16:0/1729176746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2f34a0 msgr2=0x55c5bc2f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.819+0000 7ff656a00700 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff656a00700 1 -- 172.21.15.16:0/1729176746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 msgr2=0x55c5bc2f6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff656a00700 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff656a00700 1 -- 172.21.15.16:0/1729176746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6581a65f0 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff656a00700 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f9430 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff648099c70 tx=0x7ff64809a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793640e9cce84638 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff640c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff640c00700 1 -- 172.21.15.16:0/1729176746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6480ca5f0 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff640c00700 1 -- 172.21.15.16:0/1729176746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff6480ade40 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff671636b80 1 -- 172.21.15.16:0/1729176746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c5bc2e01e0 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff640c00700 1 -- 172.21.15.16:0/1729176746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6480a23f0 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff671636b80 10 client.15483 fetch_fsmap learned FSMap version 28 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.820+0000 7ff671636b80 1 -- 172.21.15.16:0/1729176746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c5bc06bb00 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.821+0000 7ff640c00700 1 -- 172.21.15.16:0/1729176746 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c5bc2e01e0 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.821+0000 7ff640c00700 1 -- 172.21.15.16:0/1729176746 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff6480ca960 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.821+0000 7ff671636b80 10 client.15483 fetch_fsmap finished waiting for FSMap version 28 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.821+0000 7ff671636b80 10 client.15483 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.821+0000 7ff671636b80 20 client.15483 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.821+0000 7ff671636b80 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c5bc306be0 0x55c5bc309000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.821+0000 7ff671636b80 4 client.15483 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.821+0000 7ff671636b80 1 -- 172.21.15.16:0/1729176746 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5bc2366a0 con 0x55c5bc306be0 2024-04-05T23:34:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.822+0000 7ff656000700 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c5bc306be0 0x55c5bc309000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.822+0000 7ff656000700 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c5bc306be0 0x55c5bc309000 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce1c2446fab6356e server_cookie=e6efb4af6ac5523 in_seq=0 out_seq=0 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.822+0000 7ff640c00700 10 client.15483 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff640c00700 1 -- 172.21.15.16:0/1729176746 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c5bc2366a0 con 0x55c5bc306be0 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff640c00700 10 client.15483 handle_command_reply: tid=0 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 client.15483 shutdown 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 -- 172.21.15.16:0/1729176746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2ed5c0 msgr2=0x55c5bc2f9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f9430 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff648099c70 tx=0x7ff64809a730 comp rx=0 tx=0).stop 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 -- 172.21.15.16:0/1729176746 shutdown_connections 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c5bc306be0 0x55c5bc309000 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f9430 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 --2- 172.21.15.16:0/1729176746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2f6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 -- 172.21.15.16:0/1729176746 >> 172.21.15.16:0/1729176746 conn(0x55c5bc2611c0 msgr2=0x55c5bc2e7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff640c00700 0 client.15483 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:00.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 -- 172.21.15.16:0/1729176746 shutdown_connections 2024-04-05T23:34:00.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.823+0000 7ff671636b80 1 -- 172.21.15.16:0/1729176746 wait complete. 2024-04-05T23:34:00.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.824+0000 7ff671636b80 20 client.15483 trim_cache size 0 max 16384 2024-04-05T23:34:00.832 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:34:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:34:00.457+0000 7fa144c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:34:00.832 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:34:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:34:00.457+0000 7fa144c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:34:00.832 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:34:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:34:00.457+0000 7fa144c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:34:00.832 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:34:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:34:00.457+0000 7fa144c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:34:00.832 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:34:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:34:00.457+0000 7fa144c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff671636b80 1 Processor -- start 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff671636b80 1 -- start start 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff671636b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff671636b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff671636b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff671636b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc0c1640 con 0x55c5bc2f34a0 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff671636b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc2f7d20 con 0x55c5bc2f2b80 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff671636b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc2f9a80 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff656a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff656000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff656a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59672/0 (socket says 172.21.15.16:59672) 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff656000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2e7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:49608/0 (socket says 172.21.15.16:49608) 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.840+0000 7ff656a00700 1 -- 172.21.15.16:0/4255948793 learned_addr learned my addr 172.21.15.16:0/4255948793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff657400700 1 --2- 172.21.15.16:0/4255948793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff656000700 1 -- 172.21.15.16:0/4255948793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 msgr2=0x55c5bc2f6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff656000700 1 --2- 172.21.15.16:0/4255948793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff656000700 1 -- 172.21.15.16:0/4255948793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 msgr2=0x55c5bc2f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff656000700 1 --2- 172.21.15.16:0/4255948793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff656000700 1 -- 172.21.15.16:0/4255948793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5bc203e40 con 0x55c5bc2f2b80 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff656000700 1 --2- 172.21.15.16:0/4255948793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2e7a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55c5bc2f3900 tx=0x7ff644027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53ce44e9c6e50634 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff64cc00700 1 -- 172.21.15.16:0/4255948793 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64401d070 con 0x55c5bc2f2b80 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff64cc00700 1 -- 172.21.15.16:0/4255948793 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff64400be50 con 0x55c5bc2f2b80 2024-04-05T23:34:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.841+0000 7ff64cc00700 1 -- 172.21.15.16:0/4255948793 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff644004870 con 0x55c5bc2f2b80 2024-04-05T23:34:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.842+0000 7ff671636b80 1 -- 172.21.15.16:0/4255948793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 msgr2=0x55c5bc2e7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.842+0000 7ff671636b80 1 --2- 172.21.15.16:0/4255948793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2e7a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55c5bc2f3900 tx=0x7ff644027000 comp rx=0 tx=0).stop 2024-04-05T23:34:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.842+0000 7ff671636b80 1 -- 172.21.15.16:0/4255948793 shutdown_connections 2024-04-05T23:34:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.842+0000 7ff671636b80 1 --2- 172.21.15.16:0/4255948793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.842+0000 7ff671636b80 1 --2- 172.21.15.16:0/4255948793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc2f6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.842+0000 7ff671636b80 1 --2- 172.21.15.16:0/4255948793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc2e7a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.842+0000 7ff671636b80 1 -- 172.21.15.16:0/4255948793 >> 172.21.15.16:0/4255948793 conn(0x55c5bc2611c0 msgr2=0x55c5bc2f6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.842+0000 7ff671636b80 1 -- 172.21.15.16:0/4255948793 shutdown_connections 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.842+0000 7ff671636b80 1 -- 172.21.15.16:0/4255948793 wait complete. 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.843+0000 7ff671636b80 1 -- start start 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.843+0000 7ff671636b80 1 Processor -- start 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.843+0000 7ff671636b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc352c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.843+0000 7ff671636b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc353180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.843+0000 7ff671636b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff671636b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc2fa540 con 0x55c5bc2f34a0 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff671636b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc2fa6b0 con 0x55c5bc2f2b80 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff671636b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5bc2fa870 con 0x55c5bc2ed5c0 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff671636b80 10 client.0 _ll_register_callbacks cb 0x7ff650009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff657400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff656000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc353180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff656000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc353180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:49612/0 (socket says 172.21.15.16:49612) 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff656000700 1 -- 172.21.15.16:0/4108273465 learned_addr learned my addr 172.21.15.16:0/4108273465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff656a00700 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc352c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff656000700 1 -- 172.21.15.16:0/4108273465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 msgr2=0x55c5bc352c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff656000700 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc352c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff656000700 1 -- 172.21.15.16:0/4108273465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 msgr2=0x55c5bc2f9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff656000700 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.844+0000 7ff656000700 1 -- 172.21.15.16:0/4108273465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64402e040 con 0x55c5bc2f2b80 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.845+0000 7ff656000700 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc353180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff644005240 tx=0x7ff64400b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fb86c5a0ac30a72 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.845+0000 7ff640c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.845+0000 7ff640c00700 1 -- 172.21.15.16:0/4108273465 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64401d070 con 0x55c5bc2f2b80 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.845+0000 7ff671636b80 1 -- 172.21.15.16:0/4108273465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff64809dda0 con 0x55c5bc2f2b80 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.845+0000 7ff640c00700 1 -- 172.21.15.16:0/4108273465 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff644037030 con 0x55c5bc2f2b80 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.845+0000 7ff640c00700 1 -- 172.21.15.16:0/4108273465 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6440371f0 con 0x55c5bc2f2b80 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.845+0000 7ff671636b80 10 client.25511 fetch_fsmap learned FSMap version 28 2024-04-05T23:34:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.845+0000 7ff671636b80 1 -- 172.21.15.16:0/4108273465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c5bc0c3c70 con 0x55c5bc2f2b80 2024-04-05T23:34:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.845+0000 7ff640c00700 1 -- 172.21.15.16:0/4108273465 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff64809dda0 con 0x55c5bc2f2b80 2024-04-05T23:34:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff640c00700 1 -- 172.21.15.16:0/4108273465 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 28) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff644004d30 con 0x55c5bc2f2b80 2024-04-05T23:34:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff671636b80 10 client.25511 fetch_fsmap finished waiting for FSMap version 28 2024-04-05T23:34:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff671636b80 10 client.25511 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff671636b80 20 client.25511 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff671636b80 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c5bc35b830 0x55c5bc35dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff671636b80 4 client.25511 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:34:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff671636b80 1 -- 172.21.15.16:0/4108273465 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c5bc08cc90 con 0x55c5bc35b830 2024-04-05T23:34:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff656a00700 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c5bc35b830 0x55c5bc35dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff656a00700 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c5bc35b830 0x55c5bc35dc50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14f2a6c4b97b5b16 server_cookie=aca2fa094d1329d in_seq=0 out_seq=0 2024-04-05T23:34:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:00.846+0000 7ff640c00700 10 client.25511 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:01.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:01 smithi027 ceph-mon[78597]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-05T23:34:01.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:01 smithi027 ceph-mon[78597]: mgrmap e25: x(active, since 26s), standbys: y, z 2024-04-05T23:34:02.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:01 smithi016 ceph-mon[38248]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-05T23:34:02.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:01 smithi016 ceph-mon[38248]: mgrmap e25: x(active, since 26s), standbys: y, z 2024-04-05T23:34:02.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:01 smithi130 ceph-mon[79417]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-05T23:34:02.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:01 smithi130 ceph-mon[79417]: mgrmap e25: x(active, since 26s), standbys: y, z 2024-04-05T23:34:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff640c00700 1 -- 172.21.15.16:0/4108273465 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c5bc08cc90 con 0x55c5bc35b830 2024-04-05T23:34:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff640c00700 10 client.25511 handle_command_reply: tid=0 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff671636b80 1 client.25511 shutdown 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff671636b80 1 -- 172.21.15.16:0/4108273465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 msgr2=0x55c5bc353180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff671636b80 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc353180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff644005240 tx=0x7ff64400b840 comp rx=0 tx=0).stop 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff671636b80 1 -- 172.21.15.16:0/4108273465 shutdown_connections 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff671636b80 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c5bc35b830 0x55c5bc35dc50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff671636b80 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5bc2f34a0 0x55c5bc2f9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff640c00700 0 client.25511 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff671636b80 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c5bc2ed5c0 0x55c5bc352c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff671636b80 1 --2- 172.21.15.16:0/4108273465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5bc2f2b80 0x55c5bc353180 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.615+0000 7ff671636b80 1 -- 172.21.15.16:0/4108273465 >> 172.21.15.16:0/4108273465 conn(0x55c5bc2611c0 msgr2=0x55c5bc3026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.616+0000 7ff671636b80 1 -- 172.21.15.16:0/4108273465 shutdown_connections 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.616+0000 7ff671636b80 1 -- 172.21.15.16:0/4108273465 wait complete. 2024-04-05T23:34:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.616+0000 7ff671636b80 20 client.25511 trim_cache size 0 max 16384 2024-04-05T23:34:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:34:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:34:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:34:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.616+0000 7ff65dc00700 1 -- 172.21.15.16:0/1514096704 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff63c0a13b0 msgr2=0x7ff63c0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.616+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff63c0a13b0 0x7ff63c0a3870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff65819aa20 tx=0x7ff6440053d0 comp rx=0 tx=0).stop 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.616+0000 7ff65dc00700 1 -- 172.21.15.16:0/1514096704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff658103250 msgr2=0x7ff6581994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.616+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff658103250 0x7ff6581994d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff64800e9f0 tx=0x7ff64800ed00 comp rx=0 tx=0).stop 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.617+0000 7ff65dc00700 1 -- 172.21.15.16:0/1514096704 shutdown_connections 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.617+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff63c0a13b0 0x7ff63c0a3870 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.617+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff65819ddb0 0x7ff65819e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.617+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff658103250 0x7ff6581994d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.617+0000 7ff65dc00700 1 --2- 172.21.15.16:0/1514096704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff658108370 0x7ff658199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.617+0000 7ff65dc00700 1 -- 172.21.15.16:0/1514096704 >> 172.21.15.16:0/1514096704 conn(0x7ff6580fea90 msgr2=0x7ff6580ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.617+0000 7ff65dc00700 1 -- 172.21.15.16:0/1514096704 shutdown_connections 2024-04-05T23:34:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:02.617+0000 7ff65dc00700 1 -- 172.21.15.16:0/1514096704 wait complete. 2024-04-05T23:34:03.856 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-mon[79417]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 99 KiB/s wr, 59 op/s 2024-04-05T23:34:03.856 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: *** Caught signal (Segmentation fault) ** 2024-04-05T23:34:03.856 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: in thread 7fb66c43c700 thread_name:md_log_replay 2024-04-05T23:34:03.856 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable) 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb67cffcd20] 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2: /usr/bin/ceph-mds(+0x4bd732) [0x563ec2c5d732] 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x563ec2c5f983] 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 4: (MDLog::_replay_thread()+0x699) [0x563ec2cbcea9] 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 5: (MDLog::ReplayThread::entry()+0x11) [0x563ec29756e1] 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fb67cff21ca] 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 7: clone() 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2024-04-05T23:34:03.594+0000 7fb66c43c700 -1 *** Caught signal (Segmentation fault) ** 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: in thread 7fb66c43c700 thread_name:md_log_replay 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable) 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb67cffcd20] 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2: /usr/bin/ceph-mds(+0x4bd732) [0x563ec2c5d732] 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x563ec2c5f983] 2024-04-05T23:34:03.857 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 4: (MDLog::_replay_thread()+0x699) [0x563ec2cbcea9] 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 5: (MDLog::ReplayThread::entry()+0x11) [0x563ec29756e1] 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fb67cff21ca] 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 7: clone() 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 0> 2024-04-05T23:34:03.594+0000 7fb66c43c700 -1 *** Caught signal (Segmentation fault) ** 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: in thread 7fb66c43c700 thread_name:md_log_replay 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable) 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb67cffcd20] 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2: /usr/bin/ceph-mds(+0x4bd732) [0x563ec2c5d732] 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x563ec2c5f983] 2024-04-05T23:34:03.858 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 4: (MDLog::_replay_thread()+0x699) [0x563ec2cbcea9] 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 5: (MDLog::ReplayThread::entry()+0x11) [0x563ec29756e1] 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fb67cff21ca] 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 7: clone() 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 0> 2024-04-05T23:34:03.594+0000 7fb66c43c700 -1 *** Caught signal (Segmentation fault) ** 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: in thread 7fb66c43c700 thread_name:md_log_replay 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable) 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb67cffcd20] 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2: /usr/bin/ceph-mds(+0x4bd732) [0x563ec2c5d732] 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x563ec2c5f983] 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 4: (MDLog::_replay_thread()+0x699) [0x563ec2cbcea9] 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 5: (MDLog::ReplayThread::entry()+0x11) [0x563ec29756e1] 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fb67cff21ca] 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 7: clone() 2024-04-05T23:34:03.859 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-05T23:34:03.860 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[117263]: 2024-04-05T23:34:03.860 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:03 smithi130 systemd[1]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.f.service: Main process exited, code=exited, status=139/n/a 2024-04-05T23:34:03.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:03 smithi027 ceph-mon[78597]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 99 KiB/s wr, 59 op/s 2024-04-05T23:34:04.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:03 smithi016 ceph-mon[38248]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 99 KiB/s wr, 59 op/s 2024-04-05T23:34:04.334 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:04 smithi130 systemd[1]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.f.service: Failed with result 'exit-code'. 2024-04-05T23:34:05.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:05 smithi027 ceph-mon[78597]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-05T23:34:06.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:05 smithi016 ceph-mon[38248]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-05T23:34:06.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:05 smithi130 ceph-mon[79417]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-05T23:34:07.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:07 smithi027 ceph-mon[78597]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-04-05T23:34:08.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:07 smithi016 ceph-mon[38248]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-04-05T23:34:08.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:07 smithi130 ceph-mon[79417]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-04-05T23:34:09.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:09 smithi027 ceph-mon[78597]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-05T23:34:10.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:09 smithi016 ceph-mon[38248]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-05T23:34:10.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:09 smithi130 ceph-mon[79417]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-05T23:34:11.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:11 smithi027 ceph-mon[78597]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-05T23:34:12.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:11 smithi016 ceph-mon[38248]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-05T23:34:12.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:11 smithi130 ceph-mon[79417]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-05T23:34:13.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:13 smithi027 ceph-mon[78597]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 31 op/s 2024-04-05T23:34:14.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:13 smithi016 ceph-mon[38248]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 31 op/s 2024-04-05T23:34:14.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:13 smithi130 ceph-mon[79417]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 31 op/s 2024-04-05T23:34:14.337 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:14 smithi130 systemd[1]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.f.service: Service RestartSec=10s expired, scheduling restart. 2024-04-05T23:34:14.337 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:14 smithi130 systemd[1]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-04-05T23:34:14.337 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:14 smithi130 systemd[1]: Stopped Ceph mds.f for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:34:14.337 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:14 smithi130 systemd[1]: Starting Ceph mds.f for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:34:14.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:14 smithi130 ceph-mon[79417]: daemon mds.f restarted 2024-04-05T23:34:14.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:14 smithi130 ceph-mon[79417]: osdmap e83: 12 total, 12 up, 12 in 2024-04-05T23:34:14.834 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:14 smithi130 ceph-mds[121021]: -- 172.21.15.130:0/1348146184 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562c82b981a0 con 0x562c82b6ec00 2024-04-05T23:34:14.835 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: starting mds.f at 2024-04-05T23:34:14.835 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:14 smithi130 bash[120983]: 8e448bc3d6575da047e5d5abff65eb3e42ccc70c75550cf722046939ba4664c2 2024-04-05T23:34:14.835 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:34:14 smithi130 systemd[1]: Started Ceph mds.f for b3471b30-f3a3-11ee-b649-cb9ed24678a4. 2024-04-05T23:34:14.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:14 smithi027 ceph-mon[78597]: daemon mds.f restarted 2024-04-05T23:34:14.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:14 smithi027 ceph-mon[78597]: osdmap e83: 12 total, 12 up, 12 in 2024-04-05T23:34:15.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:14 smithi016 ceph-mon[38248]: daemon mds.f restarted 2024-04-05T23:34:15.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:14 smithi016 ceph-mon[38248]: osdmap e83: 12 total, 12 up, 12 in 2024-04-05T23:34:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.789+0000 7f14b8c00700 1 -- 172.21.15.16:0/2633970874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4068be0 msgr2=0x7f14b410dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.790+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2633970874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4068be0 0x7f14b410dbe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00b0a0 tx=0x7f14ac00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.790+0000 7f14b8c00700 1 -- 172.21.15.16:0/2633970874 shutdown_connections 2024-04-05T23:34:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.790+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2633970874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4068be0 0x7f14b410dbe0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.790+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2633970874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14b4106f40 0x7f14b4107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.790+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2633970874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b4068220 0x7f14b40686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.790+0000 7f14b8c00700 1 -- 172.21.15.16:0/2633970874 >> 172.21.15.16:0/2633970874 conn(0x7f14b4076e50 msgr2=0x7f14b4077260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.790+0000 7f14b8c00700 1 -- 172.21.15.16:0/2633970874 shutdown_connections 2024-04-05T23:34:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.790+0000 7f14b8c00700 1 -- 172.21.15.16:0/2633970874 wait complete. 2024-04-05T23:34:15.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.791+0000 7f14b8c00700 1 Processor -- start 2024-04-05T23:34:15.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.791+0000 7f14b8c00700 1 -- start start 2024-04-05T23:34:15.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b8c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4068220 0x7f14b4199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b8c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b4068be0 0x7f14b4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b8c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14b4106f40 0x7f14b419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b8c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b419e460 con 0x7f14b4068be0 2024-04-05T23:34:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b8c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b419e5d0 con 0x7f14b4068220 2024-04-05T23:34:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b8c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b419e790 con 0x7f14b4106f40 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4068220 0x7f14b4199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b2800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14b4106f40 0x7f14b419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4068220 0x7f14b4199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42490/0 (socket says 172.21.15.16:42490) 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b1e00700 1 -- 172.21.15.16:0/2840323634 learned_addr learned my addr 172.21.15.16:0/2840323634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b2800700 1 -- 172.21.15.16:0/2840323634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4068220 msgr2=0x7f14b4199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.792+0000 7f14b2800700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4068220 0x7f14b4199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.793+0000 7f14b2800700 1 -- 172.21.15.16:0/2840323634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b4068be0 msgr2=0x7f14b4199a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.793+0000 7f14b2800700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b4068be0 0x7f14b4199a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.793+0000 7f14b2800700 1 -- 172.21.15.16:0/2840323634 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4009520 con 0x7f14b4106f40 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.793+0000 7f14b2800700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14b4106f40 0x7f14b419de20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00b070 tx=0x7f14ac007860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.793+0000 7f14aaa00700 1 -- 172.21.15.16:0/2840323634 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac00e040 con 0x7f14b4106f40 2024-04-05T23:34:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.793+0000 7f14b8c00700 1 -- 172.21.15.16:0/2840323634 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac009d00 con 0x7f14b4106f40 2024-04-05T23:34:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.793+0000 7f14b8c00700 1 -- 172.21.15.16:0/2840323634 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b41a60d0 con 0x7f14b4106f40 2024-04-05T23:34:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.794+0000 7f14aaa00700 1 -- 172.21.15.16:0/2840323634 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14ac009b80 con 0x7f14b4106f40 2024-04-05T23:34:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.794+0000 7f14aaa00700 1 -- 172.21.15.16:0/2840323634 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac01dd30 con 0x7f14b4106f40 2024-04-05T23:34:15.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.798+0000 7f14aaa00700 1 -- 172.21.15.16:0/2840323634 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f14ac005040 con 0x7f14b4106f40 2024-04-05T23:34:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.800+0000 7f14cc77db80 1 Processor -- start 2024-04-05T23:34:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.800+0000 7f14cc77db80 1 -- start start 2024-04-05T23:34:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.801+0000 7f14aaa00700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14980a14b0 0x7f14980a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.801+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x557083088750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.801+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 0x557083082840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.801+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083082d80 0x5570830831e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.801+0000 7f14cc77db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557082fe0b80 con 0x557083088370 2024-04-05T23:34:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.801+0000 7f14cc77db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557082fd1930 con 0x557083088c90 2024-04-05T23:34:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.801+0000 7f14cc77db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557082fcf2f0 con 0x557083082d80 2024-04-05T23:34:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.801+0000 7f14aaa00700 1 -- 172.21.15.16:0/2840323634 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f14ac0ce450 con 0x7f14b4106f40 2024-04-05T23:34:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x557083088750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x557083088750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42560/0 (socket says 172.21.15.16:42560) 2024-04-05T23:34:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b1e00700 1 -- 172.21.15.16:0/566261191 learned_addr learned my addr 172.21.15.16:0/566261191 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b2800700 1 --2- 172.21.15.16:0/566261191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083082d80 0x5570830831e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b1400700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14980a14b0 0x7f14980a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b2800700 1 -- 172.21.15.16:0/566261191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 msgr2=0x557083082840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:34:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b2800700 1 --2- 172.21.15.16:0/566261191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 0x557083082840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b2800700 1 -- 172.21.15.16:0/566261191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 msgr2=0x557083088750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b2800700 1 --2- 172.21.15.16:0/566261191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x557083088750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b2800700 1 -- 172.21.15.16:0/566261191 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557082e4c350 con 0x557083082d80 2024-04-05T23:34:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.802+0000 7f14b2800700 1 --2- 172.21.15.16:0/566261191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083082d80 0x5570830831e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0cb9f0 tx=0x7f14ac09aeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7210a86eff033a55 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.803+0000 7f149f000700 1 -- 172.21.15.16:0/566261191 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac09ba70 con 0x557083082d80 2024-04-05T23:34:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.803+0000 7f149f000700 1 -- 172.21.15.16:0/566261191 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14ac01de90 con 0x557083082d80 2024-04-05T23:34:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.803+0000 7f149f000700 1 -- 172.21.15.16:0/566261191 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac09a860 con 0x557083082d80 2024-04-05T23:34:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14b1400700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14980a14b0 0x7f14980a3970 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f14b419aa90 tx=0x7f14a0009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14cc77db80 1 -- 172.21.15.16:0/566261191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083082d80 msgr2=0x5570830831e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14cc77db80 1 --2- 172.21.15.16:0/566261191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083082d80 0x5570830831e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0cb9f0 tx=0x7f14ac09aeb0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14cc77db80 1 -- 172.21.15.16:0/566261191 shutdown_connections 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14cc77db80 1 --2- 172.21.15.16:0/566261191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x557083088750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14cc77db80 1 --2- 172.21.15.16:0/566261191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083082d80 0x5570830831e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14cc77db80 1 --2- 172.21.15.16:0/566261191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 0x557083082840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14cc77db80 1 -- 172.21.15.16:0/566261191 >> 172.21.15.16:0/566261191 conn(0x557082fee1c0 msgr2=0x557083074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14cc77db80 1 -- 172.21.15.16:0/566261191 shutdown_connections 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.804+0000 7f14cc77db80 1 -- 172.21.15.16:0/566261191 wait complete. 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.805+0000 7f14cc77db80 1 -- start start 2024-04-05T23:34:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.805+0000 7f14cc77db80 1 Processor -- start 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.805+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083082d80 0x55708308c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.805+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x55708308ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.805+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 0x5570830814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.805+0000 7f14cc77db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557082fcabe0 con 0x557083082d80 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.805+0000 7f14cc77db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557082f66d20 con 0x557083088c90 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.805+0000 7f14cc77db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557082f669c0 con 0x557083088370 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.806+0000 7f14cc77db80 10 client.0 _ll_register_callbacks cb 0x557082dd2340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.806+0000 7f14b1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083082d80 0x55708308c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.806+0000 7f14b1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083082d80 0x55708308c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42570/0 (socket says 172.21.15.16:42570) 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.806+0000 7f14b1e00700 1 -- 172.21.15.16:0/297354151 learned_addr learned my addr 172.21.15.16:0/297354151 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.806+0000 7f14b2800700 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 0x5570830814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.806+0000 7f14b1400700 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x55708308ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f14b2800700 1 -- 172.21.15.16:0/297354151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 msgr2=0x55708308ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f14b2800700 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x55708308ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f14b2800700 1 -- 172.21.15.16:0/297354151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083082d80 msgr2=0x55708308c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f14b2800700 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083082d80 0x55708308c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f14b2800700 1 -- 172.21.15.16:0/297354151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b41a60d0 con 0x557083088c90 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f14b1e00700 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083082d80 0x55708308c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f14b2800700 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 0x5570830814c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14ac097870 tx=0x7f14ac09b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ca153f2349975bf server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f1493400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f1493400700 1 -- 172.21.15.16:0/297354151 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac0b5ea0 con 0x557083088c90 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f1493400700 1 -- 172.21.15.16:0/297354151 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f14ac09b370 con 0x557083088c90 2024-04-05T23:34:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.807+0000 7f14cc77db80 1 -- 172.21.15.16:0/297354151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55708306d1e0 con 0x557083088c90 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f1493400700 1 -- 172.21.15.16:0/297354151 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14ac09f1a0 con 0x557083088c90 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f14cc77db80 10 client.25523 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f14cc77db80 1 -- 172.21.15.16:0/297354151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557082df8b00 con 0x557083088c90 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f1493400700 1 -- 172.21.15.16:0/297354151 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55708306d1e0 con 0x557083088c90 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f1493400700 1 -- 172.21.15.16:0/297354151 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f14ac0a1130 con 0x557083088c90 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f14cc77db80 10 client.25523 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f14cc77db80 10 client.25523 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f14cc77db80 20 client.25523 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f14cc77db80 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5570830a4a10 0x5570830a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.808+0000 7f14cc77db80 4 client.25523 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.809+0000 7f14cc77db80 1 -- 172.21.15.16:0/297354151 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557082fc36a0 con 0x5570830a4a10 2024-04-05T23:34:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.809+0000 7f14b1e00700 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5570830a4a10 0x5570830a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.810+0000 7f14b1e00700 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5570830a4a10 0x5570830a6e00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7215d608a6d7f539 server_cookie=71a0aad08fdbce91 in_seq=0 out_seq=0 2024-04-05T23:34:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.810+0000 7f1493400700 10 client.25523 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.810+0000 7f1493400700 1 -- 172.21.15.16:0/297354151 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557082fc36a0 con 0x5570830a4a10 2024-04-05T23:34:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.810+0000 7f1493400700 10 client.25523 handle_command_reply: tid=0 2024-04-05T23:34:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.810+0000 7f14cc77db80 1 client.25523 shutdown 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 -- 172.21.15.16:0/297354151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 msgr2=0x5570830814c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 0x5570830814c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14ac097870 tx=0x7f14ac09b800 comp rx=0 tx=0).stop 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 -- 172.21.15.16:0/297354151 shutdown_connections 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5570830a4a10 0x5570830a6e00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083082d80 0x55708308c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083088c90 0x5570830814c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 --2- 172.21.15.16:0/297354151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x55708308ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 -- 172.21.15.16:0/297354151 >> 172.21.15.16:0/297354151 conn(0x557082fee1c0 msgr2=0x557083074280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f1493400700 0 client.25523 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 -- 172.21.15.16:0/297354151 shutdown_connections 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 1 -- 172.21.15.16:0/297354151 wait complete. 2024-04-05T23:34:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.811+0000 7f14cc77db80 20 client.25523 trim_cache size 0 max 16384 2024-04-05T23:34:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.828+0000 7f14cc77db80 1 Processor -- start 2024-04-05T23:34:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.828+0000 7f14cc77db80 1 -- start start 2024-04-05T23:34:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 0x55708308a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x55708308bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088c90 0x55708307e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14cc77db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557082e4e640 con 0x557083088370 2024-04-05T23:34:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14cc77db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557083076b60 con 0x557083082d80 2024-04-05T23:34:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14cc77db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55708308ef90 con 0x557083088c90 2024-04-05T23:34:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14b1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 0x55708308a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14b1400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x55708308bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14b2800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088c90 0x55708307e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14b1400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x55708308bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42582/0 (socket says 172.21.15.16:42582) 2024-04-05T23:34:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.829+0000 7f14b1400700 1 -- 172.21.15.16:0/680280398 learned_addr learned my addr 172.21.15.16:0/680280398 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.830+0000 7f14b1400700 1 -- 172.21.15.16:0/680280398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088c90 msgr2=0x55708307e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.830+0000 7f14b1400700 1 --2- 172.21.15.16:0/680280398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088c90 0x55708307e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.830+0000 7f14b1400700 1 -- 172.21.15.16:0/680280398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 msgr2=0x55708308a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.830+0000 7f14b1400700 1 --2- 172.21.15.16:0/680280398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 0x55708308a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.830+0000 7f14b1400700 1 -- 172.21.15.16:0/680280398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557082f90e40 con 0x557083088370 2024-04-05T23:34:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.830+0000 7f14b2800700 1 --2- 172.21.15.16:0/680280398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088c90 0x55708307e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.830+0000 7f14b1400700 1 --2- 172.21.15.16:0/680280398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x55708308bdc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f14a000ebe0 tx=0x7f14a0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8222004083f7652c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.831+0000 7f149fa00700 1 -- 172.21.15.16:0/680280398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0004a30 con 0x557083088370 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.831+0000 7f149fa00700 1 -- 172.21.15.16:0/680280398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14a0021070 con 0x557083088370 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.831+0000 7f149fa00700 1 -- 172.21.15.16:0/680280398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a001d760 con 0x557083088370 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.831+0000 7f14cc77db80 1 -- 172.21.15.16:0/680280398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 msgr2=0x55708308bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.831+0000 7f14cc77db80 1 --2- 172.21.15.16:0/680280398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x55708308bdc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f14a000ebe0 tx=0x7f14a0009f90 comp rx=0 tx=0).stop 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.832+0000 7f14cc77db80 1 -- 172.21.15.16:0/680280398 shutdown_connections 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.832+0000 7f14cc77db80 1 --2- 172.21.15.16:0/680280398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088c90 0x55708307e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.832+0000 7f14cc77db80 1 --2- 172.21.15.16:0/680280398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 0x55708308a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.832+0000 7f14cc77db80 1 --2- 172.21.15.16:0/680280398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088370 0x55708308bdc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.832+0000 7f14cc77db80 1 -- 172.21.15.16:0/680280398 >> 172.21.15.16:0/680280398 conn(0x557082fee1c0 msgr2=0x5570830a6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.832+0000 7f14cc77db80 1 -- 172.21.15.16:0/680280398 shutdown_connections 2024-04-05T23:34:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.832+0000 7f14cc77db80 1 -- 172.21.15.16:0/680280398 wait complete. 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.832+0000 7f14cc77db80 1 -- start start 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.832+0000 7f14cc77db80 1 Processor -- start 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 0x557083083e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x5570830843c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14cc77db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088c90 0x557083084900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14cc77db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557083084e40 con 0x557083088c90 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14cc77db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557083084fb0 con 0x557083082d80 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14cc77db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557083085170 con 0x557083088370 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14cc77db80 10 client.0 _ll_register_callbacks cb 0x557082dd2340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14b1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x5570830843c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14b1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x5570830843c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39836/0 (socket says 172.21.15.16:39836) 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.833+0000 7f14b1400700 1 -- 172.21.15.16:0/1824007422 learned_addr learned my addr 172.21.15.16:0/1824007422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f14b1e00700 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 0x557083083e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f14b1400700 1 -- 172.21.15.16:0/1824007422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 msgr2=0x557083083e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f14b2800700 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088c90 0x557083084900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f14b1400700 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 0x557083083e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f14b1400700 1 -- 172.21.15.16:0/1824007422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088c90 msgr2=0x557083084900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f14b1400700 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088c90 0x557083084900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f14b1400700 1 -- 172.21.15.16:0/1824007422 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557082df8b00 con 0x557083088370 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f14b2800700 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088c90 0x557083084900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f14b1400700 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x5570830843c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f14a000e670 tx=0x7f14a0004800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ce08532dd104b13 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.834+0000 7f1493400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.835+0000 7f1493400700 1 -- 172.21.15.16:0/1824007422 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a002a040 con 0x557083088370 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.835+0000 7f14cc77db80 1 -- 172.21.15.16:0/1824007422 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14ac01de90 con 0x557083088370 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.835+0000 7f1493400700 1 -- 172.21.15.16:0/1824007422 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f14a0004aa0 con 0x557083088370 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.835+0000 7f1493400700 1 -- 172.21.15.16:0/1824007422 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14a0005390 con 0x557083088370 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.835+0000 7f14cc77db80 10 client.25552 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.835+0000 7f14cc77db80 1 -- 172.21.15.16:0/1824007422 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f14a000e8c0 con 0x557083088370 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.835+0000 7f1493400700 1 -- 172.21.15.16:0/1824007422 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14a003c020 con 0x557083088370 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.836+0000 7f1493400700 1 -- 172.21.15.16:0/1824007422 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f14a0026480 con 0x557083088370 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.836+0000 7f14cc77db80 10 client.25552 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.836+0000 7f14cc77db80 10 client.25552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.836+0000 7f14cc77db80 20 client.25552 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.836+0000 7f14cc77db80 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5570830f93d0 0x5570830fb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.836+0000 7f14cc77db80 4 client.25552 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:34:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.836+0000 7f14cc77db80 1 -- 172.21.15.16:0/1824007422 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557082e19c90 con 0x5570830f93d0 2024-04-05T23:34:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.836+0000 7f14b1e00700 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5570830f93d0 0x5570830fb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.837+0000 7f14b1e00700 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5570830f93d0 0x5570830fb7f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca9179697f317545 server_cookie=d6a71e1810b8679b in_seq=0 out_seq=0 2024-04-05T23:34:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.837+0000 7f1493400700 10 client.25552 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.841+0000 7f1493400700 1 -- 172.21.15.16:0/1824007422 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557082e19c90 con 0x5570830f93d0 2024-04-05T23:34:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.841+0000 7f1493400700 10 client.25552 handle_command_reply: tid=0 2024-04-05T23:34:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.841+0000 7f14cc77db80 1 client.25552 shutdown 2024-04-05T23:34:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 -- 172.21.15.16:0/1824007422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 msgr2=0x5570830843c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x5570830843c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f14a000e670 tx=0x7f14a0004800 comp rx=0 tx=0).stop 2024-04-05T23:34:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 -- 172.21.15.16:0/1824007422 shutdown_connections 2024-04-05T23:34:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5570830f93d0 0x5570830fb7f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557083088c90 0x557083084900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557083082d80 0x557083083e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 --2- 172.21.15.16:0/1824007422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557083088370 0x5570830843c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 -- 172.21.15.16:0/1824007422 >> 172.21.15.16:0/1824007422 conn(0x557082fee1c0 msgr2=0x55708308c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f1493400700 0 client.25552 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 -- 172.21.15.16:0/1824007422 shutdown_connections 2024-04-05T23:34:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 1 -- 172.21.15.16:0/1824007422 wait complete. 2024-04-05T23:34:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.842+0000 7f14cc77db80 20 client.25552 trim_cache size 0 max 16384 2024-04-05T23:34:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:34:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:34:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:34:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:34:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 -- 172.21.15.16:0/2840323634 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14980a14b0 msgr2=0x7f14980a3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14980a14b0 0x7f14980a3970 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f14b419aa90 tx=0x7f14a0009260 comp rx=0 tx=0).stop 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 -- 172.21.15.16:0/2840323634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14b4106f40 msgr2=0x7f14b419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14b4106f40 0x7f14b419de20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00b070 tx=0x7f14ac007860 comp rx=0 tx=0).stop 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 -- 172.21.15.16:0/2840323634 shutdown_connections 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f14980a14b0 0x7f14980a3970 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14b4106f40 0x7f14b419de20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4068220 0x7f14b4199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 --2- 172.21.15.16:0/2840323634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b4068be0 0x7f14b4199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.843+0000 7f14b8c00700 1 -- 172.21.15.16:0/2840323634 >> 172.21.15.16:0/2840323634 conn(0x7f14b4076e50 msgr2=0x7f14b410bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.844+0000 7f14b8c00700 1 -- 172.21.15.16:0/2840323634 shutdown_connections 2024-04-05T23:34:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:15.844+0000 7f14b8c00700 1 -- 172.21.15.16:0/2840323634 wait complete. 2024-04-05T23:34:15.849 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-05T23:34:15.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:15 smithi027 ceph-mon[78597]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-05T23:34:15.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:15 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6834/1942303111,v1:172.21.15.130:6835/1942303111] up:boot 2024-04-05T23:34:15.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:15 smithi027 ceph-mon[78597]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-05T23:34:15.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:15 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:34:15.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:15 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.085+0000 7f64c4e00700 1 -- 172.21.15.16:0/585962284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 msgr2=0x7f64c0100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.085+0000 7f64c4e00700 1 --2- 172.21.15.16:0/585962284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 0x7f64c0100380 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f64b00098f0 tx=0x7f64b0009c00 comp rx=0 tx=0).stop 2024-04-05T23:34:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.085+0000 7f64c4e00700 1 -- 172.21.15.16:0/585962284 shutdown_connections 2024-04-05T23:34:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.085+0000 7f64c4e00700 1 --2- 172.21.15.16:0/585962284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c01050a0 0x7f64c010cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.085+0000 7f64c4e00700 1 --2- 172.21.15.16:0/585962284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 0x7f64c0100380 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.085+0000 7f64c4e00700 1 --2- 172.21.15.16:0/585962284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c0100950 0x7f64c0104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.085+0000 7f64c4e00700 1 -- 172.21.15.16:0/585962284 >> 172.21.15.16:0/585962284 conn(0x7f64c0076e80 msgr2=0x7f64c0077290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.085+0000 7f64c4e00700 1 -- 172.21.15.16:0/585962284 shutdown_connections 2024-04-05T23:34:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.085+0000 7f64c4e00700 1 -- 172.21.15.16:0/585962284 wait complete. 2024-04-05T23:34:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.086+0000 7f64c4e00700 1 Processor -- start 2024-04-05T23:34:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.087+0000 7f64c4e00700 1 -- start start 2024-04-05T23:34:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.087+0000 7f64c4e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 0x7f64c01995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.087+0000 7f64c4e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c01050a0 0x7f64c0199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.087+0000 7f64c4e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c019ded0 0x7f64c019e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.087+0000 7f64c4e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c019e950 con 0x7f64c019ded0 2024-04-05T23:34:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.087+0000 7f64c4e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c019eac0 con 0x7f64c01050a0 2024-04-05T23:34:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.087+0000 7f64c4e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c019ec80 con 0x7f64c00fffa0 2024-04-05T23:34:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64bde00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 0x7f64c01995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64be800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c019ded0 0x7f64c019e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64bde00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 0x7f64c01995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39860/0 (socket says 172.21.15.16:39860) 2024-04-05T23:34:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64bde00700 1 -- 172.21.15.16:0/3951164488 learned_addr learned my addr 172.21.15.16:0/3951164488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64bd400700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c01050a0 0x7f64c0199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64be800700 1 -- 172.21.15.16:0/3951164488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 msgr2=0x7f64c01995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64be800700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 0x7f64c01995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64be800700 1 -- 172.21.15.16:0/3951164488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c01050a0 msgr2=0x7f64c0199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64be800700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c01050a0 0x7f64c0199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64be800700 1 -- 172.21.15.16:0/3951164488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0009520 con 0x7f64c019ded0 2024-04-05T23:34:16.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.088+0000 7f64bde00700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 0x7f64c01995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:16.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.089+0000 7f64be800700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c019ded0 0x7f64c019e350 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f64b800efc0 tx=0x7f64b800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:16.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.089+0000 7f64b6a00700 1 -- 172.21.15.16:0/3951164488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8018070 con 0x7f64c019ded0 2024-04-05T23:34:16.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.089+0000 7f64b6a00700 1 -- 172.21.15.16:0/3951164488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f64b800f040 con 0x7f64c019ded0 2024-04-05T23:34:16.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.089+0000 7f64c4e00700 1 -- 172.21.15.16:0/3951164488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c01a6180 con 0x7f64c019ded0 2024-04-05T23:34:16.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.089+0000 7f64c4e00700 1 -- 172.21.15.16:0/3951164488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c01a6660 con 0x7f64c019ded0 2024-04-05T23:34:16.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.089+0000 7f64b6a00700 1 -- 172.21.15.16:0/3951164488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b80077f0 con 0x7f64c019ded0 2024-04-05T23:34:16.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:15 smithi016 ceph-mon[38248]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-05T23:34:16.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:15 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6834/1942303111,v1:172.21.15.130:6835/1942303111] up:boot 2024-04-05T23:34:16.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:15 smithi016 ceph-mon[38248]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-05T23:34:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:15 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:34:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:15 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:16.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:15 smithi130 ceph-mon[79417]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-05T23:34:16.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:15 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6834/1942303111,v1:172.21.15.130:6835/1942303111] up:boot 2024-04-05T23:34:16.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:15 smithi130 ceph-mon[79417]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-05T23:34:16.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:15 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:34:16.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:15 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.094+0000 7f64d8a17b80 1 Processor -- start 2024-04-05T23:34:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.095+0000 7f64b6a00700 1 -- 172.21.15.16:0/3951164488 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f64b8007950 con 0x7f64c019ded0 2024-04-05T23:34:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.095+0000 7f64d8a17b80 1 -- start start 2024-04-05T23:34:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.095+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 0x55ec62a15560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.095+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a0f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.095+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.095+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec62975be0 con 0x55ec62a0f2e0 2024-04-05T23:34:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.095+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec62966990 con 0x55ec62a15180 2024-04-05T23:34:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.095+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec62964350 con 0x55ec62a0fca0 2024-04-05T23:34:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.096+0000 7f64bd400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 0x55ec62a15560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.096+0000 7f64bd400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 0x55ec62a15560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42546/0 (socket says 172.21.15.16:42546) 2024-04-05T23:34:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.096+0000 7f64bd400700 1 -- 172.21.15.16:0/3195863879 learned_addr learned my addr 172.21.15.16:0/3195863879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.096+0000 7f64bde00700 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a0f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.096+0000 7f64bd400700 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64bd400700 1 -- 172.21.15.16:0/3195863879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 msgr2=0x55ec62a15560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64bd400700 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 0x55ec62a15560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64b6a00700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f64a40a13b0 0x7f64a40a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64bd400700 1 -- 172.21.15.16:0/3195863879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 msgr2=0x55ec62a0f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64bd400700 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a0f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64bd400700 1 -- 172.21.15.16:0/3195863879 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec627e13b0 con 0x55ec62a0fca0 2024-04-05T23:34:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64bde00700 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a0f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64b6a00700 1 -- 172.21.15.16:0/3951164488 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f64b80cca00 con 0x7f64c019ded0 2024-04-05T23:34:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64bd400700 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1bf10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f64ac011860 tx=0x7f64ac009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc210c188ac17ade server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.097+0000 7f64be800700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f64a40a13b0 0x7f64a40a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.098+0000 7f64ab400700 1 -- 172.21.15.16:0/3195863879 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac004830 con 0x55ec62a0fca0 2024-04-05T23:34:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.098+0000 7f64ab400700 1 -- 172.21.15.16:0/3195863879 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f64ac011540 con 0x55ec62a0fca0 2024-04-05T23:34:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.098+0000 7f64ab400700 1 -- 172.21.15.16:0/3195863879 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00f3c0 con 0x55ec62a0fca0 2024-04-05T23:34:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64be800700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f64a40a13b0 0x7f64a40a3870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f64b80ca040 tx=0x7f64b8009ef0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3195863879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 msgr2=0x55ec62a1bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1bf10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f64ac011860 tx=0x7f64ac009310 comp rx=0 tx=0).stop 2024-04-05T23:34:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3195863879 shutdown_connections 2024-04-05T23:34:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1bf10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 0x55ec62a15560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3195863879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a0f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3195863879 >> 172.21.15.16:0/3195863879 conn(0x55ec62983220 msgr2=0x55ec62a0a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3195863879 shutdown_connections 2024-04-05T23:34:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.099+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3195863879 wait complete. 2024-04-05T23:34:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.100+0000 7f64d8a17b80 1 -- start start 2024-04-05T23:34:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.100+0000 7f64d8a17b80 1 Processor -- start 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0f2e0 0x55ec62a1bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a15180 0x55ec62a19730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec6295fc40 con 0x55ec62a15180 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec628fbd80 con 0x55ec62a0fca0 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec628fba20 con 0x55ec62a0f2e0 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64d8a17b80 10 client.0 _ll_register_callbacks cb 0x55ec627673a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42562/0 (socket says 172.21.15.16:42562) 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64bde00700 1 -- 172.21.15.16:0/1538559760 learned_addr learned my addr 172.21.15.16:0/1538559760 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64bde00700 1 -- 172.21.15.16:0/1538559760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0f2e0 msgr2=0x55ec62a1bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64bde00700 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0f2e0 0x55ec62a1bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64bde00700 1 -- 172.21.15.16:0/1538559760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a15180 msgr2=0x55ec62a19730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64bde00700 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a15180 0x55ec62a19730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.101+0000 7f64bde00700 1 -- 172.21.15.16:0/1538559760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c00680f0 con 0x55ec62a0fca0 2024-04-05T23:34:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.102+0000 7f64bde00700 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19120 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f64b0005950 tx=0x7f64b000b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=778111d997733832 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.102+0000 7f649fe00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.102+0000 7f649fe00700 1 -- 172.21.15.16:0/1538559760 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b001c040 con 0x55ec62a0fca0 2024-04-05T23:34:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.102+0000 7f649fe00700 1 -- 172.21.15.16:0/1538559760 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f64b0005040 con 0x55ec62a0fca0 2024-04-05T23:34:16.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.102+0000 7f64d8a17b80 1 -- 172.21.15.16:0/1538559760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec62a02490 con 0x55ec62a0fca0 2024-04-05T23:34:16.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.103+0000 7f649fe00700 1 -- 172.21.15.16:0/1538559760 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64b000bed0 con 0x55ec62a0fca0 2024-04-05T23:34:16.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.103+0000 7f64d8a17b80 10 client.25541 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:16.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.103+0000 7f64d8a17b80 1 -- 172.21.15.16:0/1538559760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f64ac019040 con 0x55ec62a0fca0 2024-04-05T23:34:16.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.103+0000 7f649fe00700 1 -- 172.21.15.16:0/1538559760 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ec62a02490 con 0x55ec62a0fca0 2024-04-05T23:34:16.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.104+0000 7f649fe00700 1 -- 172.21.15.16:0/1538559760 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f64b0015020 con 0x55ec62a0fca0 2024-04-05T23:34:16.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.104+0000 7f64d8a17b80 10 client.25541 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.104+0000 7f64d8a17b80 10 client.25541 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.104+0000 7f64d8a17b80 20 client.25541 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.104+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ec62a293d0 0x55ec62a2b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.104+0000 7f64d8a17b80 4 client.25541 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.104+0000 7f64d8a17b80 1 -- 172.21.15.16:0/1538559760 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec62958700 con 0x55ec62a293d0 2024-04-05T23:34:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.104+0000 7f64bd400700 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ec62a293d0 0x55ec62a2b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.105+0000 7f64bd400700 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ec62a293d0 0x55ec62a2b7f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b9240966f556597 server_cookie=4d05df5044fce69 in_seq=0 out_seq=0 2024-04-05T23:34:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.105+0000 7f649fe00700 10 client.25541 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.106+0000 7f649fe00700 1 -- 172.21.15.16:0/1538559760 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ec62958700 con 0x55ec62a293d0 2024-04-05T23:34:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.106+0000 7f649fe00700 10 client.25541 handle_command_reply: tid=0 2024-04-05T23:34:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.106+0000 7f64d8a17b80 1 client.25541 shutdown 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.106+0000 7f64d8a17b80 1 -- 172.21.15.16:0/1538559760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 msgr2=0x55ec62a19120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.106+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19120 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f64b0005950 tx=0x7f64b000b7e0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f64d8a17b80 1 -- 172.21.15.16:0/1538559760 shutdown_connections 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ec62a293d0 0x55ec62a2b7f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a15180 0x55ec62a19730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0f2e0 0x55ec62a1bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/1538559760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19120 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f64d8a17b80 1 -- 172.21.15.16:0/1538559760 >> 172.21.15.16:0/1538559760 conn(0x55ec62983220 msgr2=0x55ec62a04460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f649fe00700 0 client.25541 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:16.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f64d8a17b80 1 -- 172.21.15.16:0/1538559760 shutdown_connections 2024-04-05T23:34:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f64d8a17b80 1 -- 172.21.15.16:0/1538559760 wait complete. 2024-04-05T23:34:16.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.107+0000 7f64d8a17b80 20 client.25541 trim_cache size 0 max 16384 2024-04-05T23:34:16.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.139+0000 7f64d8a17b80 1 Processor -- start 2024-04-05T23:34:16.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.139+0000 7f64d8a17b80 1 -- start start 2024-04-05T23:34:16.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.140+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a16df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.140+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.140+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a15180 0x55ec62a04560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.140+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec627e36a0 con 0x55ec62a0f2e0 2024-04-05T23:34:16.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.140+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec62a0b140 con 0x55ec62a0fca0 2024-04-05T23:34:16.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.140+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec62a0bb50 con 0x55ec62a15180 2024-04-05T23:34:16.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.140+0000 7f64bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.140+0000 7f64bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42568/0 (socket says 172.21.15.16:42568) 2024-04-05T23:34:16.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.140+0000 7f64bde00700 1 -- 172.21.15.16:0/4089446739 learned_addr learned my addr 172.21.15.16:0/4089446739 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:16.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.141+0000 7f64bde00700 1 -- 172.21.15.16:0/4089446739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a15180 msgr2=0x55ec62a04560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-05T23:34:16.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.141+0000 7f64bde00700 1 --2- 172.21.15.16:0/4089446739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a15180 0x55ec62a04560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.141+0000 7f64bde00700 1 -- 172.21.15.16:0/4089446739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 msgr2=0x55ec62a16df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-05T23:34:16.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.141+0000 7f64bde00700 1 --2- 172.21.15.16:0/4089446739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a16df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.141+0000 7f64bde00700 1 -- 172.21.15.16:0/4089446739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec62a30420 con 0x55ec62a0fca0 2024-04-05T23:34:16.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.141+0000 7f64bde00700 1 --2- 172.21.15.16:0/4089446739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f64b0003e20 tx=0x7f64b0005b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9abcef2568ffbb9b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:16.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.141+0000 7f64abe00700 1 -- 172.21.15.16:0/4089446739 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b001c040 con 0x55ec62a0fca0 2024-04-05T23:34:16.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.141+0000 7f64abe00700 1 -- 172.21.15.16:0/4089446739 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f64b001d040 con 0x55ec62a0fca0 2024-04-05T23:34:16.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.141+0000 7f64abe00700 1 -- 172.21.15.16:0/4089446739 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b00052d0 con 0x55ec62a0fca0 2024-04-05T23:34:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.142+0000 7f64d8a17b80 1 -- 172.21.15.16:0/4089446739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 msgr2=0x55ec62a19690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.142+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/4089446739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f64b0003e20 tx=0x7f64b0005b40 comp rx=0 tx=0).stop 2024-04-05T23:34:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.142+0000 7f64d8a17b80 1 -- 172.21.15.16:0/4089446739 shutdown_connections 2024-04-05T23:34:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.142+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/4089446739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a16df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.142+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/4089446739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a15180 0x55ec62a04560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.142+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/4089446739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a0fca0 0x55ec62a19690 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.143+0000 7f64d8a17b80 1 -- 172.21.15.16:0/4089446739 >> 172.21.15.16:0/4089446739 conn(0x55ec62983220 msgr2=0x55ec62a17a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.143+0000 7f64d8a17b80 1 -- 172.21.15.16:0/4089446739 shutdown_connections 2024-04-05T23:34:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.143+0000 7f64d8a17b80 1 -- 172.21.15.16:0/4089446739 wait complete. 2024-04-05T23:34:16.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.143+0000 7f64d8a17b80 1 -- start start 2024-04-05T23:34:16.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.143+0000 7f64d8a17b80 1 Processor -- start 2024-04-05T23:34:16.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.144+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a1e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.144+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.144+0000 7f64d8a17b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 0x55ec62a1f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.144+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec62a7c200 con 0x55ec62a0f2e0 2024-04-05T23:34:16.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.144+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec62a7c340 con 0x55ec62a15180 2024-04-05T23:34:16.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.144+0000 7f64d8a17b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec62a7c500 con 0x55ec62a0fca0 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.144+0000 7f64d8a17b80 10 client.0 _ll_register_callbacks cb 0x7f64b000f770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f64bde00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f64bde00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39894/0 (socket says 172.21.15.16:39894) 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f64bde00700 1 -- 172.21.15.16:0/3242409130 learned_addr learned my addr 172.21.15.16:0/3242409130 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f64bde00700 1 -- 172.21.15.16:0/3242409130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 msgr2=0x55ec62a1f010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f64bde00700 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 0x55ec62a1f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f64bde00700 1 -- 172.21.15.16:0/3242409130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 msgr2=0x55ec62a1e590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f64bde00700 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a1e590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f64bde00700 1 -- 172.21.15.16:0/3242409130 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac019040 con 0x55ec62a0fca0 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f64bde00700 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1ead0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f64b0005e60 tx=0x7f64b00039e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=801b5fab89e0b906 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.145+0000 7f649fe00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.146+0000 7f649fe00700 1 -- 172.21.15.16:0/3242409130 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b001c030 con 0x55ec62a0fca0 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.146+0000 7f649fe00700 1 -- 172.21.15.16:0/3242409130 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f64b0000e40 con 0x55ec62a0fca0 2024-04-05T23:34:16.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.146+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3242409130 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64ac011540 con 0x55ec62a0fca0 2024-04-05T23:34:16.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.146+0000 7f649fe00700 1 -- 172.21.15.16:0/3242409130 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64b00052e0 con 0x55ec62a0fca0 2024-04-05T23:34:16.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.146+0000 7f64d8a17b80 10 client.25564 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:16.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.146+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3242409130 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ec6278db60 con 0x55ec62a0fca0 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.146+0000 7f649fe00700 1 -- 172.21.15.16:0/3242409130 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f64ac011540 con 0x55ec62a0fca0 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.147+0000 7f649fe00700 1 -- 172.21.15.16:0/3242409130 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f64b003d070 con 0x55ec62a0fca0 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.147+0000 7f64d8a17b80 10 client.25564 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.147+0000 7f64d8a17b80 10 client.25564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.147+0000 7f64d8a17b80 20 client.25564 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.147+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ec62a7d810 0x55ec62a7fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.147+0000 7f64d8a17b80 4 client.25564 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.147+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3242409130 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ec627aecf0 con 0x55ec62a7d810 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.147+0000 7f64bd400700 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ec62a7d810 0x55ec62a7fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f649fe00700 10 client.25564 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64bd400700 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ec62a7d810 0x55ec62a7fc00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11069a7eccf7f8d4 server_cookie=91e245f375363412 in_seq=0 out_seq=0 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f649fe00700 1 -- 172.21.15.16:0/3242409130 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ec627aecf0 con 0x55ec62a7d810 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f649fe00700 10 client.25564 handle_command_reply: tid=0 2024-04-05T23:34:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64d8a17b80 1 client.25564 shutdown 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3242409130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 msgr2=0x55ec62a1ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1ead0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f64b0005e60 tx=0x7f64b00039e0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3242409130 shutdown_connections 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ec62a7d810 0x55ec62a7fc00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec62a0f2e0 0x55ec62a1e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ec62a15180 0x55ec62a1f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f649fe00700 0 client.25564 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64d8a17b80 1 --2- 172.21.15.16:0/3242409130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec62a0fca0 0x55ec62a1ead0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.148+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3242409130 >> 172.21.15.16:0/3242409130 conn(0x55ec62983220 msgr2=0x55ec62a17950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.149+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3242409130 shutdown_connections 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.149+0000 7f64d8a17b80 1 -- 172.21.15.16:0/3242409130 wait complete. 2024-04-05T23:34:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.149+0000 7f64d8a17b80 20 client.25564 trim_cache size 0 max 16384 2024-04-05T23:34:16.143 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:34:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-04-05T23:34:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-04-05T23:34:16.143 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:34:16.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 -- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f64a40a13b0 msgr2=0x7f64a40a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f64a40a13b0 0x7f64a40a3870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f64b80ca040 tx=0x7f64b8009ef0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 -- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c019ded0 msgr2=0x7f64c019e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c019ded0 0x7f64c019e350 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f64b800efc0 tx=0x7f64b800c3a0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 -- 172.21.15.16:0/3951164488 shutdown_connections 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f64a40a13b0 0x7f64a40a3870 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c019ded0 0x7f64c019e350 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64c00fffa0 0x7f64c01995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 --2- 172.21.15.16:0/3951164488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c01050a0 0x7f64c0199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 -- 172.21.15.16:0/3951164488 >> 172.21.15.16:0/3951164488 conn(0x7f64c0076e80 msgr2=0x7f64c00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.151+0000 7f64c4e00700 1 -- 172.21.15.16:0/3951164488 shutdown_connections 2024-04-05T23:34:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.152+0000 7f64c4e00700 1 -- 172.21.15.16:0/3951164488 wait complete. 2024-04-05T23:34:16.171 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-05T23:34:16.172 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:34:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.391+0000 7f3cf9a00700 1 -- 172.21.15.16:0/1847771108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4101ef0 msgr2=0x7f3cf41022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.391+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/1847771108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4101ef0 0x7f3cf41022d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009990 tx=0x7f3ce4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.391+0000 7f3cf9a00700 1 -- 172.21.15.16:0/1847771108 shutdown_connections 2024-04-05T23:34:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.391+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/1847771108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cf410b310 0x7f3cf410d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.391+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/1847771108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4101ef0 0x7f3cf41022d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.391+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/1847771108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cf4102810 0x7f3cf410acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.391+0000 7f3cf9a00700 1 -- 172.21.15.16:0/1847771108 >> 172.21.15.16:0/1847771108 conn(0x7f3cf40fb6d0 msgr2=0x7f3cf40fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.391+0000 7f3cf9a00700 1 -- 172.21.15.16:0/1847771108 shutdown_connections 2024-04-05T23:34:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.391+0000 7f3cf9a00700 1 -- 172.21.15.16:0/1847771108 wait complete. 2024-04-05T23:34:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.393+0000 7f3cf9a00700 1 Processor -- start 2024-04-05T23:34:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.393+0000 7f3cf9a00700 1 -- start start 2024-04-05T23:34:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf9a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cf4101ef0 0x7f3cf4199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf9a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4102810 0x7f3cf41999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf9a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cf410b310 0x7f3cf419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf9a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf419e330 con 0x7f3cf4101ef0 2024-04-05T23:34:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf9a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf419e4a0 con 0x7f3cf4102810 2024-04-05T23:34:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf9a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf419e660 con 0x7f3cf410b310 2024-04-05T23:34:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cf4101ef0 0x7f3cf4199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4102810 0x7f3cf41999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4102810 0x7f3cf41999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42594/0 (socket says 172.21.15.16:42594) 2024-04-05T23:34:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cf4101ef0 0x7f3cf4199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42660/0 (socket says 172.21.15.16:42660) 2024-04-05T23:34:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf3400700 1 -- 172.21.15.16:0/2789073351 learned_addr learned my addr 172.21.15.16:0/2789073351 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.394+0000 7f3cf9000700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cf410b310 0x7f3cf419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.395+0000 7f3cf3400700 1 -- 172.21.15.16:0/2789073351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cf410b310 msgr2=0x7f3cf419dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.395+0000 7f3cf3400700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cf410b310 0x7f3cf419dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.395+0000 7f3cf3400700 1 -- 172.21.15.16:0/2789073351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cf4101ef0 msgr2=0x7f3cf4199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.395+0000 7f3cf3400700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cf4101ef0 0x7f3cf4199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.395+0000 7f3cf3400700 1 -- 172.21.15.16:0/2789073351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce40095f0 con 0x7f3cf4102810 2024-04-05T23:34:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.395+0000 7f3cf9000700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cf410b310 0x7f3cf419dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.395+0000 7f3cf3400700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4102810 0x7f3cf41999a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3ce000e9a0 tx=0x7f3ce000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.396+0000 7f3cf0c00700 1 -- 172.21.15.16:0/2789073351 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0009790 con 0x7f3cf4102810 2024-04-05T23:34:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.396+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf419e940 con 0x7f3cf4102810 2024-04-05T23:34:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.396+0000 7f3cf0c00700 1 -- 172.21.15.16:0/2789073351 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ce0010430 con 0x7f3cf4102810 2024-04-05T23:34:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.396+0000 7f3cf0c00700 1 -- 172.21.15.16:0/2789073351 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0018680 con 0x7f3cf4102810 2024-04-05T23:34:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.396+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf41a6180 con 0x7f3cf4102810 2024-04-05T23:34:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.397+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf4108840 con 0x7f3cf4102810 2024-04-05T23:34:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.399+0000 7f3cf0c00700 1 -- 172.21.15.16:0/2789073351 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f3ce00105a0 con 0x7f3cf4102810 2024-04-05T23:34:16.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.401+0000 7f3cf0c00700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f3cd80a1430 0x7f3cd80a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:16.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.401+0000 7f3cf0c00700 1 -- 172.21.15.16:0/2789073351 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f3ce0014070 con 0x7f3cf4102810 2024-04-05T23:34:16.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.402+0000 7f3cf3e00700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f3cd80a1430 0x7f3cd80a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.402+0000 7f3cf3e00700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f3cd80a1430 0x7f3cd80a38f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400b2e0 tx=0x7f3ce4005fd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:16.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.405+0000 7f3cf0c00700 1 -- 172.21.15.16:0/2789073351 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ce0052ab0 con 0x7f3cf4102810 2024-04-05T23:34:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.849+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cf419a710 con 0x7f3cf4102810 2024-04-05T23:34:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.850+0000 7f3cf0c00700 1 -- 172.21.15.16:0/2789073351 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10902 (secure 0 0 0) 0x7f3ce00962e0 con 0x7f3cf4102810 2024-04-05T23:34:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:34:16.842 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:34:14.702785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_25534":{"gid":25534,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/1942303111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1942303111},{"type":"v1","addr":"172.21.15.130:6835","nonce":1942303111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:34:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f3cd80a1430 msgr2=0x7f3cd80a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f3cd80a1430 0x7f3cd80a38f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400b2e0 tx=0x7f3ce4005fd0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4102810 msgr2=0x7f3cf41999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4102810 0x7f3cf41999a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3ce000e9a0 tx=0x7f3ce000ecb0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 shutdown_connections 2024-04-05T23:34:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f3cd80a1430 0x7f3cd80a38f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cf4101ef0 0x7f3cf4199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cf410b310 0x7f3cf419dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 --2- 172.21.15.16:0/2789073351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4102810 0x7f3cf41999a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.852+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 >> 172.21.15.16:0/2789073351 conn(0x7f3cf40fb6d0 msgr2=0x7f3cf4100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.853+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 shutdown_connections 2024-04-05T23:34:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:16.853+0000 7f3cf9a00700 1 -- 172.21.15.16:0/2789073351 wait complete. 2024-04-05T23:34:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2024-04-05T23:34:16.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-05T23:34:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.121+0000 7febb7400700 1 -- 172.21.15.16:0/924884890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 msgr2=0x7febb010dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.121+0000 7febb7400700 1 --2- 172.21.15.16:0/924884890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 0x7febb010dd50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7feba8009fc0 tx=0x7feba800b330 comp rx=0 tx=0).stop 2024-04-05T23:34:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.121+0000 7febb7400700 1 -- 172.21.15.16:0/924884890 shutdown_connections 2024-04-05T23:34:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.121+0000 7febb7400700 1 --2- 172.21.15.16:0/924884890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 0x7febb010dd50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.121+0000 7febb7400700 1 --2- 172.21.15.16:0/924884890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb0068220 0x7febb0068600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.121+0000 7febb7400700 1 --2- 172.21.15.16:0/924884890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febb0068bd0 0x7febb0105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.121+0000 7febb7400700 1 -- 172.21.15.16:0/924884890 >> 172.21.15.16:0/924884890 conn(0x7febb0076e80 msgr2=0x7febb0077290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.121+0000 7febb7400700 1 -- 172.21.15.16:0/924884890 shutdown_connections 2024-04-05T23:34:17.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.121+0000 7febb7400700 1 -- 172.21.15.16:0/924884890 wait complete. 2024-04-05T23:34:17.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.123+0000 7febb7400700 1 Processor -- start 2024-04-05T23:34:17.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.123+0000 7febb7400700 1 -- start start 2024-04-05T23:34:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febb7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febb0068220 0x7febb0199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febb7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 0x7febb0199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febb7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb019df20 0x7febb019e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febb7400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb019e9a0 con 0x7febb01061d0 2024-04-05T23:34:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febb7400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb019eb10 con 0x7febb019df20 2024-04-05T23:34:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febb7400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb019ecd0 con 0x7febb0068220 2024-04-05T23:34:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febafe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 0x7febb0199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febafe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 0x7febb0199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42688/0 (socket says 172.21.15.16:42688) 2024-04-05T23:34:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febafe00700 1 -- 172.21.15.16:0/1601410008 learned_addr learned my addr 172.21.15.16:0/1601410008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febb4a00700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febb0068220 0x7febb0199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.124+0000 7febb5400700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb019df20 0x7febb019e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.125+0000 7febafe00700 1 -- 172.21.15.16:0/1601410008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febb0068220 msgr2=0x7febb0199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.125+0000 7febafe00700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febb0068220 0x7febb0199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.125+0000 7febafe00700 1 -- 172.21.15.16:0/1601410008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb019df20 msgr2=0x7febb019e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.125+0000 7febafe00700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb019df20 0x7febb019e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.125+0000 7febafe00700 1 -- 172.21.15.16:0/1601410008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8009c40 con 0x7febb01061d0 2024-04-05T23:34:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.125+0000 7febafe00700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 0x7febb0199bd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00e9f0 tx=0x7feb9c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.126+0000 7febad600700 1 -- 172.21.15.16:0/1601410008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c009690 con 0x7febb01061d0 2024-04-05T23:34:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.126+0000 7febad600700 1 -- 172.21.15.16:0/1601410008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb9c00ce80 con 0x7febb01061d0 2024-04-05T23:34:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.126+0000 7febad600700 1 -- 172.21.15.16:0/1601410008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c018420 con 0x7febb01061d0 2024-04-05T23:34:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.127+0000 7febb7400700 1 -- 172.21.15.16:0/1601410008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb01a6230 con 0x7febb01061d0 2024-04-05T23:34:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.127+0000 7febb7400700 1 -- 172.21.15.16:0/1601410008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb01a66c0 con 0x7febb01061d0 2024-04-05T23:34:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.131+0000 7febad600700 1 -- 172.21.15.16:0/1601410008 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7feb9c00c9c0 con 0x7febb01061d0 2024-04-05T23:34:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.132+0000 7febad600700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7feb980a1430 0x7feb980a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.132+0000 7febad600700 1 -- 172.21.15.16:0/1601410008 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7feb9c014070 con 0x7febb01061d0 2024-04-05T23:34:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.133+0000 7febb4a00700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7feb980a1430 0x7feb980a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.133+0000 7febb4a00700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7feb980a1430 0x7feb980a38f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feba4009c60 tx=0x7feba4009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.134+0000 7febcb00fb80 1 Processor -- start 2024-04-05T23:34:17.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.134+0000 7febcb00fb80 1 -- start start 2024-04-05T23:34:17.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.134+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b3f180 0x5653e3b3f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.134+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 0x5653e3b39760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.134+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b39ca0 0x5653e3b45f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.134+0000 7febcb00fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3a9fbe0 con 0x5653e3b39ca0 2024-04-05T23:34:17.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.134+0000 7febcb00fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3a90990 con 0x5653e3b392e0 2024-04-05T23:34:17.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.134+0000 7febcb00fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3a8e350 con 0x5653e3b3f180 2024-04-05T23:34:17.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febafe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 0x5653e3b39760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febafe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 0x5653e3b39760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42626/0 (socket says 172.21.15.16:42626) 2024-04-05T23:34:17.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febafe00700 1 -- 172.21.15.16:0/791348309 learned_addr learned my addr 172.21.15.16:0/791348309 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febb4a00700 1 --2- 172.21.15.16:0/791348309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b3f180 0x5653e3b3f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febb5400700 1 --2- 172.21.15.16:0/791348309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b39ca0 0x5653e3b45f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febafe00700 1 -- 172.21.15.16:0/791348309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b3f180 msgr2=0x5653e3b3f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febafe00700 1 --2- 172.21.15.16:0/791348309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b3f180 0x5653e3b3f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febafe00700 1 -- 172.21.15.16:0/791348309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b39ca0 msgr2=0x5653e3b45f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febafe00700 1 --2- 172.21.15.16:0/791348309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b39ca0 0x5653e3b45f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febafe00700 1 -- 172.21.15.16:0/791348309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653e390b3b0 con 0x5653e3b392e0 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.135+0000 7febafe00700 1 --2- 172.21.15.16:0/791348309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 0x5653e3b39760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7feb9c099f80 tx=0x7feb9c010430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bba8bee21877e2a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.136+0000 7feba1c00700 1 -- 172.21.15.16:0/791348309 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c099ff0 con 0x5653e3b392e0 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.136+0000 7feba1c00700 1 -- 172.21.15.16:0/791348309 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb9c09de20 con 0x5653e3b392e0 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.136+0000 7feba1c00700 1 -- 172.21.15.16:0/791348309 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0a1fe0 con 0x5653e3b392e0 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.136+0000 7febcb00fb80 1 -- 172.21.15.16:0/791348309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 msgr2=0x5653e3b39760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.136+0000 7febcb00fb80 1 --2- 172.21.15.16:0/791348309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 0x5653e3b39760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7feb9c099f80 tx=0x7feb9c010430 comp rx=0 tx=0).stop 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.136+0000 7febcb00fb80 1 -- 172.21.15.16:0/791348309 shutdown_connections 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.136+0000 7febcb00fb80 1 --2- 172.21.15.16:0/791348309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b39ca0 0x5653e3b45f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.136+0000 7febcb00fb80 1 --2- 172.21.15.16:0/791348309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b3f180 0x5653e3b3f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.136+0000 7febcb00fb80 1 --2- 172.21.15.16:0/791348309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 0x5653e3b39760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.137+0000 7febcb00fb80 1 -- 172.21.15.16:0/791348309 >> 172.21.15.16:0/791348309 conn(0x5653e3aad220 msgr2=0x5653e3b34570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.137+0000 7febcb00fb80 1 -- 172.21.15.16:0/791348309 shutdown_connections 2024-04-05T23:34:17.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.137+0000 7febcb00fb80 1 -- 172.21.15.16:0/791348309 wait complete. 2024-04-05T23:34:17.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.137+0000 7febcb00fb80 1 -- start start 2024-04-05T23:34:17.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.137+0000 7febcb00fb80 1 Processor -- start 2024-04-05T23:34:17.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.138+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b45ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.138+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.138+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b3f180 0x5653e3b438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.138+0000 7febcb00fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3a89c40 con 0x5653e3b392e0 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.138+0000 7febcb00fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3a25d80 con 0x5653e3b3f180 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.138+0000 7febcb00fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3a25a20 con 0x5653e3b39ca0 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.138+0000 7febcb00fb80 10 client.0 _ll_register_callbacks cb 0x5653e38913a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.138+0000 7febb4a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b45ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7febb4a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b45ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42702/0 (socket says 172.21.15.16:42702) 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7febb4a00700 1 -- 172.21.15.16:0/342685275 learned_addr learned my addr 172.21.15.16:0/342685275 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7febafe00700 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7febafe00700 1 -- 172.21.15.16:0/342685275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b3f180 msgr2=0x5653e3b438b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7febafe00700 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b3f180 0x5653e3b438b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7febafe00700 1 -- 172.21.15.16:0/342685275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 msgr2=0x5653e3b45ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7febafe00700 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b45ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7febafe00700 1 -- 172.21.15.16:0/342685275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb01a66c0 con 0x5653e3b39ca0 2024-04-05T23:34:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7febafe00700 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b431c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c5b0 tx=0x7feb9c005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1e48eddc1305e5f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.139+0000 7feb96000700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.140+0000 7feb96000700 1 -- 172.21.15.16:0/342685275 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c07b070 con 0x5653e3b39ca0 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.140+0000 7feb96000700 1 -- 172.21.15.16:0/342685275 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7feb9c098130 con 0x5653e3b39ca0 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.140+0000 7febcb00fb80 1 -- 172.21.15.16:0/342685275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5653e3b2c490 con 0x5653e3b39ca0 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.140+0000 7feb96000700 1 -- 172.21.15.16:0/342685275 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb9c0a1fe0 con 0x5653e3b39ca0 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.140+0000 7febcb00fb80 10 client.25579 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.140+0000 7febcb00fb80 1 -- 172.21.15.16:0/342685275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5653e38b7b60 con 0x5653e3b39ca0 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.140+0000 7feb96000700 1 -- 172.21.15.16:0/342685275 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5653e3b2c490 con 0x5653e3b39ca0 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.140+0000 7feb96000700 1 -- 172.21.15.16:0/342685275 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7feb9c07e070 con 0x5653e3b39ca0 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.141+0000 7febcb00fb80 10 client.25579 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.141+0000 7febcb00fb80 10 client.25579 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.141+0000 7febcb00fb80 20 client.25579 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.141+0000 7febcb00fb80 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5653e3b5b7e0 0x5653e3b5dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.141+0000 7febcb00fb80 4 client.25579 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.141+0000 7febcb00fb80 1 -- 172.21.15.16:0/342685275 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653e3a82700 con 0x5653e3b5b7e0 2024-04-05T23:34:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.141+0000 7febb4a00700 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5653e3b5b7e0 0x5653e3b5dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.142+0000 7febb4a00700 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5653e3b5b7e0 0x5653e3b5dbd0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1d6b96b38f380bdd server_cookie=1f143fde5889b359 in_seq=0 out_seq=0 2024-04-05T23:34:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.142+0000 7feb96000700 10 client.25579 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:34:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.142+0000 7feb96000700 1 -- 172.21.15.16:0/342685275 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5653e3a82700 con 0x5653e3b5b7e0 2024-04-05T23:34:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.142+0000 7feb96000700 10 client.25579 handle_command_reply: tid=0 2024-04-05T23:34:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 client.25579 shutdown 2024-04-05T23:34:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 -- 172.21.15.16:0/342685275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 msgr2=0x5653e3b431c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b431c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c5b0 tx=0x7feb9c005880 comp rx=0 tx=0).stop 2024-04-05T23:34:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 -- 172.21.15.16:0/342685275 shutdown_connections 2024-04-05T23:34:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5653e3b5b7e0 0x5653e3b5dbd0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b45ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b3f180 0x5653e3b438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 --2- 172.21.15.16:0/342685275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b431c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 -- 172.21.15.16:0/342685275 >> 172.21.15.16:0/342685275 conn(0x5653e3aad220 msgr2=0x5653e3b2e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7feb96000700 0 client.25579 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:34:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 -- 172.21.15.16:0/342685275 shutdown_connections 2024-04-05T23:34:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.143+0000 7febcb00fb80 1 -- 172.21.15.16:0/342685275 wait complete. 2024-04-05T23:34:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.144+0000 7febcb00fb80 20 client.25579 trim_cache size 0 max 16384 2024-04-05T23:34:17.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.158+0000 7febcb00fb80 1 Processor -- start 2024-04-05T23:34:17.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.158+0000 7febcb00fb80 1 -- start start 2024-04-05T23:34:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.158+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 0x5653e3b43160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.158+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b52d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b3f180 0x5653e3b2e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febcb00fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e390d6a0 con 0x5653e3b3f180 2024-04-05T23:34:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febcb00fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3b532a0 con 0x5653e3b392e0 2024-04-05T23:34:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febcb00fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3b41710 con 0x5653e3b39ca0 2024-04-05T23:34:17.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febafe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b52d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febafe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b52d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39942/0 (socket says 172.21.15.16:39942) 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febafe00700 1 -- 172.21.15.16:0/202371640 learned_addr learned my addr 172.21.15.16:0/202371640 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febb5400700 1 --2- 172.21.15.16:0/202371640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b3f180 0x5653e3b2e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febafe00700 1 -- 172.21.15.16:0/202371640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 msgr2=0x5653e3b43160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febafe00700 1 --2- 172.21.15.16:0/202371640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 0x5653e3b43160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.159+0000 7febafe00700 1 -- 172.21.15.16:0/202371640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b3f180 msgr2=0x5653e3b2e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.160+0000 7febafe00700 1 --2- 172.21.15.16:0/202371640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b3f180 0x5653e3b2e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.160+0000 7febafe00700 1 -- 172.21.15.16:0/202371640 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653e3a4fea0 con 0x5653e3b39ca0 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.160+0000 7febb5400700 1 --2- 172.21.15.16:0/202371640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b3f180 0x5653e3b2e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.160+0000 7febafe00700 1 --2- 172.21.15.16:0/202371640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b52d60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feb9c010a20 tx=0x7feb9c010a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=163484c4cb1f8860 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.160+0000 7feba2600700 1 -- 172.21.15.16:0/202371640 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c07b070 con 0x5653e3b39ca0 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.160+0000 7feba2600700 1 -- 172.21.15.16:0/202371640 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb9c0b29d0 con 0x5653e3b39ca0 2024-04-05T23:34:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.160+0000 7feba2600700 1 -- 172.21.15.16:0/202371640 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0ca9e0 con 0x5653e3b39ca0 2024-04-05T23:34:17.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.161+0000 7febcb00fb80 1 -- 172.21.15.16:0/202371640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 msgr2=0x5653e3b52d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.161+0000 7febcb00fb80 1 --2- 172.21.15.16:0/202371640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b52d60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feb9c010a20 tx=0x7feb9c010a70 comp rx=0 tx=0).stop 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.161+0000 7febcb00fb80 1 -- 172.21.15.16:0/202371640 shutdown_connections 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.161+0000 7febcb00fb80 1 --2- 172.21.15.16:0/202371640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b3f180 0x5653e3b2e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.161+0000 7febcb00fb80 1 --2- 172.21.15.16:0/202371640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b392e0 0x5653e3b43160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.161+0000 7febcb00fb80 1 --2- 172.21.15.16:0/202371640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b52d60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.161+0000 7febcb00fb80 1 -- 172.21.15.16:0/202371640 >> 172.21.15.16:0/202371640 conn(0x5653e3aad220 msgr2=0x5653e3b42a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.161+0000 7febcb00fb80 1 -- 172.21.15.16:0/202371640 shutdown_connections 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.161+0000 7febcb00fb80 1 -- 172.21.15.16:0/202371640 wait complete. 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.162+0000 7febcb00fb80 1 -- start start 2024-04-05T23:34:17.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.162+0000 7febcb00fb80 1 Processor -- start 2024-04-05T23:34:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b50f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b51490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febcb00fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b3f180 0x5653e3ba8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febcb00fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3ba8bc0 con 0x5653e3b392e0 2024-04-05T23:34:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febcb00fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3ba8d30 con 0x5653e3b3f180 2024-04-05T23:34:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febcb00fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e3ba8ef0 con 0x5653e3b39ca0 2024-04-05T23:34:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febcb00fb80 10 client.0 _ll_register_callbacks cb 0x7feb9c099950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febb4a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b50f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febb4a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b50f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42724/0 (socket says 172.21.15.16:42724) 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febb4a00700 1 -- 172.21.15.16:0/1432076885 learned_addr learned my addr 172.21.15.16:0/1432076885 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febafe00700 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b51490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.163+0000 7febb5400700 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b3f180 0x5653e3ba8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.164+0000 7febb4a00700 1 -- 172.21.15.16:0/1432076885 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 msgr2=0x5653e3b51490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.164+0000 7febb4a00700 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b51490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.164+0000 7febb4a00700 1 -- 172.21.15.16:0/1432076885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b3f180 msgr2=0x5653e3ba8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.164+0000 7febb4a00700 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b3f180 0x5653e3ba8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.164+0000 7febb4a00700 1 -- 172.21.15.16:0/1432076885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb010e590 con 0x5653e3b392e0 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.164+0000 7febb4a00700 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b50f30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7feba400a010 tx=0x7feba40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7b4d98725d4fc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.164+0000 7feb96000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.164+0000 7feb96000700 1 -- 172.21.15.16:0/1432076885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4023040 con 0x5653e3b392e0 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.164+0000 7feb96000700 1 -- 172.21.15.16:0/1432076885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7feba401d610 con 0x5653e3b392e0 2024-04-05T23:34:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.165+0000 7febcb00fb80 1 -- 172.21.15.16:0/1432076885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb9c09de20 con 0x5653e3b392e0 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.165+0000 7feb96000700 1 -- 172.21.15.16:0/1432076885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7feba401d7d0 con 0x5653e3b392e0 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.165+0000 7febcb00fb80 10 client.15537 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.165+0000 7febcb00fb80 1 -- 172.21.15.16:0/1432076885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5653e38b7b60 con 0x5653e3b392e0 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.165+0000 7feb96000700 1 -- 172.21.15.16:0/1432076885 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb9c09de20 con 0x5653e3b392e0 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.165+0000 7feb96000700 1 -- 172.21.15.16:0/1432076885 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7feba4026070 con 0x5653e3b392e0 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.166+0000 7febcb00fb80 10 client.15537 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.166+0000 7febcb00fb80 10 client.15537 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.166+0000 7febcb00fb80 20 client.15537 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.166+0000 7febcb00fb80 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5653e3bb07d0 0x5653e3bb2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.166+0000 7febcb00fb80 4 client.15537 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.166+0000 7febcb00fb80 1 -- 172.21.15.16:0/1432076885 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5653e38d8cf0 con 0x5653e3bb07d0 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.166+0000 7febafe00700 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5653e3bb07d0 0x5653e3bb2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febafe00700 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5653e3bb07d0 0x5653e3bb2bf0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a5aefe0a709785ad server_cookie=d2f8c0d2c6971ee3 in_seq=0 out_seq=0 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7feb96000700 10 client.15537 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7feb96000700 1 -- 172.21.15.16:0/1432076885 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5653e38d8cf0 con 0x5653e3bb07d0 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7feb96000700 10 client.15537 handle_command_reply: tid=0 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febcb00fb80 1 client.15537 shutdown 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febcb00fb80 1 -- 172.21.15.16:0/1432076885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 msgr2=0x5653e3b50f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febcb00fb80 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b50f30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7feba400a010 tx=0x7feba40047c0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febcb00fb80 1 -- 172.21.15.16:0/1432076885 shutdown_connections 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febcb00fb80 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x5653e3bb07d0 0x5653e3bb2bf0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7feb96000700 0 client.15537 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febcb00fb80 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5653e3b392e0 0x5653e3b50f30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febcb00fb80 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653e3b3f180 0x5653e3ba8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febcb00fb80 1 --2- 172.21.15.16:0/1432076885 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653e3b39ca0 0x5653e3b51490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.167+0000 7febcb00fb80 1 -- 172.21.15.16:0/1432076885 >> 172.21.15.16:0/1432076885 conn(0x5653e3aad220 msgr2=0x5653e3b57d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.168+0000 7febcb00fb80 1 -- 172.21.15.16:0/1432076885 shutdown_connections 2024-04-05T23:34:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.168+0000 7febcb00fb80 1 -- 172.21.15.16:0/1432076885 wait complete. 2024-04-05T23:34:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.168+0000 7febcb00fb80 20 client.15537 trim_cache size 0 max 16384 2024-04-05T23:34:17.162 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 -- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7feb980a1430 msgr2=0x7feb980a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7feb980a1430 0x7feb980a38f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feba4009c60 tx=0x7feba4009260 comp rx=0 tx=0).stop 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 -- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 msgr2=0x7febb0199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 0x7febb0199bd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00e9f0 tx=0x7feb9c00ed00 comp rx=0 tx=0).stop 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 -- 172.21.15.16:0/1601410008 shutdown_connections 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7feb980a1430 0x7feb980a38f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb019df20 0x7febb019e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febb0068220 0x7febb0199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 --2- 172.21.15.16:0/1601410008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb01061d0 0x7febb0199bd0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 -- 172.21.15.16:0/1601410008 >> 172.21.15.16:0/1601410008 conn(0x7febb0076e80 msgr2=0x7febb010c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 -- 172.21.15.16:0/1601410008 shutdown_connections 2024-04-05T23:34:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.169+0000 7febb7400700 1 -- 172.21.15.16:0/1601410008 wait complete. 2024-04-05T23:34:17.180 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-05T23:34:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.424+0000 7f65a2600700 1 -- 172.21.15.16:0/3091229557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c101eb0 msgr2=0x7f659c102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.424+0000 7f65a2600700 1 --2- 172.21.15.16:0/3091229557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c101eb0 0x7f659c102290 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6590009990 tx=0x7f6590009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.424+0000 7f65a2600700 1 -- 172.21.15.16:0/3091229557 shutdown_connections 2024-04-05T23:34:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.424+0000 7f65a2600700 1 --2- 172.21.15.16:0/3091229557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659c10b2d0 0x7f659c10d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.424+0000 7f65a2600700 1 --2- 172.21.15.16:0/3091229557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c101eb0 0x7f659c102290 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.425+0000 7f65a2600700 1 --2- 172.21.15.16:0/3091229557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f659c1027d0 0x7f659c10acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.425+0000 7f65a2600700 1 -- 172.21.15.16:0/3091229557 >> 172.21.15.16:0/3091229557 conn(0x7f659c0fb6b0 msgr2=0x7f659c0fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.425+0000 7f65a2600700 1 -- 172.21.15.16:0/3091229557 shutdown_connections 2024-04-05T23:34:17.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.425+0000 7f65a2600700 1 -- 172.21.15.16:0/3091229557 wait complete. 2024-04-05T23:34:17.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.426+0000 7f65a2600700 1 Processor -- start 2024-04-05T23:34:17.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.427+0000 7f65a2600700 1 -- start start 2024-04-05T23:34:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.427+0000 7f65a2600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f659c101eb0 0x7f659c199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.427+0000 7f65a2600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c1027d0 0x7f659c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.427+0000 7f65a2600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659c10b2d0 0x7f659c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.427+0000 7f65a2600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c19e350 con 0x7f659c101eb0 2024-04-05T23:34:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.427+0000 7f65a2600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c19e4c0 con 0x7f659c10b2d0 2024-04-05T23:34:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.427+0000 7f65a2600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c19e680 con 0x7f659c1027d0 2024-04-05T23:34:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f659be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c1027d0 0x7f659c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f659be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c1027d0 0x7f659c1999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39964/0 (socket says 172.21.15.16:39964) 2024-04-05T23:34:17.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f659be00700 1 -- 172.21.15.16:0/180681310 learned_addr learned my addr 172.21.15.16:0/180681310 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f65a1c00700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659c10b2d0 0x7f659c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f65a1200700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f659c101eb0 0x7f659c199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f659be00700 1 -- 172.21.15.16:0/180681310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659c10b2d0 msgr2=0x7f659c19dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f659be00700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659c10b2d0 0x7f659c19dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f659be00700 1 -- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f659c101eb0 msgr2=0x7f659c199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f659be00700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f659c101eb0 0x7f659c199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f659be00700 1 -- 172.21.15.16:0/180681310 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65900095f0 con 0x7f659c1027d0 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f65a1200700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f659c101eb0 0x7f659c199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.428+0000 7f65a1c00700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659c10b2d0 0x7f659c19dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:17.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.429+0000 7f659be00700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c1027d0 0x7f659c1999c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f658800d750 tx=0x7f658800db10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.429+0000 7f6599600700 1 -- 172.21.15.16:0/180681310 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588004220 con 0x7f659c1027d0 2024-04-05T23:34:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.429+0000 7f65a2600700 1 -- 172.21.15.16:0/180681310 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c19e960 con 0x7f659c1027d0 2024-04-05T23:34:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.429+0000 7f6599600700 1 -- 172.21.15.16:0/180681310 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6588004cf0 con 0x7f659c1027d0 2024-04-05T23:34:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.429+0000 7f6599600700 1 -- 172.21.15.16:0/180681310 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588005700 con 0x7f659c1027d0 2024-04-05T23:34:17.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.429+0000 7f65a2600700 1 -- 172.21.15.16:0/180681310 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c102e30 con 0x7f659c1027d0 2024-04-05T23:34:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.434+0000 7f65b76eeb80 1 Processor -- start 2024-04-05T23:34:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.434+0000 7f6599600700 1 -- 172.21.15.16:0/180681310 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f6588004750 con 0x7f659c1027d0 2024-04-05T23:34:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.434+0000 7f65b76eeb80 1 -- start start 2024-04-05T23:34:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.435+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df61826b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.435+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 0x558df618d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.435+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 0x558df618fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.435+0000 7f65a1c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df61826b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.435+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df60e9be0 con 0x558df618dad0 2024-04-05T23:34:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.435+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df60da990 con 0x558df61822d0 2024-04-05T23:34:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.435+0000 7f65a1c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df61826b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42654/0 (socket says 172.21.15.16:42654) 2024-04-05T23:34:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.435+0000 7f65a1c00700 1 -- 172.21.15.16:0/2162645275 learned_addr learned my addr 172.21.15.16:0/2162645275 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.435+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2162645275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df60d8350 con 0x558df6182bf0 2024-04-05T23:34:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f65a1c00700 1 -- 172.21.15.16:0/2162645275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 msgr2=0x558df618d590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:34:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f65a1c00700 1 --2- 172.21.15.16:0/2162645275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 0x558df618d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f65a1c00700 1 -- 172.21.15.16:0/2162645275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 msgr2=0x558df618fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f65a1c00700 1 --2- 172.21.15.16:0/2162645275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 0x558df618fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f65a1c00700 1 -- 172.21.15.16:0/2162645275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558df5f553b0 con 0x558df61822d0 2024-04-05T23:34:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f65a1c00700 1 --2- 172.21.15.16:0/2162645275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df61826b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f65940085a0 tx=0x7f65940088b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ece4c8340c984029 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f6599600700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65840a1430 0x7f65840a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f658e000700 1 -- 172.21.15.16:0/2162645275 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594003c00 con 0x558df61822d0 2024-04-05T23:34:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f658e000700 1 -- 172.21.15.16:0/2162645275 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6594010040 con 0x558df61822d0 2024-04-05T23:34:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f658e000700 1 -- 172.21.15.16:0/2162645275 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594012880 con 0x558df61822d0 2024-04-05T23:34:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.436+0000 7f6599600700 1 -- 172.21.15.16:0/180681310 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f65880ccde0 con 0x7f659c1027d0 2024-04-05T23:34:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.437+0000 7f65a1200700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65840a1430 0x7f65840a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.437+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2162645275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 msgr2=0x558df61826b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.437+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2162645275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df61826b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f65940085a0 tx=0x7f65940088b0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.437+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2162645275 shutdown_connections 2024-04-05T23:34:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.437+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2162645275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 0x558df618fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.437+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2162645275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df61826b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.437+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2162645275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 0x558df618d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.437+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2162645275 >> 172.21.15.16:0/2162645275 conn(0x558df60f7220 msgr2=0x558df617e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.438+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2162645275 shutdown_connections 2024-04-05T23:34:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.438+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2162645275 wait complete. 2024-04-05T23:34:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.438+0000 7f65a1200700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65840a1430 0x7f65840a38f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f659000b5c0 tx=0x7f6590005fd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.438+0000 7f65b76eeb80 1 -- start start 2024-04-05T23:34:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.439+0000 7f65b76eeb80 1 Processor -- start 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.439+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 0x558df618d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.439+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df618a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.439+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df618af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.439+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df60d3c40 con 0x558df6182bf0 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.439+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df606fd80 con 0x558df618dad0 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.439+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df606fa20 con 0x558df61822d0 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f65b76eeb80 10 client.0 _ll_register_callbacks cb 0x558df5edb3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f659be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df618a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f659be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df618a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42768/0 (socket says 172.21.15.16:42768) 2024-04-05T23:34:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f659be00700 1 -- 172.21.15.16:0/593369714 learned_addr learned my addr 172.21.15.16:0/593369714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f65a1200700 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 0x558df618d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f65a1c00700 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df618af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f659be00700 1 -- 172.21.15.16:0/593369714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 msgr2=0x558df618d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f659be00700 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 0x558df618d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f659be00700 1 -- 172.21.15.16:0/593369714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 msgr2=0x558df618af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f659be00700 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df618af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.440+0000 7f659be00700 1 -- 172.21.15.16:0/593369714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6594007dd0 con 0x558df6182bf0 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.441+0000 7f659be00700 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df618a800 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f658800df10 tx=0x7f6588099650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac81854024a50fe server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.441+0000 7f6582a00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.441+0000 7f6582a00700 1 -- 172.21.15.16:0/593369714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588099cd0 con 0x558df6182bf0 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.441+0000 7f6582a00700 1 -- 172.21.15.16:0/593369714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f658809daa0 con 0x558df6182bf0 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.441+0000 7f65b76eeb80 1 -- 172.21.15.16:0/593369714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558df6176490 con 0x558df6182bf0 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.441+0000 7f6582a00700 1 -- 172.21.15.16:0/593369714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65880a20f0 con 0x558df6182bf0 2024-04-05T23:34:17.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.441+0000 7f65b76eeb80 10 client.15543 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.441+0000 7f65b76eeb80 1 -- 172.21.15.16:0/593369714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x558df5f01b60 con 0x558df6182bf0 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.441+0000 7f6582a00700 1 -- 172.21.15.16:0/593369714 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65880b8030 con 0x558df6182bf0 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.442+0000 7f6582a00700 1 -- 172.21.15.16:0/593369714 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f65880a1c60 con 0x558df6182bf0 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.442+0000 7f65b76eeb80 10 client.15543 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.442+0000 7f65b76eeb80 10 client.15543 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.d 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.442+0000 7f65b76eeb80 20 client.15543 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.442+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x558df619d450 0x558df619f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.442+0000 7f65b76eeb80 4 client.15543 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.442+0000 7f65b76eeb80 1 -- 172.21.15.16:0/593369714 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558df60cc700 con 0x558df619d450 2024-04-05T23:34:17.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.442+0000 7f65a1200700 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x558df619d450 0x558df619f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.443+0000 7f65a1200700 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x558df619d450 0x558df619f870 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ca537a59b3c08a65 server_cookie=29e4a91d1ea367a7 in_seq=0 out_seq=0 2024-04-05T23:34:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.443+0000 7f6582a00700 10 client.15543 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:34:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.444+0000 7f6582a00700 1 -- 172.21.15.16:0/593369714 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558df60cc700 con 0x558df619d450 2024-04-05T23:34:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.444+0000 7f6582a00700 10 client.15543 handle_command_reply: tid=0 2024-04-05T23:34:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.444+0000 7f65b76eeb80 1 client.15543 shutdown 2024-04-05T23:34:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.444+0000 7f65b76eeb80 1 -- 172.21.15.16:0/593369714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 msgr2=0x558df618a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.444+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df618a800 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f658800df10 tx=0x7f6588099650 comp rx=0 tx=0).stop 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f65b76eeb80 1 -- 172.21.15.16:0/593369714 shutdown_connections 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x558df619d450 0x558df619f870 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df618af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 0x558df618d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/593369714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df618a800 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f65b76eeb80 1 -- 172.21.15.16:0/593369714 >> 172.21.15.16:0/593369714 conn(0x558df60f7220 msgr2=0x558df617e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f6582a00700 0 client.15543 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f65b76eeb80 1 -- 172.21.15.16:0/593369714 shutdown_connections 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f65b76eeb80 1 -- 172.21.15.16:0/593369714 wait complete. 2024-04-05T23:34:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.445+0000 7f65b76eeb80 20 client.15543 trim_cache size 0 max 16384 2024-04-05T23:34:17.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.471+0000 7f65b76eeb80 1 Processor -- start 2024-04-05T23:34:17.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.471+0000 7f65b76eeb80 1 -- start start 2024-04-05T23:34:17.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df618a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 0x558df6194660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 0x558df617a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df5f576a0 con 0x558df618dad0 2024-04-05T23:34:17.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df6183540 con 0x558df61822d0 2024-04-05T23:34:17.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df6194ba0 con 0x558df6182bf0 2024-04-05T23:34:17.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65a1200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df618a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65a1200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df618a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42664/0 (socket says 172.21.15.16:42664) 2024-04-05T23:34:17.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65a1200700 1 -- 172.21.15.16:0/3041983762 learned_addr learned my addr 172.21.15.16:0/3041983762 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.472+0000 7f65a1c00700 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 0x558df617a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f659be00700 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 0x558df6194660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f65a1200700 1 -- 172.21.15.16:0/3041983762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 msgr2=0x558df6194660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f65a1200700 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 0x558df6194660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f65a1200700 1 -- 172.21.15.16:0/3041983762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 msgr2=0x558df617a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f65a1200700 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 0x558df617a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f65a1200700 1 -- 172.21.15.16:0/3041983762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558df6099ea0 con 0x558df61822d0 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f659be00700 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 0x558df6194660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f65a1c00700 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 0x558df617a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f65a1200700 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df618a7c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6590009990 tx=0x7f6590020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87f02b50b1990dc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.473+0000 7f658ea00700 1 -- 172.21.15.16:0/3041983762 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659002b040 con 0x558df61822d0 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.474+0000 7f658ea00700 1 -- 172.21.15.16:0/3041983762 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6590030dc0 con 0x558df61822d0 2024-04-05T23:34:17.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.474+0000 7f658ea00700 1 -- 172.21.15.16:0/3041983762 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659001d070 con 0x558df61822d0 2024-04-05T23:34:17.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.474+0000 7f65b76eeb80 1 -- 172.21.15.16:0/3041983762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 msgr2=0x558df618a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.474+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df618a7c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6590009990 tx=0x7f6590020000 comp rx=0 tx=0).stop 2024-04-05T23:34:17.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.475+0000 7f65b76eeb80 1 -- 172.21.15.16:0/3041983762 shutdown_connections 2024-04-05T23:34:17.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.475+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df618dad0 0x558df617a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.475+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df61822d0 0x558df618a7c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.475+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/3041983762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df6182bf0 0x558df6194660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.475+0000 7f65b76eeb80 1 -- 172.21.15.16:0/3041983762 >> 172.21.15.16:0/3041983762 conn(0x558df60f7220 msgr2=0x558df618a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.475+0000 7f65b76eeb80 1 -- 172.21.15.16:0/3041983762 shutdown_connections 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.475+0000 7f65b76eeb80 1 -- 172.21.15.16:0/3041983762 wait complete. 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.475+0000 7f65b76eeb80 1 -- start start 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.475+0000 7f65b76eeb80 1 Processor -- start 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 0x558df6193e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df6194380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65b76eeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df6192640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df6192d50 con 0x558df6182bf0 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df6192ec0 con 0x558df618dad0 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65b76eeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df6193080 con 0x558df61822d0 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65b76eeb80 10 client.0 _ll_register_callbacks cb 0x7f6588099a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65a1c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df6192640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65a1c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df6192640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42680/0 (socket says 172.21.15.16:42680) 2024-04-05T23:34:17.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65a1c00700 1 -- 172.21.15.16:0/2858844719 learned_addr learned my addr 172.21.15.16:0/2858844719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.476+0000 7f65a1200700 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 0x558df6193e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f659be00700 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df6194380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f65a1c00700 1 -- 172.21.15.16:0/2858844719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 msgr2=0x558df6193e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f65a1c00700 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 0x558df6193e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f65a1c00700 1 -- 172.21.15.16:0/2858844719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 msgr2=0x558df6194380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f65a1c00700 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df6194380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f65a1c00700 1 -- 172.21.15.16:0/2858844719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c10e450 con 0x558df618dad0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f65a1c00700 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df6192640 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f65940085a0 tx=0x7f6594012cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81233ddc5cd9c599 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f6582a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f6582a00700 1 -- 172.21.15.16:0/2858844719 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594004ca0 con 0x558df618dad0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2858844719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6594010040 con 0x558df618dad0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.477+0000 7f6582a00700 1 -- 172.21.15.16:0/2858844719 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f6594004e00 con 0x558df618dad0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.478+0000 7f6582a00700 1 -- 172.21.15.16:0/2858844719 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6594005050 con 0x558df618dad0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.478+0000 7f65b76eeb80 10 client.25577 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.478+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2858844719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x558df5f59cd0 con 0x558df618dad0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.478+0000 7f6582a00700 1 -- 172.21.15.16:0/2858844719 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6594010040 con 0x558df618dad0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.478+0000 7f6582a00700 1 -- 172.21.15.16:0/2858844719 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f659401c030 con 0x558df618dad0 2024-04-05T23:34:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.479+0000 7f65b76eeb80 10 client.25577 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.479+0000 7f65b76eeb80 10 client.25577 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.d 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.479+0000 7f65b76eeb80 20 client.25577 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.479+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x558df61f2400 0x558df61f4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.479+0000 7f65b76eeb80 4 client.25577 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.479+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2858844719 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558df5f22cf0 con 0x558df61f2400 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.479+0000 7f65a1200700 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x558df61f2400 0x558df61f4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.479+0000 7f65a1200700 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x558df61f2400 0x558df61f4820 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8a4332b15a5fd52d server_cookie=288d1ebf6f878ec0 in_seq=0 out_seq=0 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.479+0000 7f6582a00700 10 client.25577 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f6582a00700 1 -- 172.21.15.16:0/2858844719 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558df5f22cf0 con 0x558df61f2400 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f6582a00700 10 client.25577 handle_command_reply: tid=0 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 client.25577 shutdown 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2858844719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 msgr2=0x558df6192640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df6192640 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f65940085a0 tx=0x7f6594012cc0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2858844719 shutdown_connections 2024-04-05T23:34:17.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x558df61f2400 0x558df61f4820 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558df618dad0 0x558df6192640 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df61822d0 0x558df6193e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 --2- 172.21.15.16:0/2858844719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558df6182bf0 0x558df6194380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2858844719 >> 172.21.15.16:0/2858844719 conn(0x558df60f7220 msgr2=0x558df617cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f6582a00700 0 client.25577 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:34:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2858844719 shutdown_connections 2024-04-05T23:34:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.480+0000 7f65b76eeb80 1 -- 172.21.15.16:0/2858844719 wait complete. 2024-04-05T23:34:17.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65b76eeb80 20 client.25577 trim_cache size 0 max 16384 2024-04-05T23:34:17.478 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 -- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65840a1430 msgr2=0x7f65840a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65840a1430 0x7f65840a38f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f659000b5c0 tx=0x7f6590005fd0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 -- 172.21.15.16:0/180681310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c1027d0 msgr2=0x7f659c1999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c1027d0 0x7f659c1999c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f658800d750 tx=0x7f658800db10 comp rx=0 tx=0).stop 2024-04-05T23:34:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 -- 172.21.15.16:0/180681310 shutdown_connections 2024-04-05T23:34:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65840a1430 0x7f65840a38f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f659c101eb0 0x7f659c199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659c10b2d0 0x7f659c19dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 --2- 172.21.15.16:0/180681310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f659c1027d0 0x7f659c1999c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.481+0000 7f65a2600700 1 -- 172.21.15.16:0/180681310 >> 172.21.15.16:0/180681310 conn(0x7f659c0fb6b0 msgr2=0x7f659c100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.482+0000 7f65a2600700 1 -- 172.21.15.16:0/180681310 shutdown_connections 2024-04-05T23:34:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.482+0000 7f65a2600700 1 -- 172.21.15.16:0/180681310 wait complete. 2024-04-05T23:34:17.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-05T23:34:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.711+0000 7fcdda000700 1 -- 172.21.15.16:0/2221745061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd4073310 msgr2=0x7fcdd40736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.711+0000 7fcdda000700 1 --2- 172.21.15.16:0/2221745061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd4073310 0x7fcdd40736f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80098f0 tx=0x7fcdc8009c00 comp rx=0 tx=0).stop 2024-04-05T23:34:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.711+0000 7fcdda000700 1 -- 172.21.15.16:0/2221745061 shutdown_connections 2024-04-05T23:34:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.711+0000 7fcdda000700 1 --2- 172.21.15.16:0/2221745061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd410d4c0 0x7fcdd410f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.711+0000 7fcdda000700 1 --2- 172.21.15.16:0/2221745061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd4073310 0x7fcdd40736f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.711+0000 7fcdda000700 1 --2- 172.21.15.16:0/2221745061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd4073c30 0x7fcdd410cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.711+0000 7fcdda000700 1 -- 172.21.15.16:0/2221745061 >> 172.21.15.16:0/2221745061 conn(0x7fcdd40fc770 msgr2=0x7fcdd40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.711+0000 7fcdda000700 1 -- 172.21.15.16:0/2221745061 shutdown_connections 2024-04-05T23:34:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.711+0000 7fcdda000700 1 -- 172.21.15.16:0/2221745061 wait complete. 2024-04-05T23:34:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.713+0000 7fcdda000700 1 Processor -- start 2024-04-05T23:34:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.713+0000 7fcdda000700 1 -- start start 2024-04-05T23:34:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.713+0000 7fcdda000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd4073310 0x7fcdd410cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.713+0000 7fcdda000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd4073c30 0x7fcdd4109eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.714+0000 7fcdda000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd410d4c0 0x7fcdd410a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.714+0000 7fcdda000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd410ab80 con 0x7fcdd4073310 2024-04-05T23:34:17.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.714+0000 7fcdda000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd410acf0 con 0x7fcdd410d4c0 2024-04-05T23:34:17.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.714+0000 7fcdda000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd41a9ce0 con 0x7fcdd4073c30 2024-04-05T23:34:17.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.714+0000 7fcdd8c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd4073310 0x7fcdd410cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.714+0000 7fcdd8c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd4073310 0x7fcdd410cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42812/0 (socket says 172.21.15.16:42812) 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.714+0000 7fcdd8c00700 1 -- 172.21.15.16:0/2878637865 learned_addr learned my addr 172.21.15.16:0/2878637865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.714+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd4073c30 0x7fcdd4109eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.714+0000 7fcdd9600700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd410d4c0 0x7fcdd410a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.715+0000 7fcdd8c00700 1 -- 172.21.15.16:0/2878637865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd4073c30 msgr2=0x7fcdd4109eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.715+0000 7fcdd8c00700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd4073c30 0x7fcdd4109eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.715+0000 7fcdd8c00700 1 -- 172.21.15.16:0/2878637865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd410d4c0 msgr2=0x7fcdd410a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.715+0000 7fcdd8c00700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd410d4c0 0x7fcdd410a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.715+0000 7fcdd8c00700 1 -- 172.21.15.16:0/2878637865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc8009520 con 0x7fcdd4073310 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.715+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd4073c30 0x7fcdd4109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.715+0000 7fcdd9600700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd410d4c0 0x7fcdd410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.715+0000 7fcdd8c00700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd4073310 0x7fcdd410cd90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009230 tx=0x7fcdc800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.715+0000 7fcdd1600700 1 -- 172.21.15.16:0/2878637865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc801c070 con 0x7fcdd4073310 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.716+0000 7fcdd1600700 1 -- 172.21.15.16:0/2878637865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdc8005040 con 0x7fcdd4073310 2024-04-05T23:34:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.716+0000 7fcdd1600700 1 -- 172.21.15.16:0/2878637865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8003a50 con 0x7fcdd4073310 2024-04-05T23:34:17.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.716+0000 7fcdda000700 1 -- 172.21.15.16:0/2878637865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd41a9f60 con 0x7fcdd4073310 2024-04-05T23:34:17.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.716+0000 7fcdda000700 1 -- 172.21.15.16:0/2878637865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd41aa2d0 con 0x7fcdd4073310 2024-04-05T23:34:17.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.722+0000 7fcdd1600700 1 -- 172.21.15.16:0/2878637865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7fcdc8005420 con 0x7fcdd4073310 2024-04-05T23:34:17.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.722+0000 7fcdef097b80 1 Processor -- start 2024-04-05T23:34:17.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.722+0000 7fcdef097b80 1 -- start start 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.723+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec1376b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.723+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec142590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.723+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec142ad0 0x55feec144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.723+0000 7fcdef097b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec09ebe0 con 0x55feec137bf0 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.723+0000 7fcdef097b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec08f990 con 0x55feec142ad0 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.723+0000 7fcdef097b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec08d350 con 0x55feec1372d0 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.723+0000 7fcdd9600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec1376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.723+0000 7fcdd9600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec1376b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40018/0 (socket says 172.21.15.16:40018) 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.723+0000 7fcdd9600700 1 -- 172.21.15.16:0/1823715793 learned_addr learned my addr 172.21.15.16:0/1823715793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.724+0000 7fcdd9600700 1 -- 172.21.15.16:0/1823715793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec142ad0 msgr2=0x55feec144ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:34:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.724+0000 7fcdd9600700 1 --2- 172.21.15.16:0/1823715793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec142ad0 0x55feec144ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.724+0000 7fcdd9600700 1 -- 172.21.15.16:0/1823715793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 msgr2=0x55feec142590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-05T23:34:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.724+0000 7fcdd8c00700 1 --2- 172.21.15.16:0/1823715793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.724+0000 7fcdd9600700 1 --2- 172.21.15.16:0/1823715793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec142590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.724+0000 7fcdd9600700 1 -- 172.21.15.16:0/1823715793 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55feebf0a3b0 con 0x55feec1372d0 2024-04-05T23:34:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.724+0000 7fcdd8c00700 1 --2- 172.21.15.16:0/1823715793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.725+0000 7fcdd1600700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcdbc0a13b0 0x7fcdbc0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.725+0000 7fcdd9600700 1 --2- 172.21.15.16:0/1823715793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec1376b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc003eb0 tx=0x7fcdcc0099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8ac2801fa3f6132 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.725+0000 7fcdd1600700 1 -- 172.21.15.16:0/2878637865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fcdc80cdb60 con 0x7fcdd4073310 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.725+0000 7fcdd8c00700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcdbc0a13b0 0x7fcdbc0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.725+0000 7fcdc5c00700 1 -- 172.21.15.16:0/1823715793 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0092d0 con 0x55feec1372d0 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.725+0000 7fcdc5c00700 1 -- 172.21.15.16:0/1823715793 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdcc00f040 con 0x55feec1372d0 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.725+0000 7fcdc5c00700 1 -- 172.21.15.16:0/1823715793 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc00e750 con 0x55feec1372d0 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.726+0000 7fcdd8c00700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcdbc0a13b0 0x7fcdbc0a3870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80cb1a0 tx=0x7fcdc800bfd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.726+0000 7fcdef097b80 1 -- 172.21.15.16:0/1823715793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 msgr2=0x55feec1376b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.726+0000 7fcdef097b80 1 --2- 172.21.15.16:0/1823715793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec1376b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc003eb0 tx=0x7fcdcc0099e0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.726+0000 7fcdef097b80 1 -- 172.21.15.16:0/1823715793 shutdown_connections 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.726+0000 7fcdef097b80 1 --2- 172.21.15.16:0/1823715793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec142ad0 0x55feec144ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.726+0000 7fcdef097b80 1 --2- 172.21.15.16:0/1823715793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec1376b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.726+0000 7fcdef097b80 1 --2- 172.21.15.16:0/1823715793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.726+0000 7fcdef097b80 1 -- 172.21.15.16:0/1823715793 >> 172.21.15.16:0/1823715793 conn(0x55feec0ac220 msgr2=0x55feec1337d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.727+0000 7fcdef097b80 1 -- 172.21.15.16:0/1823715793 shutdown_connections 2024-04-05T23:34:17.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.727+0000 7fcdef097b80 1 -- 172.21.15.16:0/1823715793 wait complete. 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.727+0000 7fcdef097b80 1 -- start start 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.728+0000 7fcdef097b80 1 Processor -- start 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.728+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec1372d0 0x55feec142450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.728+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec137bf0 0x55feec13f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 0x55feec13fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdef097b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec088c40 con 0x55feec142ad0 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdef097b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec024d80 con 0x55feec1372d0 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdef097b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec024a20 con 0x55feec137bf0 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdef097b80 10 client.0 _ll_register_callbacks cb 0x55feebe903a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd9600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec1372d0 0x55feec142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd9600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec1372d0 0x55feec142450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42712/0 (socket says 172.21.15.16:42712) 2024-04-05T23:34:17.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd9600700 1 -- 172.21.15.16:0/2391980 learned_addr learned my addr 172.21.15.16:0/2391980 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec137bf0 0x55feec13f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd9600700 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 0x55feec13fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd3e00700 1 -- 172.21.15.16:0/2391980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec1372d0 msgr2=0x55feec142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec1372d0 0x55feec142450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd3e00700 1 -- 172.21.15.16:0/2391980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 msgr2=0x55feec13fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 0x55feec13fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd3e00700 1 -- 172.21.15.16:0/2391980 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdcc003b60 con 0x55feec137bf0 2024-04-05T23:34:17.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.729+0000 7fcdd9600700 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec1372d0 0x55feec142450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.730+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec137bf0 0x55feec13f740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00119d0 tx=0x7fcdc0011d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ce9e2e58ab86cbe server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.730+0000 7fcdba000700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.730+0000 7fcdba000700 1 -- 172.21.15.16:0/2391980 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc000faa0 con 0x55feec137bf0 2024-04-05T23:34:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.730+0000 7fcdef097b80 1 -- 172.21.15.16:0/2391980 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55feec12b490 con 0x55feec137bf0 2024-04-05T23:34:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.730+0000 7fcdba000700 1 -- 172.21.15.16:0/2391980 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fcdc000fc00 con 0x55feec137bf0 2024-04-05T23:34:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.731+0000 7fcdba000700 1 -- 172.21.15.16:0/2391980 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdc000fdc0 con 0x55feec137bf0 2024-04-05T23:34:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.731+0000 7fcdef097b80 10 client.25612 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.731+0000 7fcdef097b80 1 -- 172.21.15.16:0/2391980 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55feebeb6b60 con 0x55feec137bf0 2024-04-05T23:34:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.731+0000 7fcdba000700 1 -- 172.21.15.16:0/2391980 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdc0026020 con 0x55feec137bf0 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.731+0000 7fcdba000700 1 -- 172.21.15.16:0/2391980 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fcdc0003d00 con 0x55feec137bf0 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.731+0000 7fcdef097b80 10 client.25612 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.731+0000 7fcdef097b80 10 client.25612 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.731+0000 7fcdef097b80 20 client.25612 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.732+0000 7fcdef097b80 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55feec1526a0 0x55feec154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.732+0000 7fcdef097b80 4 client.25612 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.732+0000 7fcdef097b80 1 -- 172.21.15.16:0/2391980 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55feec081700 con 0x55feec1526a0 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.732+0000 7fcdd9600700 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55feec1526a0 0x55feec154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.733+0000 7fcdd9600700 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55feec1526a0 0x55feec154ac0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ca6129c4387d5c7 server_cookie=2f7bdae91ad3365b in_seq=0 out_seq=0 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.733+0000 7fcdba000700 10 client.25612 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.733+0000 7fcdba000700 1 -- 172.21.15.16:0/2391980 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x7fcdcc02b730 con 0x55feec1526a0 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.733+0000 7fcdba000700 10 client.25612 handle_command_reply: tid=0 2024-04-05T23:34:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.733+0000 7fcdef097b80 1 client.25612 shutdown 2024-04-05T23:34:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 -- 172.21.15.16:0/2391980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec137bf0 msgr2=0x55feec13f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec137bf0 0x55feec13f740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00119d0 tx=0x7fcdc0011d90 comp rx=0 tx=0).stop 2024-04-05T23:34:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 -- 172.21.15.16:0/2391980 shutdown_connections 2024-04-05T23:34:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55feec1526a0 0x55feec154ac0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 0x55feec13fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec1372d0 0x55feec142450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 --2- 172.21.15.16:0/2391980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec137bf0 0x55feec13f740 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdba000700 0 client.25612 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:34:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 -- 172.21.15.16:0/2391980 >> 172.21.15.16:0/2391980 conn(0x55feec0ac220 msgr2=0x55feec1333c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 -- 172.21.15.16:0/2391980 shutdown_connections 2024-04-05T23:34:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 1 -- 172.21.15.16:0/2391980 wait complete. 2024-04-05T23:34:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.734+0000 7fcdef097b80 20 client.25612 trim_cache size 0 max 16384 2024-04-05T23:34:17.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.760+0000 7fcdef097b80 1 Processor -- start 2024-04-05T23:34:17.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdef097b80 1 -- start start 2024-04-05T23:34:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec133970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec13e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec142ad0 0x55feec13e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdef097b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feebf0c6a0 con 0x55feec137bf0 2024-04-05T23:34:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdef097b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec1391a0 con 0x55feec142ad0 2024-04-05T23:34:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdef097b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec140650 con 0x55feec1372d0 2024-04-05T23:34:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdd9600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdd9600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec133970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40030/0 (socket says 172.21.15.16:40030) 2024-04-05T23:34:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.761+0000 7fcdd9600700 1 -- 172.21.15.16:0/4285765091 learned_addr learned my addr 172.21.15.16:0/4285765091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdd9600700 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec142ad0 0x55feec13e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec13e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdd9600700 1 -- 172.21.15.16:0/4285765091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec142ad0 msgr2=0x55feec13e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdd9600700 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec142ad0 0x55feec13e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdd9600700 1 -- 172.21.15.16:0/4285765091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 msgr2=0x55feec13e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdd9600700 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec13e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdd9600700 1 -- 172.21.15.16:0/4285765091 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55feec04eea0 con 0x55feec1372d0 2024-04-05T23:34:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec13e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdd9600700 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec133970 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc003eb0 tx=0x7fcdcc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b51d49e05761669 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdc6600700 1 -- 172.21.15.16:0/4285765091 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc007ea0 con 0x55feec1372d0 2024-04-05T23:34:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdc6600700 1 -- 172.21.15.16:0/4285765091 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdcc027070 con 0x55feec1372d0 2024-04-05T23:34:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.762+0000 7fcdc6600700 1 -- 172.21.15.16:0/4285765091 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc005460 con 0x55feec1372d0 2024-04-05T23:34:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.763+0000 7fcdef097b80 1 -- 172.21.15.16:0/4285765091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 msgr2=0x55feec133970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.763+0000 7fcdef097b80 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec133970 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc003eb0 tx=0x7fcdcc00b040 comp rx=0 tx=0).stop 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.763+0000 7fcdef097b80 1 -- 172.21.15.16:0/4285765091 shutdown_connections 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.763+0000 7fcdef097b80 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec142ad0 0x55feec13e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.763+0000 7fcdef097b80 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec133970 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.763+0000 7fcdef097b80 1 --2- 172.21.15.16:0/4285765091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec137bf0 0x55feec13e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.763+0000 7fcdef097b80 1 -- 172.21.15.16:0/4285765091 >> 172.21.15.16:0/4285765091 conn(0x55feec0ac220 msgr2=0x55feec13aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.764+0000 7fcdef097b80 1 -- 172.21.15.16:0/4285765091 shutdown_connections 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.764+0000 7fcdef097b80 1 -- 172.21.15.16:0/4285765091 wait complete. 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.764+0000 7fcdef097b80 1 -- start start 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.764+0000 7fcdef097b80 1 Processor -- start 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.765+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec149490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.765+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec137bf0 0x55feec147b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.765+0000 7fcdef097b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 0x55feec148040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.765+0000 7fcdef097b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec148580 con 0x55feec142ad0 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.765+0000 7fcdef097b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec1486f0 con 0x55feec137bf0 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.765+0000 7fcdef097b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feec148860 con 0x55feec1372d0 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.765+0000 7fcdef097b80 10 client.0 _ll_register_callbacks cb 0x7fcdc0011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.765+0000 7fcdd9600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd9600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec149490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40032/0 (socket says 172.21.15.16:40032) 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd9600700 1 -- 172.21.15.16:0/458193 learned_addr learned my addr 172.21.15.16:0/458193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd9600700 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 0x55feec148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec137bf0 0x55feec147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd3e00700 1 -- 172.21.15.16:0/458193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 msgr2=0x55feec149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd3e00700 1 -- 172.21.15.16:0/458193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 msgr2=0x55feec148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:17.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 0x55feec148040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd3e00700 1 -- 172.21.15.16:0/458193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55feebf0a3b0 con 0x55feec137bf0 2024-04-05T23:34:17.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd9600700 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 0x55feec148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:17.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdd3e00700 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec137bf0 0x55feec147b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55feec14fdc0 tx=0x7fcdc0019d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26322b54b6ad6e67 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.766+0000 7fcdba000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.767+0000 7fcdba000700 1 -- 172.21.15.16:0/458193 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00054c0 con 0x55feec137bf0 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.767+0000 7fcdba000700 1 -- 172.21.15.16:0/458193 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fcdc0000e40 con 0x55feec137bf0 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.767+0000 7fcdef097b80 1 -- 172.21.15.16:0/458193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcdcc00f040 con 0x55feec137bf0 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.767+0000 7fcdba000700 1 -- 172.21.15.16:0/458193 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdc00092d0 con 0x55feec137bf0 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.767+0000 7fcdef097b80 10 client.25592 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.767+0000 7fcdef097b80 1 -- 172.21.15.16:0/458193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55feebf0ecd0 con 0x55feec137bf0 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.767+0000 7fcdba000700 1 -- 172.21.15.16:0/458193 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdcc00f040 con 0x55feec137bf0 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.768+0000 7fcdba000700 1 -- 172.21.15.16:0/458193 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fcdc0003b40 con 0x55feec137bf0 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.768+0000 7fcdef097b80 10 client.25592 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.768+0000 7fcdef097b80 10 client.25592 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-05T23:34:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.768+0000 7fcdef097b80 20 client.25592 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.768+0000 7fcdef097b80 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55feec1a18d0 0x55feec1a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.768+0000 7fcdef097b80 4 client.25592 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.768+0000 7fcdef097b80 1 -- 172.21.15.16:0/458193 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55feebed7cf0 con 0x55feec1a18d0 2024-04-05T23:34:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.768+0000 7fcdd9600700 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55feec1a18d0 0x55feec1a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.769+0000 7fcdd9600700 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55feec1a18d0 0x55feec1a1ce0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5aafdc8d34b38d06 server_cookie=d838cfb512b91799 in_seq=0 out_seq=0 2024-04-05T23:34:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.769+0000 7fcdba000700 10 client.25592 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:34:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.769+0000 7fcdba000700 1 -- 172.21.15.16:0/458193 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55feebed7cf0 con 0x55feec1a18d0 2024-04-05T23:34:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.769+0000 7fcdba000700 10 client.25592 handle_command_reply: tid=0 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.769+0000 7fcdef097b80 1 client.25592 shutdown 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.769+0000 7fcdef097b80 1 -- 172.21.15.16:0/458193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec137bf0 msgr2=0x55feec147b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.769+0000 7fcdef097b80 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec137bf0 0x55feec147b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55feec14fdc0 tx=0x7fcdc0019d30 comp rx=0 tx=0).stop 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdef097b80 1 -- 172.21.15.16:0/458193 shutdown_connections 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdef097b80 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55feec1a18d0 0x55feec1a1ce0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdef097b80 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55feec142ad0 0x55feec148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdef097b80 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55feec1372d0 0x55feec149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdba000700 0 client.25592 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdef097b80 1 --2- 172.21.15.16:0/458193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55feec137bf0 0x55feec147b00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdef097b80 1 -- 172.21.15.16:0/458193 >> 172.21.15.16:0/458193 conn(0x55feec0ac220 msgr2=0x55feec13dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdef097b80 1 -- 172.21.15.16:0/458193 shutdown_connections 2024-04-05T23:34:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdef097b80 1 -- 172.21.15.16:0/458193 wait complete. 2024-04-05T23:34:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.770+0000 7fcdef097b80 20 client.25592 trim_cache size 0 max 16384 2024-04-05T23:34:17.766 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 -- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcdbc0a13b0 msgr2=0x7fcdbc0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcdbc0a13b0 0x7fcdbc0a3870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80cb1a0 tx=0x7fcdc800bfd0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 -- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd4073310 msgr2=0x7fcdd410cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd4073310 0x7fcdd410cd90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009230 tx=0x7fcdc800fa00 comp rx=0 tx=0).stop 2024-04-05T23:34:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 -- 172.21.15.16:0/2878637865 shutdown_connections 2024-04-05T23:34:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcdbc0a13b0 0x7fcdbc0a3870 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd4073310 0x7fcdd410cd90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd410d4c0 0x7fcdd410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 --2- 172.21.15.16:0/2878637865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd4073c30 0x7fcdd4109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.771+0000 7fcdda000700 1 -- 172.21.15.16:0/2878637865 >> 172.21.15.16:0/2878637865 conn(0x7fcdd40fc770 msgr2=0x7fcdd40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.772+0000 7fcdda000700 1 -- 172.21.15.16:0/2878637865 shutdown_connections 2024-04-05T23:34:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:17.772+0000 7fcdda000700 1 -- 172.21.15.16:0/2878637865 wait complete. 2024-04-05T23:34:17.786 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:17 smithi016 ceph-mon[38248]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-04-05T23:34:17.786 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:17 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2789073351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:34:17.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-05T23:34:17.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:17 smithi027 ceph-mon[78597]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-04-05T23:34:17.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:17 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2789073351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:34:18.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.032+0000 7f997fe00700 1 -- 172.21.15.16:0/3030739961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9980108240 msgr2=0x7f998010fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.032+0000 7f997fe00700 1 --2- 172.21.15.16:0/3030739961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9980108240 0x7f998010fdc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9978009f90 tx=0x7f997800b330 comp rx=0 tx=0).stop 2024-04-05T23:34:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.032+0000 7f997fe00700 1 -- 172.21.15.16:0/3030739961 shutdown_connections 2024-04-05T23:34:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.032+0000 7f997fe00700 1 --2- 172.21.15.16:0/3030739961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9980108240 0x7f998010fdc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.032+0000 7f997fe00700 1 --2- 172.21.15.16:0/3030739961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9980103120 0x7f9980103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.032+0000 7f997fe00700 1 --2- 172.21.15.16:0/3030739961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9980103ad0 0x7f9980107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.032+0000 7f997fe00700 1 -- 172.21.15.16:0/3030739961 >> 172.21.15.16:0/3030739961 conn(0x7f99800fe940 msgr2=0x7f9980100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.032+0000 7f997fe00700 1 -- 172.21.15.16:0/3030739961 shutdown_connections 2024-04-05T23:34:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.032+0000 7f997fe00700 1 -- 172.21.15.16:0/3030739961 wait complete. 2024-04-05T23:34:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.033+0000 7f997fe00700 1 Processor -- start 2024-04-05T23:34:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.034+0000 7f997fe00700 1 -- start start 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.034+0000 7f997fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9980103120 0x7f9980199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.034+0000 7f997fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9980108240 0x7f9980199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.034+0000 7f997fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f998019dc30 0x7f998019e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.034+0000 7f997fe00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998019e6b0 con 0x7f9980108240 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.034+0000 7f997fe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998019e820 con 0x7f998019dc30 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.034+0000 7f997fe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998019e9e0 con 0x7f9980103120 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.034+0000 7f997ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9980103120 0x7f9980199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.034+0000 7f997f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f998019dc30 0x7f998019e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.035+0000 7f997ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9980103120 0x7f9980199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40048/0 (socket says 172.21.15.16:40048) 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.035+0000 7f997f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f998019dc30 0x7f998019e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42746/0 (socket says 172.21.15.16:42746) 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.035+0000 7f997ea00700 1 -- 172.21.15.16:0/3502722530 learned_addr learned my addr 172.21.15.16:0/3502722530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.035+0000 7f997f400700 1 -- 172.21.15.16:0/3502722530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9980103120 msgr2=0x7f9980199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.035+0000 7f997f400700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9980103120 0x7f9980199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.035+0000 7f997f400700 1 -- 172.21.15.16:0/3502722530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9980108240 msgr2=0x7f9980199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.035+0000 7f997f400700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9980108240 0x7f9980199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.035+0000 7f997f400700 1 -- 172.21.15.16:0/3502722530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99700095f0 con 0x7f998019dc30 2024-04-05T23:34:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.035+0000 7f997f400700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f998019dc30 0x7f998019e0b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f997800b6c0 tx=0x7f9978008fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.036+0000 7f9984a00700 1 -- 172.21.15.16:0/3502722530 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997800e040 con 0x7f998019dc30 2024-04-05T23:34:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.036+0000 7f9984a00700 1 -- 172.21.15.16:0/3502722530 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f997801d470 con 0x7f998019dc30 2024-04-05T23:34:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.036+0000 7f9984a00700 1 -- 172.21.15.16:0/3502722530 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99780052a0 con 0x7f998019dc30 2024-04-05T23:34:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.036+0000 7f997fe00700 1 -- 172.21.15.16:0/3502722530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9978009c40 con 0x7f998019dc30 2024-04-05T23:34:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.036+0000 7f997fe00700 1 -- 172.21.15.16:0/3502722530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99801a6180 con 0x7f998019dc30 2024-04-05T23:34:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.039+0000 7f9984a00700 1 -- 172.21.15.16:0/3502722530 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f997801d5e0 con 0x7f998019dc30 2024-04-05T23:34:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.040+0000 7f99998ecb80 1 Processor -- start 2024-04-05T23:34:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f9984a00700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f99640a1430 0x7f99640a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f9984a00700 1 -- 172.21.15.16:0/3502722530 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f99780ce510 con 0x7f998019dc30 2024-04-05T23:34:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f99998ecb80 1 -- start start 2024-04-05T23:34:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f997e000700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f99640a1430 0x7f99640a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372816b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x55973728c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x55973728eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f99998ecb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597371e8be0 con 0x559737281bf0 2024-04-05T23:34:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f99998ecb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597371d9990 con 0x5597372812d0 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.041+0000 7f99998ecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597371d7350 con 0x55973728cad0 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372816b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372816b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42758/0 (socket says 172.21.15.16:42758) 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997ea00700 1 -- 172.21.15.16:0/3980741955 learned_addr learned my addr 172.21.15.16:0/3980741955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997f400700 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x55973728eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997e000700 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x55973728c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997f400700 1 -- 172.21.15.16:0/3980741955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 msgr2=0x5597372816b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997f400700 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372816b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997f400700 1 -- 172.21.15.16:0/3980741955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 msgr2=0x55973728c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997f400700 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x55973728c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997f400700 1 -- 172.21.15.16:0/3980741955 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597370543b0 con 0x55973728cad0 2024-04-05T23:34:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997e000700 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x55973728c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.042+0000 7f997f400700 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x55973728eec0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f99780cbab0 tx=0x7f997801df90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ac507d62ff87b9f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.043+0000 7f996be00700 1 -- 172.21.15.16:0/3980741955 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99780860e0 con 0x55973728cad0 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.043+0000 7f997e000700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f99640a1430 0x7f99640a38f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f998019a8a0 tx=0x7f996c0096e0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.043+0000 7f996be00700 1 -- 172.21.15.16:0/3980741955 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f997809bb30 con 0x55973728cad0 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.043+0000 7f996be00700 1 -- 172.21.15.16:0/3980741955 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997809b8f0 con 0x55973728cad0 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.043+0000 7f99998ecb80 1 -- 172.21.15.16:0/3980741955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 msgr2=0x55973728eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.043+0000 7f99998ecb80 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x55973728eec0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f99780cbab0 tx=0x7f997801df90 comp rx=0 tx=0).stop 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.044+0000 7f99998ecb80 1 -- 172.21.15.16:0/3980741955 shutdown_connections 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.044+0000 7f99998ecb80 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x55973728eec0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.044+0000 7f99998ecb80 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372816b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.044+0000 7f99998ecb80 1 --2- 172.21.15.16:0/3980741955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x55973728c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.044+0000 7f99998ecb80 1 -- 172.21.15.16:0/3980741955 >> 172.21.15.16:0/3980741955 conn(0x5597371f6220 msgr2=0x55973727d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.044+0000 7f99998ecb80 1 -- 172.21.15.16:0/3980741955 shutdown_connections 2024-04-05T23:34:18.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.044+0000 7f99998ecb80 1 -- 172.21.15.16:0/3980741955 wait complete. 2024-04-05T23:34:18.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.044+0000 7f99998ecb80 1 -- start start 2024-04-05T23:34:18.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.045+0000 7f99998ecb80 1 Processor -- start 2024-04-05T23:34:18.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.045+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5597372812d0 0x55973728c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.045+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559737281bf0 0x559737289840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.045+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x559737289f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.045+0000 7f99998ecb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597371d2c40 con 0x5597372812d0 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.045+0000 7f99998ecb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55973716ed80 con 0x559737281bf0 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.045+0000 7f99998ecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55973716ea20 con 0x55973728cad0 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.045+0000 7f99998ecb80 10 client.0 _ll_register_callbacks cb 0x559736fda3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5597372812d0 0x55973728c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x559737289f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x559737289f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40066/0 (socket says 172.21.15.16:40066) 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997f400700 1 -- 172.21.15.16:0/1045173813 learned_addr learned my addr 172.21.15.16:0/1045173813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997e000700 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559737281bf0 0x559737289840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997f400700 1 -- 172.21.15.16:0/1045173813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559737281bf0 msgr2=0x559737289840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997f400700 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559737281bf0 0x559737289840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997f400700 1 -- 172.21.15.16:0/1045173813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5597372812d0 msgr2=0x55973728c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997f400700 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5597372812d0 0x55973728c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997f400700 1 -- 172.21.15.16:0/1045173813 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99801a6180 con 0x55973728cad0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f997f400700 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x559737289f80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9978009790 tx=0x7f997801daf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27104f952370eb70 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f9968c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.046+0000 7f9968c00700 1 -- 172.21.15.16:0/1045173813 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99780a3820 con 0x55973728cad0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.047+0000 7f9968c00700 1 -- 172.21.15.16:0/1045173813 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f997809f260 con 0x55973728cad0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.047+0000 7f99998ecb80 1 -- 172.21.15.16:0/1045173813 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559737275490 con 0x55973728cad0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.047+0000 7f9968c00700 1 -- 172.21.15.16:0/1045173813 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99780860b0 con 0x55973728cad0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.047+0000 7f99998ecb80 10 client.25630 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.047+0000 7f99998ecb80 1 -- 172.21.15.16:0/1045173813 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x559737000b60 con 0x55973728cad0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.047+0000 7f9968c00700 1 -- 172.21.15.16:0/1045173813 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559737275490 con 0x55973728cad0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.047+0000 7f9968c00700 1 -- 172.21.15.16:0/1045173813 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f99780cbe70 con 0x55973728cad0 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.047+0000 7f99998ecb80 10 client.25630 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:18.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.047+0000 7f99998ecb80 10 client.25630 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-05T23:34:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.048+0000 7f99998ecb80 20 client.25630 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.048+0000 7f99998ecb80 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55973729bee0 0x55973729e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.048+0000 7f99998ecb80 4 client.25630 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.048+0000 7f99998ecb80 1 -- 172.21.15.16:0/1045173813 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597371cb700 con 0x55973729bee0 2024-04-05T23:34:18.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.048+0000 7f997ea00700 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55973729bee0 0x55973729e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.048+0000 7f997ea00700 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55973729bee0 0x55973729e300 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b97c8bf859a5366 server_cookie=57f97a8297cf920f in_seq=0 out_seq=0 2024-04-05T23:34:18.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.048+0000 7f9968c00700 10 client.25630 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:34:18.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.049+0000 7f9968c00700 1 -- 172.21.15.16:0/1045173813 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5597371cb700 con 0x55973729bee0 2024-04-05T23:34:18.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.049+0000 7f9968c00700 10 client.25630 handle_command_reply: tid=0 2024-04-05T23:34:18.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.049+0000 7f99998ecb80 1 client.25630 shutdown 2024-04-05T23:34:18.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.049+0000 7f99998ecb80 1 -- 172.21.15.16:0/1045173813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 msgr2=0x559737289f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.049+0000 7f99998ecb80 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x559737289f80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9978009790 tx=0x7f997801daf0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f99998ecb80 1 -- 172.21.15.16:0/1045173813 shutdown_connections 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f99998ecb80 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5597372812d0 0x55973728c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f99998ecb80 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55973729bee0 0x55973729e300 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f99998ecb80 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x559737289f80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f99998ecb80 1 --2- 172.21.15.16:0/1045173813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559737281bf0 0x559737289840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f99998ecb80 1 -- 172.21.15.16:0/1045173813 >> 172.21.15.16:0/1045173813 conn(0x5597371f6220 msgr2=0x55973727d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f9968c00700 0 client.25630 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f99998ecb80 1 -- 172.21.15.16:0/1045173813 shutdown_connections 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f99998ecb80 1 -- 172.21.15.16:0/1045173813 wait complete. 2024-04-05T23:34:18.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.050+0000 7f99998ecb80 20 client.25630 trim_cache size 0 max 16384 2024-04-05T23:34:18.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.064+0000 7f99998ecb80 1 Processor -- start 2024-04-05T23:34:18.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.064+0000 7f99998ecb80 1 -- start start 2024-04-05T23:34:18.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.064+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372934c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.064+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559737281bf0 0x55973727bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.064+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55973728cad0 0x55973728c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.064+0000 7f99998ecb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597370566a0 con 0x55973728cad0 2024-04-05T23:34:18.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.064+0000 7f99998ecb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559737282540 con 0x5597372812d0 2024-04-05T23:34:18.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.064+0000 7f99998ecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559737293a00 con 0x559737281bf0 2024-04-05T23:34:18.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372934c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372934c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42768/0 (socket says 172.21.15.16:42768) 2024-04-05T23:34:18.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997ea00700 1 -- 172.21.15.16:0/3124289016 learned_addr learned my addr 172.21.15.16:0/3124289016 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997e000700 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559737281bf0 0x55973727bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997f400700 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55973728cad0 0x55973728c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997ea00700 1 -- 172.21.15.16:0/3124289016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559737281bf0 msgr2=0x55973727bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997ea00700 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559737281bf0 0x55973727bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997ea00700 1 -- 172.21.15.16:0/3124289016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55973728cad0 msgr2=0x55973728c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997ea00700 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55973728cad0 0x55973728c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997ea00700 1 -- 172.21.15.16:0/3124289016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559737198ea0 con 0x5597372812d0 2024-04-05T23:34:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997f400700 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55973728cad0 0x55973728c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.065+0000 7f997ea00700 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372934c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9970005d10 tx=0x7f9970009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=469228d5b1265d33 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.066+0000 7f9975200700 1 -- 172.21.15.16:0/3124289016 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9970011bc0 con 0x5597372812d0 2024-04-05T23:34:18.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.066+0000 7f9975200700 1 -- 172.21.15.16:0/3124289016 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9970018040 con 0x5597372812d0 2024-04-05T23:34:18.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.066+0000 7f9975200700 1 -- 172.21.15.16:0/3124289016 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9970004e00 con 0x5597372812d0 2024-04-05T23:34:18.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.066+0000 7f99998ecb80 1 -- 172.21.15.16:0/3124289016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 msgr2=0x5597372934c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.067+0000 7f99998ecb80 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372934c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9970005d10 tx=0x7f9970009310 comp rx=0 tx=0).stop 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.067+0000 7f99998ecb80 1 -- 172.21.15.16:0/3124289016 shutdown_connections 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.067+0000 7f99998ecb80 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55973728cad0 0x55973728c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.067+0000 7f99998ecb80 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x5597372934c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.067+0000 7f99998ecb80 1 --2- 172.21.15.16:0/3124289016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559737281bf0 0x55973727bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.067+0000 7f99998ecb80 1 -- 172.21.15.16:0/3124289016 >> 172.21.15.16:0/3124289016 conn(0x5597371f6220 msgr2=0x5597372791a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.067+0000 7f99998ecb80 1 -- 172.21.15.16:0/3124289016 shutdown_connections 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.067+0000 7f99998ecb80 1 -- 172.21.15.16:0/3124289016 wait complete. 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.068+0000 7f99998ecb80 1 -- start start 2024-04-05T23:34:18.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.068+0000 7f99998ecb80 1 Processor -- start 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.068+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x559737291690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.068+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x559737291bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.068+0000 7f99998ecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x5597372eed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.068+0000 7f99998ecb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597372922b0 con 0x559737281bf0 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.068+0000 7f99998ecb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597372ef250 con 0x5597372812d0 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.068+0000 7f99998ecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597372ef390 con 0x55973728cad0 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f99998ecb80 10 client.0 _ll_register_callbacks cb 0x7f9978097860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x559737291bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x559737291bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42910/0 (socket says 172.21.15.16:42910) 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997e000700 1 -- 172.21.15.16:0/19392005 learned_addr learned my addr 172.21.15.16:0/19392005 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997ea00700 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x559737291690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997f400700 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x5597372eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997e000700 1 -- 172.21.15.16:0/19392005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 msgr2=0x5597372eed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997e000700 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x5597372eed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997e000700 1 -- 172.21.15.16:0/19392005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 msgr2=0x559737291690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997e000700 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x559737291690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.069+0000 7f997e000700 1 -- 172.21.15.16:0/19392005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9970009e50 con 0x559737281bf0 2024-04-05T23:34:18.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.070+0000 7f997f400700 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x5597372eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:18.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.070+0000 7f997ea00700 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x559737291690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:18.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.070+0000 7f997e000700 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x559737291bd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f998006a370 tx=0x7f996c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223ae9c58aa84a00 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.070+0000 7f9968c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:18.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.070+0000 7f9968c00700 1 -- 172.21.15.16:0/19392005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c004a30 con 0x559737281bf0 2024-04-05T23:34:18.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.070+0000 7f99998ecb80 1 -- 172.21.15.16:0/19392005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f997809bb30 con 0x559737281bf0 2024-04-05T23:34:18.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.070+0000 7f9968c00700 1 -- 172.21.15.16:0/19392005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f996c01b040 con 0x559737281bf0 2024-04-05T23:34:18.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.070+0000 7f9968c00700 1 -- 172.21.15.16:0/19392005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f996c004b90 con 0x559737281bf0 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.070+0000 7f99998ecb80 10 client.15570 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.071+0000 7f99998ecb80 1 -- 172.21.15.16:0/19392005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x559737058cd0 con 0x559737281bf0 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.071+0000 7f9968c00700 1 -- 172.21.15.16:0/19392005 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f997809bb30 con 0x559737281bf0 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.071+0000 7f9968c00700 1 -- 172.21.15.16:0/19392005 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f996c033800 con 0x559737281bf0 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.071+0000 7f99998ecb80 10 client.15570 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.071+0000 7f99998ecb80 10 client.15570 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.071+0000 7f99998ecb80 20 client.15570 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.071+0000 7f99998ecb80 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x5597372f0b10 0x5597372f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.071+0000 7f99998ecb80 4 client.15570 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.071+0000 7f99998ecb80 1 -- 172.21.15.16:0/19392005 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559737021cf0 con 0x5597372f0b10 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.072+0000 7f997ea00700 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x5597372f0b10 0x5597372f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.072+0000 7f997ea00700 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x5597372f0b10 0x5597372f2f30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f782bc33722235b server_cookie=77fab9aa7b7dae19 in_seq=0 out_seq=0 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.072+0000 7f9968c00700 10 client.15570 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:34:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.072+0000 7f9968c00700 1 -- 172.21.15.16:0/19392005 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559737021cf0 con 0x5597372f0b10 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.072+0000 7f9968c00700 10 client.15570 handle_command_reply: tid=0 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.072+0000 7f99998ecb80 1 client.15570 shutdown 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 -- 172.21.15.16:0/19392005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 msgr2=0x559737291bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x559737291bd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f998006a370 tx=0x7f996c004750 comp rx=0 tx=0).stop 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 -- 172.21.15.16:0/19392005 shutdown_connections 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x5597372f0b10 0x5597372f2f30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55973728cad0 0x5597372eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597372812d0 0x559737291690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 --2- 172.21.15.16:0/19392005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559737281bf0 0x559737291bd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 -- 172.21.15.16:0/19392005 >> 172.21.15.16:0/19392005 conn(0x5597371f6220 msgr2=0x55973729def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f9968c00700 0 client.15570 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 -- 172.21.15.16:0/19392005 shutdown_connections 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 1 -- 172.21.15.16:0/19392005 wait complete. 2024-04-05T23:34:18.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.073+0000 7f99998ecb80 20 client.15570 trim_cache size 0 max 16384 2024-04-05T23:34:18.068 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:18.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 -- 172.21.15.16:0/3502722530 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f99640a1430 msgr2=0x7f99640a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f99640a1430 0x7f99640a38f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f998019a8a0 tx=0x7f996c0096e0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 -- 172.21.15.16:0/3502722530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f998019dc30 msgr2=0x7f998019e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f998019dc30 0x7f998019e0b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f997800b6c0 tx=0x7f9978008fb0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 -- 172.21.15.16:0/3502722530 shutdown_connections 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f99640a1430 0x7f99640a38f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f998019dc30 0x7f998019e0b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9980103120 0x7f9980199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 --2- 172.21.15.16:0/3502722530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9980108240 0x7f9980199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 -- 172.21.15.16:0/3502722530 >> 172.21.15.16:0/3502722530 conn(0x7f99800fe940 msgr2=0x7f9980100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 -- 172.21.15.16:0/3502722530 shutdown_connections 2024-04-05T23:34:18.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.074+0000 7f997fe00700 1 -- 172.21.15.16:0/3502722530 wait complete. 2024-04-05T23:34:18.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:17 smithi130 ceph-mon[79417]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-04-05T23:34:18.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:17 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2789073351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:34:18.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-05T23:34:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.328+0000 7fbac9800700 1 -- 172.21.15.16:0/4075777127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbac41010a0 msgr2=0x7fbac4101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.328+0000 7fbac9800700 1 --2- 172.21.15.16:0/4075777127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbac41010a0 0x7fbac4101480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009990 tx=0x7fbab4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.328+0000 7fbac9800700 1 -- 172.21.15.16:0/4075777127 shutdown_connections 2024-04-05T23:34:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.328+0000 7fbac9800700 1 --2- 172.21.15.16:0/4075777127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41061c0 0x7fbac410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.328+0000 7fbac9800700 1 --2- 172.21.15.16:0/4075777127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbac41010a0 0x7fbac4101480 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.328+0000 7fbac9800700 1 --2- 172.21.15.16:0/4075777127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac4101a50 0x7fbac4105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.328+0000 7fbac9800700 1 -- 172.21.15.16:0/4075777127 >> 172.21.15.16:0/4075777127 conn(0x7fbac40fc900 msgr2=0x7fbac40fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.328+0000 7fbac9800700 1 -- 172.21.15.16:0/4075777127 shutdown_connections 2024-04-05T23:34:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.328+0000 7fbac9800700 1 -- 172.21.15.16:0/4075777127 wait complete. 2024-04-05T23:34:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.329+0000 7fbac9800700 1 Processor -- start 2024-04-05T23:34:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.329+0000 7fbac9800700 1 -- start start 2024-04-05T23:34:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac9800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41010a0 0x7fbac40731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac9800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac41061c0 0x7fbac4073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac9800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbac4073c60 0x7fbac41a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac9800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41a5ba0 con 0x7fbac41061c0 2024-04-05T23:34:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac9800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41a5d10 con 0x7fbac41010a0 2024-04-05T23:34:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac9800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41a5ed0 con 0x7fbac4073c60 2024-04-05T23:34:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41010a0 0x7fbac40731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41010a0 0x7fbac40731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42784/0 (socket says 172.21.15.16:42784) 2024-04-05T23:34:18.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac2a00700 1 -- 172.21.15.16:0/2522687685 learned_addr learned my addr 172.21.15.16:0/2522687685 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac2000700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac41061c0 0x7fbac4073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.330+0000 7fbac3400700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbac4073c60 0x7fbac41a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.331+0000 7fbac2a00700 1 -- 172.21.15.16:0/2522687685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbac4073c60 msgr2=0x7fbac41a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.331+0000 7fbac2a00700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbac4073c60 0x7fbac41a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.331+0000 7fbac2a00700 1 -- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac41061c0 msgr2=0x7fbac4073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.331+0000 7fbac2a00700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac41061c0 0x7fbac4073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.331+0000 7fbac2a00700 1 -- 172.21.15.16:0/2522687685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab40095f0 con 0x7fbac41010a0 2024-04-05T23:34:18.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.331+0000 7fbac2a00700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41010a0 0x7fbac40731e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbab4000c00 tx=0x7fbab4005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.332+0000 7fbabb400700 1 -- 172.21.15.16:0/2522687685 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab401d070 con 0x7fbac41010a0 2024-04-05T23:34:18.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.332+0000 7fbabb400700 1 -- 172.21.15.16:0/2522687685 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbab4022470 con 0x7fbac41010a0 2024-04-05T23:34:18.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.332+0000 7fbabb400700 1 -- 172.21.15.16:0/2522687685 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4017650 con 0x7fbac41010a0 2024-04-05T23:34:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.332+0000 7fbac9800700 1 -- 172.21.15.16:0/2522687685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac41a6150 con 0x7fbac41010a0 2024-04-05T23:34:18.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.332+0000 7fbac9800700 1 -- 172.21.15.16:0/2522687685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac41a6520 con 0x7fbac41010a0 2024-04-05T23:34:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.336+0000 7fbadd291b80 1 Processor -- start 2024-04-05T23:34:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.337+0000 7fbadd291b80 1 -- start start 2024-04-05T23:34:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.337+0000 7fbabb400700 1 -- 172.21.15.16:0/2522687685 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7fbab40225e0 con 0x7fbac41010a0 2024-04-05T23:34:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.337+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 0x55a5d83fa560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.337+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d83f4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.337+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d8400f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.337+0000 7fbadd291b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d835abe0 con 0x55a5d83f42e0 2024-04-05T23:34:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.337+0000 7fbadd291b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d834b990 con 0x55a5d83f4ca0 2024-04-05T23:34:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.337+0000 7fbadd291b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d8349350 con 0x55a5d83fa180 2024-04-05T23:34:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.338+0000 7fbac2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d83f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.338+0000 7fbac2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d83f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42932/0 (socket says 172.21.15.16:42932) 2024-04-05T23:34:18.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.338+0000 7fbac2a00700 1 -- 172.21.15.16:0/3506281996 learned_addr learned my addr 172.21.15.16:0/3506281996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.338+0000 7fbac3400700 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 0x55a5d83fa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.338+0000 7fbac2000700 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d8400f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.338+0000 7fbac2a00700 1 -- 172.21.15.16:0/3506281996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 msgr2=0x55a5d83fa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.338+0000 7fbac2a00700 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 0x55a5d83fa560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.338+0000 7fbac2a00700 1 -- 172.21.15.16:0/3506281996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 msgr2=0x55a5d8400f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.338+0000 7fbac2a00700 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d8400f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.339+0000 7fbac2a00700 1 -- 172.21.15.16:0/3506281996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5d81c63b0 con 0x55a5d83f42e0 2024-04-05T23:34:18.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.339+0000 7fbac3400700 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 0x55a5d83fa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:18.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.339+0000 7fbabb400700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbaa80a13b0 0x7fbaa80a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.339+0000 7fbac2a00700 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d83f4760 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbab40cb920 tx=0x7fbab40d65d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef80aa6812a0ee3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.339+0000 7fbabb400700 1 -- 172.21.15.16:0/2522687685 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fbab40ce2b0 con 0x7fbac41010a0 2024-04-05T23:34:18.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.340+0000 7fbac3400700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbaa80a13b0 0x7fbaa80a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.340+0000 7fbaafe00700 1 -- 172.21.15.16:0/3506281996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4097550 con 0x55a5d83f42e0 2024-04-05T23:34:18.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.340+0000 7fbaafe00700 1 -- 172.21.15.16:0/3506281996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbab409afe0 con 0x55a5d83f42e0 2024-04-05T23:34:18.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.340+0000 7fbaafe00700 1 -- 172.21.15.16:0/3506281996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab409ad50 con 0x55a5d83f42e0 2024-04-05T23:34:18.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbadd291b80 1 -- 172.21.15.16:0/3506281996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 msgr2=0x55a5d83f4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbac3400700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbaa80a13b0 0x7fbaa80a3870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009710 tx=0x7fbabc009100 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbadd291b80 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d83f4760 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbab40cb920 tx=0x7fbab40d65d0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbadd291b80 1 -- 172.21.15.16:0/3506281996 shutdown_connections 2024-04-05T23:34:18.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbadd291b80 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d8400f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbadd291b80 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 0x55a5d83fa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbadd291b80 1 --2- 172.21.15.16:0/3506281996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d83f4760 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbadd291b80 1 -- 172.21.15.16:0/3506281996 >> 172.21.15.16:0/3506281996 conn(0x55a5d8368220 msgr2=0x55a5d83ef570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbadd291b80 1 -- 172.21.15.16:0/3506281996 shutdown_connections 2024-04-05T23:34:18.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.341+0000 7fbadd291b80 1 -- 172.21.15.16:0/3506281996 wait complete. 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.342+0000 7fbadd291b80 1 -- start start 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.342+0000 7fbadd291b80 1 Processor -- start 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83f42e0 0x55a5d8400c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d83fdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83fa180 0x55a5d83fe4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbadd291b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d8344c40 con 0x55a5d83fa180 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbadd291b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d82e0d80 con 0x55a5d83f4ca0 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbadd291b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d82e0a20 con 0x55a5d83f42e0 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbadd291b80 10 client.0 _ll_register_callbacks cb 0x55a5d814c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbac3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83fa180 0x55a5d83fe4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbac2000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d83fdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbac3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83fa180 0x55a5d83fe4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42946/0 (socket says 172.21.15.16:42946) 2024-04-05T23:34:18.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbac2000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d83fdf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42800/0 (socket says 172.21.15.16:42800) 2024-04-05T23:34:18.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.343+0000 7fbac3400700 1 -- 172.21.15.16:0/4257527351 learned_addr learned my addr 172.21.15.16:0/4257527351 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.344+0000 7fbac2000700 1 -- 172.21.15.16:0/4257527351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83f42e0 msgr2=0x55a5d8400c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.344+0000 7fbac2000700 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83f42e0 0x55a5d8400c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.344+0000 7fbac2000700 1 -- 172.21.15.16:0/4257527351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83fa180 msgr2=0x55a5d83fe4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.344+0000 7fbac2000700 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83fa180 0x55a5d83fe4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.344+0000 7fbac2000700 1 -- 172.21.15.16:0/4257527351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc003ae0 con 0x55a5d83f4ca0 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.344+0000 7fbac2000700 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d83fdf80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbac406a4c0 tx=0x7fbab0012a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=510e3b51b67b82b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.344+0000 7fbaacc00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.344+0000 7fbaacc00700 1 -- 172.21.15.16:0/4257527351 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0017070 con 0x55a5d83f4ca0 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.344+0000 7fbadd291b80 1 -- 172.21.15.16:0/4257527351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5d83e7490 con 0x55a5d83f4ca0 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.345+0000 7fbaacc00700 1 -- 172.21.15.16:0/4257527351 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fbab0004030 con 0x55a5d83f4ca0 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.345+0000 7fbaacc00700 1 -- 172.21.15.16:0/4257527351 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbab00041f0 con 0x55a5d83f4ca0 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.345+0000 7fbadd291b80 10 client.25622 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.345+0000 7fbadd291b80 1 -- 172.21.15.16:0/4257527351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fbac41a6520 con 0x55a5d83f4ca0 2024-04-05T23:34:18.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.345+0000 7fbaacc00700 1 -- 172.21.15.16:0/4257527351 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a5d83e7490 con 0x55a5d83f4ca0 2024-04-05T23:34:18.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.346+0000 7fbaacc00700 1 -- 172.21.15.16:0/4257527351 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fbab0023070 con 0x55a5d83f4ca0 2024-04-05T23:34:18.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.346+0000 7fbadd291b80 10 client.25622 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:18.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.346+0000 7fbadd291b80 10 client.25622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:18.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.346+0000 7fbadd291b80 20 client.25622 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:18.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.346+0000 7fbadd291b80 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5d8416a60 0x55a5d8418e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.346+0000 7fbadd291b80 4 client.25622 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:18.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.346+0000 7fbadd291b80 1 -- 172.21.15.16:0/4257527351 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a5d833d700 con 0x55a5d8416a60 2024-04-05T23:34:18.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.346+0000 7fbac2a00700 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5d8416a60 0x55a5d8418e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.347+0000 7fbac2a00700 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5d8416a60 0x55a5d8418e80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23d2f690cf27df50 server_cookie=419b4d566f78a6c0 in_seq=0 out_seq=0 2024-04-05T23:34:18.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.347+0000 7fbaacc00700 10 client.25622 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:18.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.347+0000 7fbaacc00700 1 -- 172.21.15.16:0/4257527351 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a5d833d700 con 0x55a5d8416a60 2024-04-05T23:34:18.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.347+0000 7fbaacc00700 10 client.25622 handle_command_reply: tid=0 2024-04-05T23:34:18.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 client.25622 shutdown 2024-04-05T23:34:18.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 -- 172.21.15.16:0/4257527351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 msgr2=0x55a5d83fdf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d83fdf80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbac406a4c0 tx=0x7fbab0012a20 comp rx=0 tx=0).stop 2024-04-05T23:34:18.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 -- 172.21.15.16:0/4257527351 shutdown_connections 2024-04-05T23:34:18.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5d8416a60 0x55a5d8418e80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83fa180 0x55a5d83fe4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83f42e0 0x55a5d8400c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbaacc00700 0 client.25622 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:18.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 --2- 172.21.15.16:0/4257527351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f4ca0 0x55a5d83fdf80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 -- 172.21.15.16:0/4257527351 >> 172.21.15.16:0/4257527351 conn(0x55a5d8368220 msgr2=0x55a5d83e9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 -- 172.21.15.16:0/4257527351 shutdown_connections 2024-04-05T23:34:18.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.348+0000 7fbadd291b80 1 -- 172.21.15.16:0/4257527351 wait complete. 2024-04-05T23:34:18.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.349+0000 7fbadd291b80 20 client.25622 trim_cache size 0 max 16384 2024-04-05T23:34:18.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbadd291b80 1 Processor -- start 2024-04-05T23:34:18.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbadd291b80 1 -- start start 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f42e0 0x55a5d83e9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f4ca0 0x55a5d83ef040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 0x55a5d83ef580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbadd291b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d81c86a0 con 0x55a5d83f4ca0 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbadd291b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d83efac0 con 0x55a5d83f42e0 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbadd291b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d8414140 con 0x55a5d83fa180 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbac2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f42e0 0x55a5d83e9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbac2000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f4ca0 0x55a5d83ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbac2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f42e0 0x55a5d83e9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42804/0 (socket says 172.21.15.16:42804) 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbac2000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f4ca0 0x55a5d83ef040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42958/0 (socket says 172.21.15.16:42958) 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbac2a00700 1 -- 172.21.15.16:0/3162044031 learned_addr learned my addr 172.21.15.16:0/3162044031 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.363+0000 7fbac3400700 1 --2- 172.21.15.16:0/3162044031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 0x55a5d83ef580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.364+0000 7fbac2a00700 1 -- 172.21.15.16:0/3162044031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 msgr2=0x55a5d83ef580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.364+0000 7fbac2a00700 1 --2- 172.21.15.16:0/3162044031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 0x55a5d83ef580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.364+0000 7fbac2a00700 1 -- 172.21.15.16:0/3162044031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f4ca0 msgr2=0x55a5d83ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.364+0000 7fbac2a00700 1 --2- 172.21.15.16:0/3162044031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f4ca0 0x55a5d83ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.364+0000 7fbac2a00700 1 -- 172.21.15.16:0/3162044031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5d8172b60 con 0x55a5d83f42e0 2024-04-05T23:34:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.364+0000 7fbac2a00700 1 --2- 172.21.15.16:0/3162044031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f42e0 0x55a5d83e9560 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbab400f740 tx=0x7fbab40d1cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b80787a90cbdd8c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbab8c00700 1 -- 172.21.15.16:0/3162044031 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab40cb7d0 con 0x55a5d83f42e0 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbab8c00700 1 -- 172.21.15.16:0/3162044031 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbab409ab70 con 0x55a5d83f42e0 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbab8c00700 1 -- 172.21.15.16:0/3162044031 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab409ad50 con 0x55a5d83f42e0 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbadd291b80 1 -- 172.21.15.16:0/3162044031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f42e0 msgr2=0x55a5d83e9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbadd291b80 1 --2- 172.21.15.16:0/3162044031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f42e0 0x55a5d83e9560 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbab400f740 tx=0x7fbab40d1cc0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbadd291b80 1 -- 172.21.15.16:0/3162044031 shutdown_connections 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbadd291b80 1 --2- 172.21.15.16:0/3162044031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83fa180 0x55a5d83ef580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbadd291b80 1 --2- 172.21.15.16:0/3162044031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83f42e0 0x55a5d83e9560 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbadd291b80 1 --2- 172.21.15.16:0/3162044031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f4ca0 0x55a5d83ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.365+0000 7fbadd291b80 1 -- 172.21.15.16:0/3162044031 >> 172.21.15.16:0/3162044031 conn(0x55a5d8368220 msgr2=0x55a5d83fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.366+0000 7fbadd291b80 1 -- 172.21.15.16:0/3162044031 shutdown_connections 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.366+0000 7fbadd291b80 1 -- 172.21.15.16:0/3162044031 wait complete. 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.366+0000 7fbadd291b80 1 -- start start 2024-04-05T23:34:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.366+0000 7fbadd291b80 1 Processor -- start 2024-04-05T23:34:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d840dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83f4ca0 0x55a5d840c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbadd291b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83fa180 0x55a5d840c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbadd291b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d840cf90 con 0x55a5d83f42e0 2024-04-05T23:34:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbadd291b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d840d100 con 0x55a5d83fa180 2024-04-05T23:34:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbadd291b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d846a8f0 con 0x55a5d83f4ca0 2024-04-05T23:34:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbadd291b80 10 client.0 _ll_register_callbacks cb 0x55a5d814c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbac2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d840dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbac2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d840dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42960/0 (socket says 172.21.15.16:42960) 2024-04-05T23:34:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbac2a00700 1 -- 172.21.15.16:0/558693906 learned_addr learned my addr 172.21.15.16:0/558693906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.367+0000 7fbac3400700 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83fa180 0x55a5d840c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.368+0000 7fbac2000700 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83f4ca0 0x55a5d840c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.368+0000 7fbac2a00700 1 -- 172.21.15.16:0/558693906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83f4ca0 msgr2=0x55a5d840c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.368+0000 7fbac2a00700 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83f4ca0 0x55a5d840c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.368+0000 7fbac2a00700 1 -- 172.21.15.16:0/558693906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83fa180 msgr2=0x55a5d840c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.368+0000 7fbac2a00700 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83fa180 0x55a5d840c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.368+0000 7fbac2a00700 1 -- 172.21.15.16:0/558693906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5d81c63b0 con 0x55a5d83f42e0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.368+0000 7fbac2a00700 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d840dd50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbab40092f0 tx=0x7fbab40ce550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291deab3b1c05fdc server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.368+0000 7fbaacc00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.369+0000 7fbaacc00700 1 -- 172.21.15.16:0/558693906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab409ad50 con 0x55a5d83f42e0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.369+0000 7fbaacc00700 1 -- 172.21.15.16:0/558693906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fbab409b910 con 0x55a5d83f42e0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.369+0000 7fbadd291b80 1 -- 172.21.15.16:0/558693906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbab409afe0 con 0x55a5d83f42e0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.369+0000 7fbaacc00700 1 -- 172.21.15.16:0/558693906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbab4097550 con 0x55a5d83f42e0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.369+0000 7fbadd291b80 10 client.15588 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.369+0000 7fbadd291b80 1 -- 172.21.15.16:0/558693906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a5d830aea0 con 0x55a5d83f42e0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.370+0000 7fbaacc00700 1 -- 172.21.15.16:0/558693906 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbab409afe0 con 0x55a5d83f42e0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.370+0000 7fbaacc00700 1 -- 172.21.15.16:0/558693906 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fbab40d6ca0 con 0x55a5d83f42e0 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.370+0000 7fbadd291b80 10 client.15588 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.370+0000 7fbadd291b80 10 client.15588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.370+0000 7fbadd291b80 20 client.15588 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.370+0000 7fbadd291b80 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5d846bbe0 0x55a5d846dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.370+0000 7fbadd291b80 4 client.15588 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.370+0000 7fbadd291b80 1 -- 172.21.15.16:0/558693906 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a5d8193cf0 con 0x55a5d846bbe0 2024-04-05T23:34:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.370+0000 7fbac2000700 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5d846bbe0 0x55a5d846dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.371+0000 7fbaacc00700 10 client.15588 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.371+0000 7fbac2000700 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5d846bbe0 0x55a5d846dfd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c8f392ceba9e323 server_cookie=b0164e35507abdc7 in_seq=0 out_seq=0 2024-04-05T23:34:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.371+0000 7fbaacc00700 1 -- 172.21.15.16:0/558693906 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a5d8193cf0 con 0x55a5d846bbe0 2024-04-05T23:34:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.371+0000 7fbaacc00700 10 client.15588 handle_command_reply: tid=0 2024-04-05T23:34:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.371+0000 7fbadd291b80 1 client.15588 shutdown 2024-04-05T23:34:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.371+0000 7fbadd291b80 1 -- 172.21.15.16:0/558693906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 msgr2=0x55a5d840dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.371+0000 7fbadd291b80 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d840dd50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbab40092f0 tx=0x7fbab40ce550 comp rx=0 tx=0).stop 2024-04-05T23:34:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.372+0000 7fbadd291b80 1 -- 172.21.15.16:0/558693906 shutdown_connections 2024-04-05T23:34:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.372+0000 7fbadd291b80 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5d846bbe0 0x55a5d846dfd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.372+0000 7fbadd291b80 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5d83f42e0 0x55a5d840dd50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.372+0000 7fbaacc00700 0 client.15588 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.372+0000 7fbadd291b80 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5d83fa180 0x55a5d840c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.372+0000 7fbadd291b80 1 --2- 172.21.15.16:0/558693906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d83f4ca0 0x55a5d840c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.372+0000 7fbadd291b80 1 -- 172.21.15.16:0/558693906 >> 172.21.15.16:0/558693906 conn(0x55a5d8368220 msgr2=0x55a5d84123e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.372+0000 7fbadd291b80 1 -- 172.21.15.16:0/558693906 shutdown_connections 2024-04-05T23:34:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.372+0000 7fbadd291b80 1 -- 172.21.15.16:0/558693906 wait complete. 2024-04-05T23:34:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.373+0000 7fbadd291b80 20 client.15588 trim_cache size 0 max 16384 2024-04-05T23:34:18.366 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.373+0000 7fbac9800700 1 -- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbaa80a13b0 msgr2=0x7fbaa80a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.373+0000 7fbac9800700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbaa80a13b0 0x7fbaa80a3870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009710 tx=0x7fbabc009100 comp rx=0 tx=0).stop 2024-04-05T23:34:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.373+0000 7fbac9800700 1 -- 172.21.15.16:0/2522687685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41010a0 msgr2=0x7fbac40731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.373+0000 7fbac9800700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41010a0 0x7fbac40731e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbab4000c00 tx=0x7fbab4005740 comp rx=0 tx=0).stop 2024-04-05T23:34:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.374+0000 7fbac9800700 1 -- 172.21.15.16:0/2522687685 shutdown_connections 2024-04-05T23:34:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.374+0000 7fbac9800700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbaa80a13b0 0x7fbaa80a3870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.374+0000 7fbac9800700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbac4073c60 0x7fbac41a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.374+0000 7fbac9800700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41010a0 0x7fbac40731e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.374+0000 7fbac9800700 1 --2- 172.21.15.16:0/2522687685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac41061c0 0x7fbac4073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:18.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.374+0000 7fbac9800700 1 -- 172.21.15.16:0/2522687685 >> 172.21.15.16:0/2522687685 conn(0x7fbac40fc900 msgr2=0x7fbac40fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:18.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.374+0000 7fbac9800700 1 -- 172.21.15.16:0/2522687685 shutdown_connections 2024-04-05T23:34:18.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:18.374+0000 7fbac9800700 1 -- 172.21.15.16:0/2522687685 wait complete. 2024-04-05T23:34:19.393 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-05T23:34:19.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-05T23:34:19.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.745+0000 7f4414a00700 1 -- 172.21.15.16:0/78056514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 msgr2=0x7f4410073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.745+0000 7f4414a00700 1 --2- 172.21.15.16:0/78056514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 0x7f4410073660 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4400009990 tx=0x7f4400009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.745+0000 7f4414a00700 1 -- 172.21.15.16:0/78056514 shutdown_connections 2024-04-05T23:34:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.745+0000 7f4414a00700 1 --2- 172.21.15.16:0/78056514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4410111c70 0x7f4410114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.745+0000 7f4414a00700 1 --2- 172.21.15.16:0/78056514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 0x7f4410073660 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.745+0000 7f4414a00700 1 --2- 172.21.15.16:0/78056514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4410073ba0 0x7f4410111730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.745+0000 7f4414a00700 1 -- 172.21.15.16:0/78056514 >> 172.21.15.16:0/78056514 conn(0x7f44100fc750 msgr2=0x7f44100feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.745+0000 7f4414a00700 1 -- 172.21.15.16:0/78056514 shutdown_connections 2024-04-05T23:34:19.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.745+0000 7f4414a00700 1 -- 172.21.15.16:0/78056514 wait complete. 2024-04-05T23:34:19.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.747+0000 7f4414a00700 1 Processor -- start 2024-04-05T23:34:19.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.747+0000 7f4414a00700 1 -- start start 2024-04-05T23:34:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f4414a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 0x7f441019d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f4414a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4410073ba0 0x7f441019dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f4414a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4410111c70 0x7f44101a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f4414a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44101a15c0 con 0x7f4410111c70 2024-04-05T23:34:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f4414a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44101a1730 con 0x7f4410073ba0 2024-04-05T23:34:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f4414a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44101a18f0 con 0x7f4410073280 2024-04-05T23:34:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f440f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 0x7f441019d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f440f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 0x7f441019d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40162/0 (socket says 172.21.15.16:40162) 2024-04-05T23:34:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f440f400700 1 -- 172.21.15.16:0/3982160328 learned_addr learned my addr 172.21.15.16:0/3982160328 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f440ea00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4410073ba0 0x7f441019dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.748+0000 7f440fe00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4410111c70 0x7f44101a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.749+0000 7f440f400700 1 -- 172.21.15.16:0/3982160328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4410073ba0 msgr2=0x7f441019dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.749+0000 7f440f400700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4410073ba0 0x7f441019dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.749+0000 7f440f400700 1 -- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4410111c70 msgr2=0x7f44101a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.749+0000 7f440f400700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4410111c70 0x7f44101a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.749+0000 7f440f400700 1 -- 172.21.15.16:0/3982160328 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000095f0 con 0x7f4410073280 2024-04-05T23:34:19.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.749+0000 7f440f400700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 0x7f441019d790 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f440000b6f0 tx=0x7f440000bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:19.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.750+0000 7f4407e00700 1 -- 172.21.15.16:0/3982160328 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440001d070 con 0x7f4410073280 2024-04-05T23:34:19.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.750+0000 7f4407e00700 1 -- 172.21.15.16:0/3982160328 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4400005da0 con 0x7f4410073280 2024-04-05T23:34:19.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.750+0000 7f4407e00700 1 -- 172.21.15.16:0/3982160328 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400004750 con 0x7f4410073280 2024-04-05T23:34:19.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.751+0000 7f4414a00700 1 -- 172.21.15.16:0/3982160328 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44101a1b70 con 0x7f4410073280 2024-04-05T23:34:19.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.751+0000 7f4414a00700 1 -- 172.21.15.16:0/3982160328 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44101aa350 con 0x7f4410073280 2024-04-05T23:34:19.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.756+0000 7f4407e00700 1 -- 172.21.15.16:0/3982160328 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f4400005200 con 0x7f4410073280 2024-04-05T23:34:19.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.758+0000 7f4429bbdb80 1 Processor -- start 2024-04-05T23:34:19.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.758+0000 7f4429bbdb80 1 -- start start 2024-04-05T23:34:19.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.758+0000 7f4407e00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43f40a1430 0x7f43f40a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.758+0000 7f4407e00700 1 -- 172.21.15.16:0/3982160328 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f44000ce0a0 con 0x7f4410073280 2024-04-05T23:34:19.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0350 0x55f1b11f0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0c70 0x55f1b11fb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b11fdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1157c60 con 0x55f1b11fbb50 2024-04-05T23:34:19.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1148a10 con 0x55f1b11f0350 2024-04-05T23:34:19.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b11463d0 con 0x55f1b11f0c70 2024-04-05T23:34:19.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f440fe00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43f40a1430 0x7f43f40a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f440ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0c70 0x55f1b11fb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f440ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0c70 0x55f1b11fb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40164/0 (socket says 172.21.15.16:40164) 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f440fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b11fdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f440ea00700 1 -- 172.21.15.16:0/3974000235 learned_addr learned my addr 172.21.15.16:0/3974000235 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.759+0000 7f440f400700 1 --2- 172.21.15.16:0/3974000235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0350 0x55f1b11f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.760+0000 7f440f400700 1 -- 172.21.15.16:0/3974000235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0c70 msgr2=0x55f1b11fb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.760+0000 7f440f400700 1 --2- 172.21.15.16:0/3974000235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0c70 0x55f1b11fb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.760+0000 7f440f400700 1 -- 172.21.15.16:0/3974000235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 msgr2=0x55f1b11fdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.760+0000 7f440f400700 1 --2- 172.21.15.16:0/3974000235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b11fdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.760+0000 7f440f400700 1 -- 172.21.15.16:0/3974000235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1b0fc3430 con 0x55f1b11f0350 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.760+0000 7f440fe00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43f40a1430 0x7f43f40a38f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f441006a3b0 tx=0x7f440800fc00 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.760+0000 7f440f400700 1 --2- 172.21.15.16:0/3974000235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0350 0x55f1b11f0730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f44000cb640 tx=0x7f4400005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4a3530ecab975d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:19.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.761+0000 7f43fbe00700 1 -- 172.21.15.16:0/3974000235 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440009afc0 con 0x55f1b11f0350 2024-04-05T23:34:19.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.761+0000 7f43fbe00700 1 -- 172.21.15.16:0/3974000235 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f440009edf0 con 0x55f1b11f0350 2024-04-05T23:34:19.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.761+0000 7f43fbe00700 1 -- 172.21.15.16:0/3974000235 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400095e30 con 0x55f1b11f0350 2024-04-05T23:34:19.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.762+0000 7f4429bbdb80 1 -- 172.21.15.16:0/3974000235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0350 msgr2=0x55f1b11f0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:19.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.762+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/3974000235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0350 0x55f1b11f0730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f44000cb640 tx=0x7f4400005f10 comp rx=0 tx=0).stop 2024-04-05T23:34:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.762+0000 7f4429bbdb80 1 -- 172.21.15.16:0/3974000235 shutdown_connections 2024-04-05T23:34:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.762+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/3974000235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b11fdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.762+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/3974000235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0350 0x55f1b11f0730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.762+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/3974000235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0c70 0x55f1b11fb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.762+0000 7f4429bbdb80 1 -- 172.21.15.16:0/3974000235 >> 172.21.15.16:0/3974000235 conn(0x55f1b11652a0 msgr2=0x55f1b11ec850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.762+0000 7f4429bbdb80 1 -- 172.21.15.16:0/3974000235 shutdown_connections 2024-04-05T23:34:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.762+0000 7f4429bbdb80 1 -- 172.21.15.16:0/3974000235 wait complete. 2024-04-05T23:34:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.763+0000 7f4429bbdb80 1 -- start start 2024-04-05T23:34:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.763+0000 7f4429bbdb80 1 Processor -- start 2024-04-05T23:34:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.764+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.764+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11f0c70 0x55f1b11fc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.764+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11fbb50 0x55f1b11fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.764+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1141cc0 con 0x55f1b11f0c70 2024-04-05T23:34:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.764+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b10dde00 con 0x55f1b11fbb50 2024-04-05T23:34:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.764+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b10ddaa0 con 0x55f1b11f0350 2024-04-05T23:34:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.764+0000 7f4429bbdb80 10 client.0 _ll_register_callbacks cb 0x55f1b0f49420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11fbb50 0x55f1b11fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11fbb50 0x55f1b11fc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42858/0 (socket says 172.21.15.16:42858) 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440fe00700 1 -- 172.21.15.16:0/4134915734 learned_addr learned my addr 172.21.15.16:0/4134915734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440f400700 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440ea00700 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11f0c70 0x55f1b11fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440fe00700 1 -- 172.21.15.16:0/4134915734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 msgr2=0x55f1b11fbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440fe00700 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440fe00700 1 -- 172.21.15.16:0/4134915734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11f0c70 msgr2=0x55f1b11fc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440fe00700 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11f0c70 0x55f1b11fc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.765+0000 7f440fe00700 1 -- 172.21.15.16:0/4134915734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44101aa350 con 0x55f1b11fbb50 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.766+0000 7f440fe00700 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11fbb50 0x55f1b11fc9b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4408018850 tx=0x7f4408004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deab2ad9f46b65e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.766+0000 7f43f8c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.766+0000 7f43f8c00700 1 -- 172.21.15.16:0/4134915734 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440800e040 con 0x55f1b11fbb50 2024-04-05T23:34:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.766+0000 7f4429bbdb80 1 -- 172.21.15.16:0/4134915734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1b11e4510 con 0x55f1b11fbb50 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.766+0000 7f43f8c00700 1 -- 172.21.15.16:0/4134915734 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f44080089e0 con 0x55f1b11fbb50 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f43f8c00700 1 -- 172.21.15.16:0/4134915734 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4408008ba0 con 0x55f1b11fbb50 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f4429bbdb80 10 client.25640 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f4429bbdb80 1 -- 172.21.15.16:0/4134915734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f1b0f6fbe0 con 0x55f1b11fbb50 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f43f8c00700 1 -- 172.21.15.16:0/4134915734 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f1b11e4510 con 0x55f1b11fbb50 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f43f8c00700 1 -- 172.21.15.16:0/4134915734 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f440801a440 con 0x55f1b11fbb50 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f4429bbdb80 10 client.25640 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f4429bbdb80 10 client.25640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f4429bbdb80 20 client.25640 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f1b120b280 0x55f1b120d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f4429bbdb80 4 client.25640 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.767+0000 7f4429bbdb80 1 -- 172.21.15.16:0/4134915734 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1b113a780 con 0x55f1b120b280 2024-04-05T23:34:19.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.768+0000 7f440f400700 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f1b120b280 0x55f1b120d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.768+0000 7f440f400700 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f1b120b280 0x55f1b120d6a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62567b69ae28561c server_cookie=bf9ed9102d010204 in_seq=0 out_seq=0 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.768+0000 7f43f8c00700 10 client.25640 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.769+0000 7f43f8c00700 1 -- 172.21.15.16:0/4134915734 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f1b113a780 con 0x55f1b120b280 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.769+0000 7f43f8c00700 10 client.25640 handle_command_reply: tid=0 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.769+0000 7f4429bbdb80 1 client.25640 shutdown 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.769+0000 7f4429bbdb80 1 -- 172.21.15.16:0/4134915734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11fbb50 msgr2=0x55f1b11fc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.769+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11fbb50 0x55f1b11fc9b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4408018850 tx=0x7f4408004b40 comp rx=0 tx=0).stop 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f4429bbdb80 1 -- 172.21.15.16:0/4134915734 shutdown_connections 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f1b120b280 0x55f1b120d6a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11fbb50 0x55f1b11fc9b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f43f8c00700 0 client.25640 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/4134915734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11f0c70 0x55f1b11fc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f4429bbdb80 1 -- 172.21.15.16:0/4134915734 >> 172.21.15.16:0/4134915734 conn(0x55f1b11652a0 msgr2=0x55f1b11ec440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f4429bbdb80 1 -- 172.21.15.16:0/4134915734 shutdown_connections 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f4429bbdb80 1 -- 172.21.15.16:0/4134915734 wait complete. 2024-04-05T23:34:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.770+0000 7f4429bbdb80 20 client.25640 trim_cache size 0 max 16384 2024-04-05T23:34:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.805+0000 7f4429bbdb80 1 Processor -- start 2024-04-05T23:34:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.805+0000 7f4429bbdb80 1 -- start start 2024-04-05T23:34:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.805+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.805+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 0x55f1b11f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.805+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b11f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.805+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b0fc5720 con 0x55f1b11fbb50 2024-04-05T23:34:19.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.805+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1207080 con 0x55f1b11f0c70 2024-04-05T23:34:19.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.805+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b11f15c0 con 0x55f1b11f0350 2024-04-05T23:34:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.806+0000 7f440f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.806+0000 7f440f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40184/0 (socket says 172.21.15.16:40184) 2024-04-05T23:34:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.806+0000 7f440f400700 1 -- 172.21.15.16:0/368435914 learned_addr learned my addr 172.21.15.16:0/368435914 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.806+0000 7f440ea00700 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 0x55f1b11f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.806+0000 7f440fe00700 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b11f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f440f400700 1 -- 172.21.15.16:0/368435914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 msgr2=0x55f1b11f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f440f400700 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 0x55f1b11f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f440f400700 1 -- 172.21.15.16:0/368435914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 msgr2=0x55f1b11f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f440f400700 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b11f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f440f400700 1 -- 172.21.15.16:0/368435914 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1b1107f20 con 0x55f1b11f0350 2024-04-05T23:34:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f440ea00700 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 0x55f1b11f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f440fe00700 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b11f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f440f400700 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fff60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f440009acc0 tx=0x7f4400009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ba9382436a6e07 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f4405200700 1 -- 172.21.15.16:0/368435914 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440009afc0 con 0x55f1b11f0350 2024-04-05T23:34:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f4405200700 1 -- 172.21.15.16:0/368435914 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f440009a4b0 con 0x55f1b11f0350 2024-04-05T23:34:19.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.807+0000 7f4405200700 1 -- 172.21.15.16:0/368435914 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44000a2f20 con 0x55f1b11f0350 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.808+0000 7f4429bbdb80 1 -- 172.21.15.16:0/368435914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 msgr2=0x55f1b11fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.808+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fff60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f440009acc0 tx=0x7f4400009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.808+0000 7f4429bbdb80 1 -- 172.21.15.16:0/368435914 shutdown_connections 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.808+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b11f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.808+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b11fff60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.808+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/368435914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 0x55f1b11f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.808+0000 7f4429bbdb80 1 -- 172.21.15.16:0/368435914 >> 172.21.15.16:0/368435914 conn(0x55f1b11652a0 msgr2=0x55f1b11eb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.808+0000 7f4429bbdb80 1 -- 172.21.15.16:0/368435914 shutdown_connections 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.809+0000 7f4429bbdb80 1 -- 172.21.15.16:0/368435914 wait complete. 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.809+0000 7f4429bbdb80 1 -- start start 2024-04-05T23:34:19.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.809+0000 7f4429bbdb80 1 Processor -- start 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.810+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b1258d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.810+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 0x55f1b1259280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.810+0000 7f4429bbdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b12531a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.810+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1253780 con 0x55f1b11fbb50 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.810+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b12538f0 con 0x55f1b11f0c70 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.810+0000 7f4429bbdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1253ab0 con 0x55f1b11f0350 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.810+0000 7f4429bbdb80 10 client.0 _ll_register_callbacks cb 0x7f44080090f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b1258d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b12531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 0x55f1b1259280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b1258d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40198/0 (socket says 172.21.15.16:40198) 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440f400700 1 -- 172.21.15.16:0/2146781612 learned_addr learned my addr 172.21.15.16:0/2146781612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440f400700 1 -- 172.21.15.16:0/2146781612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 msgr2=0x55f1b1259280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440f400700 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 0x55f1b1259280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440f400700 1 -- 172.21.15.16:0/2146781612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 msgr2=0x55f1b12531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440f400700 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b12531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440f400700 1 -- 172.21.15.16:0/2146781612 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1b0fc3430 con 0x55f1b11f0350 2024-04-05T23:34:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f440f400700 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b1258d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4400005b40 tx=0x7f44000cb780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff085499fbb71a40 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.811+0000 7f43f8c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:19.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.812+0000 7f43f8c00700 1 -- 172.21.15.16:0/2146781612 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44000b9340 con 0x55f1b11f0350 2024-04-05T23:34:19.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.812+0000 7f43f8c00700 1 -- 172.21.15.16:0/2146781612 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f4400095bd0 con 0x55f1b11f0350 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.812+0000 7f4429bbdb80 1 -- 172.21.15.16:0/2146781612 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f440009edf0 con 0x55f1b11f0350 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.812+0000 7f43f8c00700 1 -- 172.21.15.16:0/2146781612 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4400095d90 con 0x55f1b11f0350 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.812+0000 7f4429bbdb80 10 client.25669 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.812+0000 7f4429bbdb80 1 -- 172.21.15.16:0/2146781612 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f1b0fc7d50 con 0x55f1b11f0350 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.812+0000 7f43f8c00700 1 -- 172.21.15.16:0/2146781612 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f440009edf0 con 0x55f1b11f0350 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.812+0000 7f43f8c00700 1 -- 172.21.15.16:0/2146781612 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f44000b9060 con 0x55f1b11f0350 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.813+0000 7f4429bbdb80 10 client.25669 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.813+0000 7f4429bbdb80 10 client.25669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.813+0000 7f4429bbdb80 20 client.25669 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.813+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f1b1260710 0x55f1b1262b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.813+0000 7f4429bbdb80 4 client.25669 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-05T23:34:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.813+0000 7f4429bbdb80 1 -- 172.21.15.16:0/2146781612 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f1b0f90d70 con 0x55f1b1260710 2024-04-05T23:34:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.813+0000 7f440ea00700 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f1b1260710 0x55f1b1262b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.814+0000 7f440ea00700 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f1b1260710 0x55f1b1262b00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a841534b10842e75 server_cookie=80db34bf6b5d4bed in_seq=0 out_seq=0 2024-04-05T23:34:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.814+0000 7f43f8c00700 10 client.25669 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f43f8c00700 1 -- 172.21.15.16:0/2146781612 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f1b0f90d70 con 0x55f1b1260710 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f43f8c00700 10 client.25669 handle_command_reply: tid=0 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f4429bbdb80 1 client.25669 shutdown 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f4429bbdb80 1 -- 172.21.15.16:0/2146781612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 msgr2=0x55f1b1258d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b1258d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4400005b40 tx=0x7f44000cb780 comp rx=0 tx=0).stop 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f4429bbdb80 1 -- 172.21.15.16:0/2146781612 shutdown_connections 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f1b1260710 0x55f1b1262b00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b11fbb50 0x55f1b12531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f1b11f0350 0x55f1b1258d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f4429bbdb80 1 --2- 172.21.15.16:0/2146781612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1b11f0c70 0x55f1b1259280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f43f8c00700 0 client.25669 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.815+0000 7f4429bbdb80 1 -- 172.21.15.16:0/2146781612 >> 172.21.15.16:0/2146781612 conn(0x55f1b11652a0 msgr2=0x55f1b11f5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.816+0000 7f4429bbdb80 1 -- 172.21.15.16:0/2146781612 shutdown_connections 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.816+0000 7f4429bbdb80 1 -- 172.21.15.16:0/2146781612 wait complete. 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.816+0000 7f4429bbdb80 20 client.25669 trim_cache size 0 max 16384 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "4b10b23f-761f-4400-a75c-70fa6aafe81d", 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-04-05T23:34:19.809 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:34:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 -- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43f40a1430 msgr2=0x7f43f40a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43f40a1430 0x7f43f40a38f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f441006a3b0 tx=0x7f440800fc00 comp rx=0 tx=0).stop 2024-04-05T23:34:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 -- 172.21.15.16:0/3982160328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 msgr2=0x7f441019d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 0x7f441019d790 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f440000b6f0 tx=0x7f440000bfd0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 -- 172.21.15.16:0/3982160328 shutdown_connections 2024-04-05T23:34:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43f40a1430 0x7f43f40a38f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4410111c70 0x7f44101a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4410073280 0x7f441019d790 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 --2- 172.21.15.16:0/3982160328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4410073ba0 0x7f441019dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.817+0000 7f4414a00700 1 -- 172.21.15.16:0/3982160328 >> 172.21.15.16:0/3982160328 conn(0x7f44100fc750 msgr2=0x7f44100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.818+0000 7f4414a00700 1 -- 172.21.15.16:0/3982160328 shutdown_connections 2024-04-05T23:34:19.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:19.818+0000 7f4414a00700 1 -- 172.21.15.16:0/3982160328 wait complete. 2024-04-05T23:34:19.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:19 smithi016 ceph-mon[38248]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-04-05T23:34:19.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-05T23:34:19.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:19 smithi027 ceph-mon[78597]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-04-05T23:34:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.074+0000 7fa624a00700 1 -- 172.21.15.16:0/422396700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa620101eb0 msgr2=0x7fa620102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.074+0000 7fa624a00700 1 --2- 172.21.15.16:0/422396700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa620101eb0 0x7fa620102290 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa610009990 tx=0x7fa610009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.074+0000 7fa624a00700 1 -- 172.21.15.16:0/422396700 shutdown_connections 2024-04-05T23:34:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.074+0000 7fa624a00700 1 --2- 172.21.15.16:0/422396700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62010b2d0 0x7fa62010d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.074+0000 7fa624a00700 1 --2- 172.21.15.16:0/422396700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa620101eb0 0x7fa620102290 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.074+0000 7fa624a00700 1 --2- 172.21.15.16:0/422396700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6201027d0 0x7fa62010acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.074+0000 7fa624a00700 1 -- 172.21.15.16:0/422396700 >> 172.21.15.16:0/422396700 conn(0x7fa6200fb6b0 msgr2=0x7fa6200fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.075+0000 7fa624a00700 1 -- 172.21.15.16:0/422396700 shutdown_connections 2024-04-05T23:34:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.075+0000 7fa624a00700 1 -- 172.21.15.16:0/422396700 wait complete. 2024-04-05T23:34:20.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.076+0000 7fa624a00700 1 Processor -- start 2024-04-05T23:34:20.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.076+0000 7fa624a00700 1 -- start start 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa624a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101eb0 0x7fa620199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa624a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6201027d0 0x7fa620199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa624a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62010b2d0 0x7fa62019dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa624a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62019e2f0 con 0x7fa620101eb0 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa624a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62019e460 con 0x7fa62010b2d0 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa624a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62019e620 con 0x7fa6201027d0 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa61e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101eb0 0x7fa620199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa61e800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101eb0 0x7fa620199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43066/0 (socket says 172.21.15.16:43066) 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa61e800700 1 -- 172.21.15.16:0/2284894390 learned_addr learned my addr 172.21.15.16:0/2284894390 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa617e00700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6201027d0 0x7fa620199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.077+0000 7fa61f200700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62010b2d0 0x7fa62019dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.078+0000 7fa61e800700 1 -- 172.21.15.16:0/2284894390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6201027d0 msgr2=0x7fa620199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.078+0000 7fa61e800700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6201027d0 0x7fa620199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.078+0000 7fa61e800700 1 -- 172.21.15.16:0/2284894390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62010b2d0 msgr2=0x7fa62019dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.078+0000 7fa61e800700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62010b2d0 0x7fa62019dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.078+0000 7fa61e800700 1 -- 172.21.15.16:0/2284894390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6100095f0 con 0x7fa620101eb0 2024-04-05T23:34:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.078+0000 7fa61f200700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62010b2d0 0x7fa62019dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.078+0000 7fa61e800700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101eb0 0x7fa620199440 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa610005910 tx=0x7fa610004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.078+0000 7fa617400700 1 -- 172.21.15.16:0/2284894390 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61001d070 con 0x7fa620101eb0 2024-04-05T23:34:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.079+0000 7fa617400700 1 -- 172.21.15.16:0/2284894390 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa610022470 con 0x7fa620101eb0 2024-04-05T23:34:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.079+0000 7fa617400700 1 -- 172.21.15.16:0/2284894390 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610017650 con 0x7fa620101eb0 2024-04-05T23:34:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.079+0000 7fa624a00700 1 -- 172.21.15.16:0/2284894390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62019e8a0 con 0x7fa620101eb0 2024-04-05T23:34:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.080+0000 7fa624a00700 1 -- 172.21.15.16:0/2284894390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6201a5f60 con 0x7fa620101eb0 2024-04-05T23:34:20.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.084+0000 7fa617400700 1 -- 172.21.15.16:0/2284894390 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7fa6100225e0 con 0x7fa620101eb0 2024-04-05T23:34:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.085+0000 7fa617400700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa6040a1430 0x7fa6040a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.085+0000 7fa617400700 1 -- 172.21.15.16:0/2284894390 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa6100cdd50 con 0x7fa620101eb0 2024-04-05T23:34:20.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.086+0000 7fa617e00700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa6040a1430 0x7fa6040a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.086+0000 7fa638f99b80 1 Processor -- start 2024-04-05T23:34:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.087+0000 7fa617e00700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa6040a1430 0x7fa6040a38f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6200fdb80 tx=0x7fa60800b360 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:20.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.087+0000 7fa638f99b80 1 -- start start 2024-04-05T23:34:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.087+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6f86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.087+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e703590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.087+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 0x55ee4e705ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.087+0000 7fa638f99b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e65fbe0 con 0x55ee4e6f82d0 2024-04-05T23:34:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.087+0000 7fa638f99b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e650990 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.087+0000 7fa638f99b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e64e350 con 0x55ee4e703ad0 2024-04-05T23:34:20.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.087+0000 7fa617e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e703590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa617e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e703590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42906/0 (socket says 172.21.15.16:42906) 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa617e00700 1 -- 172.21.15.16:0/1668231302 learned_addr learned my addr 172.21.15.16:0/1668231302 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa61e800700 1 --2- 172.21.15.16:0/1668231302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6f86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa61f200700 1 --2- 172.21.15.16:0/1668231302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 0x55ee4e705ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa617e00700 1 -- 172.21.15.16:0/1668231302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 msgr2=0x55ee4e705ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa617e00700 1 --2- 172.21.15.16:0/1668231302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 0x55ee4e705ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa617e00700 1 -- 172.21.15.16:0/1668231302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f82d0 msgr2=0x55ee4e6f86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa617e00700 1 --2- 172.21.15.16:0/1668231302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6f86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa617e00700 1 -- 172.21.15.16:0/1668231302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee4e4cb3b0 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.088+0000 7fa617e00700 1 --2- 172.21.15.16:0/1668231302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e703590 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa608010f70 tx=0x7fa608013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a32eaff611416d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:20.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.089+0000 7fa60f400700 1 -- 172.21.15.16:0/1668231302 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa608011db0 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.089+0000 7fa60f400700 1 -- 172.21.15.16:0/1668231302 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa608015070 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.089+0000 7fa60f400700 1 -- 172.21.15.16:0/1668231302 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60801c610 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.090+0000 7fa638f99b80 1 -- 172.21.15.16:0/1668231302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f8bf0 msgr2=0x55ee4e703590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:20.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.090+0000 7fa638f99b80 1 --2- 172.21.15.16:0/1668231302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e703590 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa608010f70 tx=0x7fa608013000 comp rx=0 tx=0).stop 2024-04-05T23:34:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.090+0000 7fa638f99b80 1 -- 172.21.15.16:0/1668231302 shutdown_connections 2024-04-05T23:34:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.090+0000 7fa638f99b80 1 --2- 172.21.15.16:0/1668231302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6f86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.090+0000 7fa638f99b80 1 --2- 172.21.15.16:0/1668231302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 0x55ee4e705ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.090+0000 7fa638f99b80 1 --2- 172.21.15.16:0/1668231302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e703590 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.090+0000 7fa638f99b80 1 -- 172.21.15.16:0/1668231302 >> 172.21.15.16:0/1668231302 conn(0x55ee4e66d220 msgr2=0x55ee4e6f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.090+0000 7fa638f99b80 1 -- 172.21.15.16:0/1668231302 shutdown_connections 2024-04-05T23:34:20.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.090+0000 7fa638f99b80 1 -- 172.21.15.16:0/1668231302 wait complete. 2024-04-05T23:34:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.091+0000 7fa638f99b80 1 -- start start 2024-04-05T23:34:20.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.091+0000 7fa638f99b80 1 Processor -- start 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e7034e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e7007d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 0x55ee4e700e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa638f99b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e649c40 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa638f99b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e5e5d80 con 0x55ee4e6f82d0 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa638f99b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e5e5a20 con 0x55ee4e703ad0 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa638f99b80 10 client.0 _ll_register_callbacks cb 0x55ee4e4513a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa61e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e7034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa61e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e7034e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42910/0 (socket says 172.21.15.16:42910) 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa61e800700 1 -- 172.21.15.16:0/4005218653 learned_addr learned my addr 172.21.15.16:0/4005218653 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:20.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.092+0000 7fa61f200700 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 0x55ee4e700e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.085 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:19 smithi130 ceph-mon[79417]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-04-05T23:34:20.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa617e00700 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e7007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa61e800700 1 -- 172.21.15.16:0/4005218653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 msgr2=0x55ee4e700e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:20.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa61e800700 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 0x55ee4e700e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa61e800700 1 -- 172.21.15.16:0/4005218653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f8bf0 msgr2=0x55ee4e7007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:20.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa61e800700 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e7007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa61e800700 1 -- 172.21.15.16:0/4005218653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa608010c20 con 0x55ee4e6f82d0 2024-04-05T23:34:20.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa617e00700 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e7007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa61e800700 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e7034e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa610005cb0 tx=0x7fa61009a7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa6d7503c6875984 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa603e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa603e00700 1 -- 172.21.15.16:0/4005218653 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61009ac70 con 0x55ee4e6f82d0 2024-04-05T23:34:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa603e00700 1 -- 172.21.15.16:0/4005218653 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa61009eaa0 con 0x55ee4e6f82d0 2024-04-05T23:34:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.093+0000 7fa638f99b80 1 -- 172.21.15.16:0/4005218653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee4e6ec490 con 0x55ee4e6f82d0 2024-04-05T23:34:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.094+0000 7fa603e00700 1 -- 172.21.15.16:0/4005218653 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6100a3060 con 0x55ee4e6f82d0 2024-04-05T23:34:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.094+0000 7fa638f99b80 10 client.25652 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.094+0000 7fa638f99b80 1 -- 172.21.15.16:0/4005218653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ee4e477b60 con 0x55ee4e6f82d0 2024-04-05T23:34:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.094+0000 7fa603e00700 1 -- 172.21.15.16:0/4005218653 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ee4e6ec490 con 0x55ee4e6f82d0 2024-04-05T23:34:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.095+0000 7fa603e00700 1 -- 172.21.15.16:0/4005218653 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fa6100a2cf0 con 0x55ee4e6f82d0 2024-04-05T23:34:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.095+0000 7fa638f99b80 10 client.25652 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.095+0000 7fa638f99b80 10 client.25652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.095+0000 7fa638f99b80 20 client.25652 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.096+0000 7fa638f99b80 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ee4e713230 0x55ee4e715650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.096+0000 7fa638f99b80 4 client.25652 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.096+0000 7fa638f99b80 1 -- 172.21.15.16:0/4005218653 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee4e642700 con 0x55ee4e713230 2024-04-05T23:34:20.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.096+0000 7fa617e00700 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ee4e713230 0x55ee4e715650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.097+0000 7fa617e00700 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ee4e713230 0x55ee4e715650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c9b1d93a4f9a7c7 server_cookie=9a935d6e176e0b6 in_seq=0 out_seq=0 2024-04-05T23:34:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.097+0000 7fa603e00700 10 client.25652 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.097+0000 7fa603e00700 1 -- 172.21.15.16:0/4005218653 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ee4e642700 con 0x55ee4e713230 2024-04-05T23:34:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.097+0000 7fa603e00700 10 client.25652 handle_command_reply: tid=0 2024-04-05T23:34:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.097+0000 7fa638f99b80 1 client.25652 shutdown 2024-04-05T23:34:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 -- 172.21.15.16:0/4005218653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 msgr2=0x55ee4e7034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e7034e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa610005cb0 tx=0x7fa61009a7b0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 -- 172.21.15.16:0/4005218653 shutdown_connections 2024-04-05T23:34:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ee4e713230 0x55ee4e715650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa603e00700 0 client.25652 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e703ad0 0x55ee4e700e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e7034e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 --2- 172.21.15.16:0/4005218653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e7007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 -- 172.21.15.16:0/4005218653 >> 172.21.15.16:0/4005218653 conn(0x55ee4e66d220 msgr2=0x55ee4e6f43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 -- 172.21.15.16:0/4005218653 shutdown_connections 2024-04-05T23:34:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.098+0000 7fa638f99b80 1 -- 172.21.15.16:0/4005218653 wait complete. 2024-04-05T23:34:20.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.099+0000 7fa638f99b80 20 client.25652 trim_cache size 0 max 16384 2024-04-05T23:34:20.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.117+0000 7fa638f99b80 1 Processor -- start 2024-04-05T23:34:20.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.117+0000 7fa638f99b80 1 -- start start 2024-04-05T23:34:20.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.117+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.117+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 0x55ee4e6f4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa638f99b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e4cd6a0 con 0x55ee4e703ad0 2024-04-05T23:34:20.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa638f99b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e6fee30 con 0x55ee4e6f82d0 2024-04-05T23:34:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa638f99b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e70ac80 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa61e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa61f200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 0x55ee4e6f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa61e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42920/0 (socket says 172.21.15.16:42920) 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa61e800700 1 -- 172.21.15.16:0/918139782 learned_addr learned my addr 172.21.15.16:0/918139782 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa617e00700 1 --2- 172.21.15.16:0/918139782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa61e800700 1 -- 172.21.15.16:0/918139782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 msgr2=0x55ee4e70a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa61e800700 1 --2- 172.21.15.16:0/918139782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa61e800700 1 -- 172.21.15.16:0/918139782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 msgr2=0x55ee4e6f4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa61e800700 1 --2- 172.21.15.16:0/918139782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 0x55ee4e6f4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.118+0000 7fa61e800700 1 -- 172.21.15.16:0/918139782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee4e60fea0 con 0x55ee4e6f82d0 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.119+0000 7fa61e800700 1 --2- 172.21.15.16:0/918139782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6fb1b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa610097690 tx=0x7fa6100094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10a9e2e340365302 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.119+0000 7fa60fe00700 1 -- 172.21.15.16:0/918139782 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61009ac70 con 0x55ee4e6f82d0 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.119+0000 7fa60fe00700 1 -- 172.21.15.16:0/918139782 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6100cb230 con 0x55ee4e6f82d0 2024-04-05T23:34:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.119+0000 7fa60fe00700 1 -- 172.21.15.16:0/918139782 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6100cbe20 con 0x55ee4e6f82d0 2024-04-05T23:34:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.120+0000 7fa638f99b80 1 -- 172.21.15.16:0/918139782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 msgr2=0x55ee4e6fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.120+0000 7fa638f99b80 1 --2- 172.21.15.16:0/918139782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6fb1b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa610097690 tx=0x7fa6100094b0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.120+0000 7fa638f99b80 1 -- 172.21.15.16:0/918139782 shutdown_connections 2024-04-05T23:34:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.120+0000 7fa638f99b80 1 --2- 172.21.15.16:0/918139782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 0x55ee4e6f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.120+0000 7fa638f99b80 1 --2- 172.21.15.16:0/918139782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e6fb1b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.120+0000 7fa638f99b80 1 --2- 172.21.15.16:0/918139782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.120+0000 7fa638f99b80 1 -- 172.21.15.16:0/918139782 >> 172.21.15.16:0/918139782 conn(0x55ee4e66d220 msgr2=0x55ee4e6f2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.120+0000 7fa638f99b80 1 -- 172.21.15.16:0/918139782 shutdown_connections 2024-04-05T23:34:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.120+0000 7fa638f99b80 1 -- 172.21.15.16:0/918139782 wait complete. 2024-04-05T23:34:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.121+0000 7fa638f99b80 1 -- start start 2024-04-05T23:34:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.121+0000 7fa638f99b80 1 Processor -- start 2024-04-05T23:34:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.122+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e709e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.122+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.122+0000 7fa638f99b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 0x55ee4e7086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa61e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e709e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa61e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e709e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42924/0 (socket says 172.21.15.16:42924) 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa617e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa617e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:40236/0 (socket says 172.21.15.16:40236) 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa61e800700 1 -- 172.21.15.16:0/2238928339 learned_addr learned my addr 172.21.15.16:0/2238928339 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e708d80 con 0x55ee4e703ad0 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa61f200700 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 0x55ee4e7086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e708ef0 con 0x55ee4e6f82d0 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e7090b0 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.123+0000 7fa638f99b80 10 client.0 _ll_register_callbacks cb 0x55ee4e4513a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.124+0000 7fa617e00700 1 -- 172.21.15.16:0/2238928339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 msgr2=0x55ee4e709e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.124+0000 7fa617e00700 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e709e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.124+0000 7fa617e00700 1 -- 172.21.15.16:0/2238928339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 msgr2=0x55ee4e7086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.124+0000 7fa617e00700 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 0x55ee4e7086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.124+0000 7fa617e00700 1 -- 172.21.15.16:0/2238928339 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee4e477b60 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.124+0000 7fa617e00700 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a3b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa60801ed20 tx=0x7fa60800be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16b1247334de0a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.124+0000 7fa603e00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.125+0000 7fa603e00700 1 -- 172.21.15.16:0/2238928339 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60802c040 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.125+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa608015070 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.125+0000 7fa603e00700 1 -- 172.21.15.16:0/2238928339 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa608017070 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.125+0000 7fa603e00700 1 -- 172.21.15.16:0/2238928339 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa608031580 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.125+0000 7fa638f99b80 10 client.25687 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.125+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ee4e4cfcd0 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.125+0000 7fa603e00700 1 -- 172.21.15.16:0/2238928339 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa608015070 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.126+0000 7fa603e00700 1 -- 172.21.15.16:0/2238928339 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fa608040070 con 0x55ee4e6f8bf0 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.126+0000 7fa638f99b80 10 client.25687 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.126+0000 7fa638f99b80 10 client.25687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.126+0000 7fa638f99b80 20 client.25687 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.126+0000 7fa638f99b80 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ee4e713230 0x55ee4e76a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.126+0000 7fa638f99b80 4 client.25687 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.126+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ee4e498cf0 con 0x55ee4e713230 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.127+0000 7fa61e800700 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ee4e713230 0x55ee4e76a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.127+0000 7fa61e800700 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ee4e713230 0x55ee4e76a150 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4ed782c51274fbc server_cookie=f4f4b2fc8656bfe3 in_seq=0 out_seq=0 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.127+0000 7fa603e00700 10 client.25687 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.127+0000 7fa603e00700 1 -- 172.21.15.16:0/2238928339 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ee4e498cf0 con 0x55ee4e713230 2024-04-05T23:34:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.127+0000 7fa603e00700 10 client.25687 handle_command_reply: tid=0 2024-04-05T23:34:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.127+0000 7fa638f99b80 1 client.25687 shutdown 2024-04-05T23:34:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 msgr2=0x55ee4e70a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a3b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa60801ed20 tx=0x7fa60800be50 comp rx=0 tx=0).stop 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 shutdown_connections 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ee4e713230 0x55ee4e76a150 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ee4e703ad0 0x55ee4e7086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6f82d0 0x55ee4e709e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 --2- 172.21.15.16:0/2238928339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee4e6f8bf0 0x55ee4e70a3b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 >> 172.21.15.16:0/2238928339 conn(0x55ee4e66d220 msgr2=0x55ee4e6f4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa603e00700 0 client.25687 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 shutdown_connections 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 1 -- 172.21.15.16:0/2238928339 wait complete. 2024-04-05T23:34:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.128+0000 7fa638f99b80 20 client.25687 trim_cache size 0 max 16384 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout: "4b10b23f-761f-4400-a75c-70fa6aafe81d": { 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "4b10b23f-761f-4400-a75c-70fa6aafe81d", 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:34:20.122 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:34:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 -- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa6040a1430 msgr2=0x7fa6040a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa6040a1430 0x7fa6040a38f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6200fdb80 tx=0x7fa60800b360 comp rx=0 tx=0).stop 2024-04-05T23:34:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 -- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101eb0 msgr2=0x7fa620199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101eb0 0x7fa620199440 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa610005910 tx=0x7fa610004730 comp rx=0 tx=0).stop 2024-04-05T23:34:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 -- 172.21.15.16:0/2284894390 shutdown_connections 2024-04-05T23:34:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa6040a1430 0x7fa6040a38f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa620101eb0 0x7fa620199440 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62010b2d0 0x7fa62019dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 --2- 172.21.15.16:0/2284894390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6201027d0 0x7fa620199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 -- 172.21.15.16:0/2284894390 >> 172.21.15.16:0/2284894390 conn(0x7fa6200fb6b0 msgr2=0x7fa620100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 -- 172.21.15.16:0/2284894390 shutdown_connections 2024-04-05T23:34:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:20.129+0000 7fa624a00700 1 -- 172.21.15.16:0/2284894390 wait complete. 2024-04-05T23:34:20.146 INFO:tasks.cephfs.filesystem:scrub status for tag:4b10b23f-761f-4400-a75c-70fa6aafe81d - {'path': '/', 'tag': '4b10b23f-761f-4400-a75c-70fa6aafe81d', 'options': 'recursive,force'} 2024-04-05T23:34:20.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:20 smithi027 ceph-mon[78597]: scrub queued for path: / 2024-04-05T23:34:20.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:20 smithi027 ceph-mon[78597]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:20.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:20 smithi027 ceph-mon[78597]: scrub summary: active paths [/] 2024-04-05T23:34:20.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:20 smithi027 ceph-mon[78597]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:21.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:20 smithi016 ceph-mon[38248]: scrub queued for path: / 2024-04-05T23:34:21.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:20 smithi016 ceph-mon[38248]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:21.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:20 smithi016 ceph-mon[38248]: scrub summary: active paths [/] 2024-04-05T23:34:21.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:20 smithi016 ceph-mon[38248]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:21.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:20 smithi130 ceph-mon[79417]: scrub queued for path: / 2024-04-05T23:34:21.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:20 smithi130 ceph-mon[79417]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:21.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:20 smithi130 ceph-mon[79417]: scrub summary: active paths [/] 2024-04-05T23:34:21.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:20 smithi130 ceph-mon[79417]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:21.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:21 smithi027 ceph-mon[78597]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-04-05T23:34:22.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:21 smithi016 ceph-mon[38248]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-04-05T23:34:22.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:21 smithi130 ceph-mon[79417]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-04-05T23:34:22.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:22 smithi027 ceph-mon[78597]: scrub summary: idle 2024-04-05T23:34:23.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:22 smithi016 ceph-mon[38248]: scrub summary: idle 2024-04-05T23:34:23.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:22 smithi130 ceph-mon[79417]: scrub summary: idle 2024-04-05T23:34:23.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:23 smithi027 ceph-mon[78597]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 307 B/s wr, 11 op/s 2024-04-05T23:34:24.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:23 smithi016 ceph-mon[38248]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 307 B/s wr, 11 op/s 2024-04-05T23:34:24.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:23 smithi130 ceph-mon[79417]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 307 B/s wr, 11 op/s 2024-04-05T23:34:26.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:25 smithi016 ceph-mon[38248]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 302 B/s wr, 11 op/s 2024-04-05T23:34:26.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:25 smithi130 ceph-mon[79417]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 302 B/s wr, 11 op/s 2024-04-05T23:34:26.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:25 smithi027 ceph-mon[78597]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 302 B/s wr, 11 op/s 2024-04-05T23:34:28.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:27 smithi016 ceph-mon[38248]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-04-05T23:34:28.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:27 smithi130 ceph-mon[79417]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-04-05T23:34:28.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:27 smithi027 ceph-mon[78597]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-04-05T23:34:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.050+0000 7fd620a00700 1 -- 172.21.15.16:0/4025971595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd61c100e80 msgr2=0x7fd61c101260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.050+0000 7fd620a00700 1 --2- 172.21.15.16:0/4025971595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd61c100e80 0x7fd61c101260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009990 tx=0x7fd60c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.050+0000 7fd620a00700 1 -- 172.21.15.16:0/4025971595 shutdown_connections 2024-04-05T23:34:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.050+0000 7fd620a00700 1 --2- 172.21.15.16:0/4025971595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd61c105fa0 0x7fd61c10db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.050+0000 7fd620a00700 1 --2- 172.21.15.16:0/4025971595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd61c100e80 0x7fd61c101260 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.050+0000 7fd620a00700 1 --2- 172.21.15.16:0/4025971595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd61c101830 0x7fd61c105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.050+0000 7fd620a00700 1 -- 172.21.15.16:0/4025971595 >> 172.21.15.16:0/4025971595 conn(0x7fd61c0fc740 msgr2=0x7fd61c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.050+0000 7fd620a00700 1 -- 172.21.15.16:0/4025971595 shutdown_connections 2024-04-05T23:34:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.050+0000 7fd620a00700 1 -- 172.21.15.16:0/4025971595 wait complete. 2024-04-05T23:34:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.052+0000 7fd620a00700 1 Processor -- start 2024-04-05T23:34:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.052+0000 7fd620a00700 1 -- start start 2024-04-05T23:34:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.052+0000 7fd620a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd61c100e80 0x7fd61c197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.052+0000 7fd620a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd61c105fa0 0x7fd61c1976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.052+0000 7fd620a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd61c19ba60 0x7fd61c19bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.052+0000 7fd620a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c19c4e0 con 0x7fd61c105fa0 2024-04-05T23:34:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.052+0000 7fd620a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c19c650 con 0x7fd61c19ba60 2024-04-05T23:34:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.052+0000 7fd620a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c19c810 con 0x7fd61c100e80 2024-04-05T23:34:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd61a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd61c100e80 0x7fd61c197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd61a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd61c100e80 0x7fd61c197180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:52744/0 (socket says 172.21.15.16:52744) 2024-04-05T23:34:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd61a800700 1 -- 172.21.15.16:0/1800974122 learned_addr learned my addr 172.21.15.16:0/1800974122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd619e00700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd61c105fa0 0x7fd61c1976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd61b200700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd61c19ba60 0x7fd61c19bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd61a800700 1 -- 172.21.15.16:0/1800974122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd61c19ba60 msgr2=0x7fd61c19bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd61a800700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd61c19ba60 0x7fd61c19bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd61a800700 1 -- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd61c105fa0 msgr2=0x7fd61c1976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd61a800700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd61c105fa0 0x7fd61c1976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.053+0000 7fd61a800700 1 -- 172.21.15.16:0/1800974122 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c0095f0 con 0x7fd61c100e80 2024-04-05T23:34:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.054+0000 7fd61a800700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd61c100e80 0x7fd61c197180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd60c000c00 tx=0x7fd60c005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.054+0000 7fd613400700 1 -- 172.21.15.16:0/1800974122 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c01d070 con 0x7fd61c100e80 2024-04-05T23:34:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.054+0000 7fd620a00700 1 -- 172.21.15.16:0/1800974122 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61c1a3d10 con 0x7fd61c100e80 2024-04-05T23:34:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.054+0000 7fd620a00700 1 -- 172.21.15.16:0/1800974122 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61c1a4190 con 0x7fd61c100e80 2024-04-05T23:34:29.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.054+0000 7fd613400700 1 -- 172.21.15.16:0/1800974122 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd60c022470 con 0x7fd61c100e80 2024-04-05T23:34:29.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.054+0000 7fd613400700 1 -- 172.21.15.16:0/1800974122 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c017610 con 0x7fd61c100e80 2024-04-05T23:34:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.057+0000 7fd613400700 1 -- 172.21.15.16:0/1800974122 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7fd60c00fb00 con 0x7fd61c100e80 2024-04-05T23:34:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.060+0000 7fd613400700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd6000a13b0 0x7fd6000a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.060+0000 7fd613400700 1 -- 172.21.15.16:0/1800974122 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd60c0ce0a0 con 0x7fd61c100e80 2024-04-05T23:34:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.060+0000 7fd619e00700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd6000a13b0 0x7fd6000a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.060+0000 7fd634fe5b80 1 Processor -- start 2024-04-05T23:34:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.061+0000 7fd634fe5b80 1 -- start start 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.061+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0361ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.061+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 0x564d03622d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.061+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 0x564d03623730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.061+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d03580b80 con 0x564d0361e630 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.061+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d03571930 con 0x564d0361ef50 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.061+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0356f2f0 con 0x564d036232d0 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0361ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0361ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47904/0 (socket says 172.21.15.16:47904) 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61a800700 1 -- 172.21.15.16:0/1885273818 learned_addr learned my addr 172.21.15.16:0/1885273818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61b200700 1 --2- 172.21.15.16:0/1885273818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 0x564d03623730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd619e00700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd6000a13b0 0x7fd6000a3870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd61c1986d0 tx=0x7fd608009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61a800700 1 -- 172.21.15.16:0/1885273818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 msgr2=0x564d03623730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61a800700 1 --2- 172.21.15.16:0/1885273818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 0x564d03623730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61a800700 1 -- 172.21.15.16:0/1885273818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 msgr2=0x564d03622d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:34:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61a800700 1 --2- 172.21.15.16:0/1885273818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 0x564d03622d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61a800700 1 -- 172.21.15.16:0/1885273818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d033ec350 con 0x564d0361e630 2024-04-05T23:34:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.062+0000 7fd61a800700 1 --2- 172.21.15.16:0/1885273818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0361ea10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0a2e80 tx=0x7fd60c00b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66dd6b550a16739 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.063+0000 7fd607400700 1 -- 172.21.15.16:0/1885273818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c09a920 con 0x564d0361e630 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.063+0000 7fd607400700 1 -- 172.21.15.16:0/1885273818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd60c09b6c0 con 0x564d0361e630 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.063+0000 7fd607400700 1 -- 172.21.15.16:0/1885273818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c09a710 con 0x564d0361e630 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.064+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1885273818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 msgr2=0x564d0361ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.064+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/1885273818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0361ea10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0a2e80 tx=0x7fd60c00b610 comp rx=0 tx=0).stop 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.064+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1885273818 shutdown_connections 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.064+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/1885273818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0361ea10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.064+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/1885273818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 0x564d03623730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.064+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/1885273818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 0x564d03622d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.064+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1885273818 >> 172.21.15.16:0/1885273818 conn(0x564d0358e1c0 msgr2=0x564d03615060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.064+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1885273818 shutdown_connections 2024-04-05T23:34:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.064+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1885273818 wait complete. 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.065+0000 7fd634fe5b80 1 -- start start 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.065+0000 7fd634fe5b80 1 Processor -- start 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0362aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 0x564d03627dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 0x564d03628470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0356abe0 con 0x564d0361e630 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d03506d20 con 0x564d0361ef50 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d035069c0 con 0x564d036232d0 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd61a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0362aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd634fe5b80 10 client.0 _ll_register_callbacks cb 0x564d03372340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd61a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0362aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47906/0 (socket says 172.21.15.16:47906) 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd61a800700 1 -- 172.21.15.16:0/305197161 learned_addr learned my addr 172.21.15.16:0/305197161 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.066+0000 7fd61b200700 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 0x564d03628470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd61b200700 1 -- 172.21.15.16:0/305197161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 msgr2=0x564d03627dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd619e00700 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 0x564d03627dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd61b200700 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 0x564d03627dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd61b200700 1 -- 172.21.15.16:0/305197161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 msgr2=0x564d0362aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd61b200700 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0362aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd61b200700 1 -- 172.21.15.16:0/305197161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61c1a4190 con 0x564d036232d0 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd619e00700 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 0x564d03627dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd61a800700 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0362aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd61b200700 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 0x564d03628470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd614009fb0 tx=0x7fd614004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9544c3d7d61a8918 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd5fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd5fbe00700 1 -- 172.21.15.16:0/305197161 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd614004db0 con 0x564d036232d0 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.067+0000 7fd634fe5b80 1 -- 172.21.15.16:0/305197161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d0360d1e0 con 0x564d036232d0 2024-04-05T23:34:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd5fbe00700 1 -- 172.21.15.16:0/305197161 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd614007500 con 0x564d036232d0 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd5fbe00700 1 -- 172.21.15.16:0/305197161 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6140076c0 con 0x564d036232d0 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd634fe5b80 10 client.25699 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd634fe5b80 1 -- 172.21.15.16:0/305197161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564d03398b00 con 0x564d036232d0 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd5fbe00700 1 -- 172.21.15.16:0/305197161 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564d0360d1e0 con 0x564d036232d0 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd5fbe00700 1 -- 172.21.15.16:0/305197161 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fd614016870 con 0x564d036232d0 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd634fe5b80 10 client.25699 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd634fe5b80 10 client.25699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd634fe5b80 20 client.25699 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x564d03638250 0x564d0363a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd634fe5b80 4 client.25699 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.068+0000 7fd634fe5b80 1 -- 172.21.15.16:0/305197161 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d035636a0 con 0x564d03638250 2024-04-05T23:34:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.069+0000 7fd61a800700 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x564d03638250 0x564d0363a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.070+0000 7fd5fbe00700 10 client.25699 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.070+0000 7fd61a800700 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x564d03638250 0x564d0363a670 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ad9d2fa99ef85e1 server_cookie=fb361b8d4dc13338 in_seq=0 out_seq=0 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.070+0000 7fd5fbe00700 1 -- 172.21.15.16:0/305197161 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564d035636a0 con 0x564d03638250 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.070+0000 7fd5fbe00700 10 client.25699 handle_command_reply: tid=0 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.070+0000 7fd634fe5b80 1 client.25699 shutdown 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 -- 172.21.15.16:0/305197161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 msgr2=0x564d03628470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 0x564d03628470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd614009fb0 tx=0x7fd614004040 comp rx=0 tx=0).stop 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 -- 172.21.15.16:0/305197161 shutdown_connections 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x564d03638250 0x564d0363a670 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d0362aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd5fbe00700 0 client.25699 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d036232d0 0x564d03628470 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/305197161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361ef50 0x564d03627dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 -- 172.21.15.16:0/305197161 >> 172.21.15.16:0/305197161 conn(0x564d0358e1c0 msgr2=0x564d03614db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 -- 172.21.15.16:0/305197161 shutdown_connections 2024-04-05T23:34:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 1 -- 172.21.15.16:0/305197161 wait complete. 2024-04-05T23:34:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.071+0000 7fd634fe5b80 20 client.25699 trim_cache size 0 max 16384 2024-04-05T23:34:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.087+0000 7fd634fe5b80 1 Processor -- start 2024-04-05T23:34:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.087+0000 7fd634fe5b80 1 -- start start 2024-04-05T23:34:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.087+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361e630 0x564d03627d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.087+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 0x564d0362f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.087+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d036232d0 0x564d03610ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.087+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d033ee640 con 0x564d036232d0 2024-04-05T23:34:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.087+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0362f730 con 0x564d0361e630 2024-04-05T23:34:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.087+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d036263a0 con 0x564d0361ef50 2024-04-05T23:34:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361e630 0x564d03627d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61b200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d036232d0 0x564d03610ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361e630 0x564d03627d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:58416/0 (socket says 172.21.15.16:58416) 2024-04-05T23:34:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61a800700 1 -- 172.21.15.16:0/1892313586 learned_addr learned my addr 172.21.15.16:0/1892313586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd619e00700 1 --2- 172.21.15.16:0/1892313586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 0x564d0362f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61a800700 1 -- 172.21.15.16:0/1892313586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 msgr2=0x564d0362f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61a800700 1 --2- 172.21.15.16:0/1892313586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 0x564d0362f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61a800700 1 -- 172.21.15.16:0/1892313586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d036232d0 msgr2=0x564d03610ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61a800700 1 --2- 172.21.15.16:0/1892313586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d036232d0 0x564d03610ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61a800700 1 -- 172.21.15.16:0/1892313586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d03530e40 con 0x564d0361e630 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.088+0000 7fd61a800700 1 --2- 172.21.15.16:0/1892313586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361e630 0x564d03627d90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0974c0 tx=0x7fd60c024040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9034381ce7d16ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.089+0000 7fd607e00700 1 -- 172.21.15.16:0/1892313586 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c09ab00 con 0x564d0361e630 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.089+0000 7fd607e00700 1 -- 172.21.15.16:0/1892313586 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd60c0b92c0 con 0x564d0361e630 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.089+0000 7fd607e00700 1 -- 172.21.15.16:0/1892313586 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c099380 con 0x564d0361e630 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.089+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1892313586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361e630 msgr2=0x564d03627d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.090+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/1892313586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361e630 0x564d03627d90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0974c0 tx=0x7fd60c024040 comp rx=0 tx=0).stop 2024-04-05T23:34:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.090+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1892313586 shutdown_connections 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.090+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/1892313586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d036232d0 0x564d03610ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.090+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/1892313586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d0361e630 0x564d03627d90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.090+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/1892313586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 0x564d0362f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.090+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1892313586 >> 172.21.15.16:0/1892313586 conn(0x564d0358e1c0 msgr2=0x564d0363efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.090+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1892313586 shutdown_connections 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.090+0000 7fd634fe5b80 1 -- 172.21.15.16:0/1892313586 wait complete. 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.090+0000 7fd634fe5b80 1 -- start start 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.091+0000 7fd634fe5b80 1 Processor -- start 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.091+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d036201d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.091+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 0x564d03620710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.091+0000 7fd634fe5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d036232d0 0x564d0368a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.091+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0368a5b0 con 0x564d0361e630 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.091+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0368a6f0 con 0x564d036232d0 2024-04-05T23:34:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.091+0000 7fd634fe5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0368a8b0 con 0x564d0361ef50 2024-04-05T23:34:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.091+0000 7fd634fe5b80 10 client.0 _ll_register_callbacks cb 0x564d03372340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd61b200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d036232d0 0x564d0368a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd61b200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d036232d0 0x564d0368a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:58420/0 (socket says 172.21.15.16:58420) 2024-04-05T23:34:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd61b200700 1 -- 172.21.15.16:0/3305464249 learned_addr learned my addr 172.21.15.16:0/3305464249 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd619e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 0x564d03620710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd61b200700 1 -- 172.21.15.16:0/3305464249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 msgr2=0x564d03620710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd61b200700 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 0x564d03620710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd61b200700 1 -- 172.21.15.16:0/3305464249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 msgr2=0x564d036201d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd61b200700 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d036201d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd61b200700 1 -- 172.21.15.16:0/3305464249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d033ec350 con 0x564d036232d0 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd61b200700 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d036232d0 0x564d0368a070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd6140099b0 tx=0x7fd614004bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb4da2e17efabb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.092+0000 7fd5fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.093+0000 7fd5fbe00700 1 -- 172.21.15.16:0/3305464249 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd614016630 con 0x564d036232d0 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.093+0000 7fd634fe5b80 1 -- 172.21.15.16:0/3305464249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd60c09b6c0 con 0x564d036232d0 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.093+0000 7fd5fbe00700 1 -- 172.21.15.16:0/3305464249 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd614009170 con 0x564d036232d0 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.093+0000 7fd5fbe00700 1 -- 172.21.15.16:0/3305464249 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd614016790 con 0x564d036232d0 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.093+0000 7fd634fe5b80 10 client.25670 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.093+0000 7fd634fe5b80 1 -- 172.21.15.16:0/3305464249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564d033f0c70 con 0x564d036232d0 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.093+0000 7fd5fbe00700 1 -- 172.21.15.16:0/3305464249 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd60c09b6c0 con 0x564d036232d0 2024-04-05T23:34:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.094+0000 7fd5fbe00700 1 -- 172.21.15.16:0/3305464249 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fd614005140 con 0x564d036232d0 2024-04-05T23:34:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.094+0000 7fd634fe5b80 10 client.25670 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.094+0000 7fd634fe5b80 10 client.25670 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.094+0000 7fd634fe5b80 20 client.25670 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.094+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x564d0368c130 0x564d0368e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.094+0000 7fd634fe5b80 4 client.25670 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:34:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.094+0000 7fd634fe5b80 1 -- 172.21.15.16:0/3305464249 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564d033b9c90 con 0x564d0368c130 2024-04-05T23:34:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.094+0000 7fd61a800700 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x564d0368c130 0x564d0368e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.095+0000 7fd61a800700 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x564d0368c130 0x564d0368e550 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e2b010419f5b487 server_cookie=9f4a32db837bb688 in_seq=0 out_seq=0 2024-04-05T23:34:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.095+0000 7fd5fbe00700 10 client.25670 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd5fbe00700 1 -- 172.21.15.16:0/3305464249 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564d033b9c90 con 0x564d0368c130 2024-04-05T23:34:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd5fbe00700 10 client.25670 handle_command_reply: tid=0 2024-04-05T23:34:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 client.25670 shutdown 2024-04-05T23:34:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 -- 172.21.15.16:0/3305464249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d036232d0 msgr2=0x564d0368a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d036232d0 0x564d0368a070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd6140099b0 tx=0x7fd614004bb0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 -- 172.21.15.16:0/3305464249 shutdown_connections 2024-04-05T23:34:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x564d0368c130 0x564d0368e550 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564d0361e630 0x564d036201d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd5fbe00700 0 client.25670 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564d036232d0 0x564d0368a070 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 --2- 172.21.15.16:0/3305464249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564d0361ef50 0x564d03620710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 -- 172.21.15.16:0/3305464249 >> 172.21.15.16:0/3305464249 conn(0x564d0358e1c0 msgr2=0x564d03610ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 -- 172.21.15.16:0/3305464249 shutdown_connections 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.103+0000 7fd634fe5b80 1 -- 172.21.15.16:0/3305464249 wait complete. 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.104+0000 7fd634fe5b80 20 client.25670 trim_cache size 0 max 16384 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:34:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:34:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.104+0000 7fd620a00700 1 -- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd6000a13b0 msgr2=0x7fd6000a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.104+0000 7fd620a00700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd6000a13b0 0x7fd6000a3870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd61c1986d0 tx=0x7fd608009260 comp rx=0 tx=0).stop 2024-04-05T23:34:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.104+0000 7fd620a00700 1 -- 172.21.15.16:0/1800974122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd61c100e80 msgr2=0x7fd61c197180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.104+0000 7fd620a00700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd61c100e80 0x7fd61c197180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd60c000c00 tx=0x7fd60c005740 comp rx=0 tx=0).stop 2024-04-05T23:34:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.104+0000 7fd620a00700 1 -- 172.21.15.16:0/1800974122 shutdown_connections 2024-04-05T23:34:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.104+0000 7fd620a00700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd6000a13b0 0x7fd6000a3870 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.105+0000 7fd620a00700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd61c19ba60 0x7fd61c19bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.105+0000 7fd620a00700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd61c100e80 0x7fd61c197180 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.105+0000 7fd620a00700 1 --2- 172.21.15.16:0/1800974122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd61c105fa0 0x7fd61c1976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.105+0000 7fd620a00700 1 -- 172.21.15.16:0/1800974122 >> 172.21.15.16:0/1800974122 conn(0x7fd61c0fc740 msgr2=0x7fd61c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.105+0000 7fd620a00700 1 -- 172.21.15.16:0/1800974122 shutdown_connections 2024-04-05T23:34:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:29.105+0000 7fd620a00700 1 -- 172.21.15.16:0/1800974122 wait complete. 2024-04-05T23:34:30.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:29 smithi016 ceph-mon[38248]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-04-05T23:34:30.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:29 smithi130 ceph-mon[79417]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-04-05T23:34:30.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:29 smithi027 ceph-mon[78597]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-04-05T23:34:32.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:31 smithi016 ceph-mon[38248]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-04-05T23:34:32.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:31 smithi130 ceph-mon[79417]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-04-05T23:34:32.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:31 smithi027 ceph-mon[78597]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-04-05T23:34:34.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:33 smithi016 ceph-mon[38248]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-05T23:34:34.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:33 smithi130 ceph-mon[79417]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-05T23:34:34.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:33 smithi027 ceph-mon[78597]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-05T23:34:36.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:35 smithi016 ceph-mon[38248]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-04-05T23:34:36.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:35 smithi130 ceph-mon[79417]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-04-05T23:34:36.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:35 smithi027 ceph-mon[78597]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-04-05T23:34:38.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:37 smithi016 ceph-mon[38248]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2024-04-05T23:34:38.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:37 smithi130 ceph-mon[79417]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2024-04-05T23:34:38.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:37 smithi027 ceph-mon[78597]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2024-04-05T23:34:38.717 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 63% (7699/12088) Updating files: 64% (7737/12088) Updating files: 65% (7858/12088) Updating files: 66% (7979/12088) Updating files: 67% (8099/12088) Updating files: 68% (8220/12088) Updating files: 69% (8341/12088) Updating files: 70% (8462/12088) Updating files: 71% (8583/12088) Updating files: 72% (8704/12088) Updating files: 73% (8825/12088) Updating files: 74% (8946/12088) Updating files: 75% (9066/12088) Updating files: 76% (9187/12088) Updating files: 77% (9308/12088) Updating files: 78% (9429/12088) Updating files: 79% (9550/12088) Updating files: 80% (9671/12088) Updating files: 81% (9792/12088) Updating files: 82% (9913/12088) Updating files: 83% (10034/12088) Updating files: 84% (10154/12088) Updating files: 85% (10275/12088) Updating files: 86% (10396/12088) Updating files: 87% (10517/12088) Updating files: 88% (10638/12088) Updating files: 89% (10759/12088) Updating files: 90% (10880/12088) Updating files: 91% (11001/12088) Updating files: 92% (11121/12088) Updating files: 93% (11242/12088) Updating files: 94% (11363/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-04-05T23:34:39.747 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to '8919605b0a5dc1f1650c8aec3051c77c0e3e240e'. 2024-04-05T23:34:39.747 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-05T23:34:39.747 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-05T23:34:39.747 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-05T23:34:39.747 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-05T23:34:39.748 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at 8919605b0a5 Merge branch 'wip-65294-reef' of https://github.com/batrick/ceph into wip-yuri4-testing-2024-04-04-1128-reef 2024-04-05T23:34:39.761 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-05T23:34:39.810 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-05T23:34:39.815 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-05T23:34:39.815 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-05T23:34:40.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:39 smithi016 ceph-mon[38248]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-05T23:34:40.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:39 smithi130 ceph-mon[79417]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-05T23:34:40.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:39 smithi027 ceph-mon[78597]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-05T23:34:40.426 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-05T23:34:40.558 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-05T23:34:40.624 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-05T23:34:40.628 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-05T23:34:40.628 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-05T23:34:40.692 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-05T23:34:40.699 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:34:40.699 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-05T23:34:40.740 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-05T23:34:40.742 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-05T23:34:40.742 DEBUG:teuthology.orchestra.run.smithi016:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8919605b0a5dc1f1650c8aec3051c77c0e3e240e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-05T23:34:40.869 INFO:tasks.workunit.client.0.smithi016.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-05T23:34:40.869 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-04-05T23:34:40.869 INFO:tasks.workunit.client.0.smithi016.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-05T23:34:40.879 INFO:tasks.workunit.client.0.smithi016.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-05T23:34:40.979 INFO:tasks.workunit.client.0.smithi016.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-05T23:34:40.979 INFO:tasks.workunit.client.0.smithi016.stdout:releasing clients 2024-04-05T23:34:41.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 3189 34.24 MB/sec warmup 1 sec latency 6.534 ms 2024-04-05T23:34:42.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:41 smithi016 ceph-mon[38248]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-05T23:34:42.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:41 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:34:42.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:41 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:34:42.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:41 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:34:42.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:41 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:34:42.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:41 smithi130 ceph-mon[79417]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-05T23:34:42.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:41 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:34:42.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:41 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:34:42.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:41 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:34:42.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:41 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:34:42.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:41 smithi027 ceph-mon[78597]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-05T23:34:42.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:41 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:34:42.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:41 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:34:42.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:41 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:34:42.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:41 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:34:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.261+0000 7f6602a00700 1 -- 172.21.15.16:0/3093134129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65fc101030 msgr2=0x7f65fc101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.261+0000 7f6602a00700 1 --2- 172.21.15.16:0/3093134129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65fc101030 0x7f65fc101410 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009990 tx=0x7f65ec009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.261+0000 7f6602a00700 1 -- 172.21.15.16:0/3093134129 shutdown_connections 2024-04-05T23:34:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.261+0000 7f6602a00700 1 --2- 172.21.15.16:0/3093134129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65fc106150 0x7f65fc10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.261+0000 7f6602a00700 1 --2- 172.21.15.16:0/3093134129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65fc101030 0x7f65fc101410 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.261+0000 7f6602a00700 1 --2- 172.21.15.16:0/3093134129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc1019e0 0x7f65fc105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.261+0000 7f6602a00700 1 -- 172.21.15.16:0/3093134129 >> 172.21.15.16:0/3093134129 conn(0x7f65fc0fc8b0 msgr2=0x7f65fc0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.261+0000 7f6602a00700 1 -- 172.21.15.16:0/3093134129 shutdown_connections 2024-04-05T23:34:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.261+0000 7f6602a00700 1 -- 172.21.15.16:0/3093134129 wait complete. 2024-04-05T23:34:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.262+0000 7f6602a00700 1 Processor -- start 2024-04-05T23:34:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f6602a00700 1 -- start start 2024-04-05T23:34:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f6602a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65fc101030 0x7f65fc0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f6602a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65fc106150 0x7f65fc073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f6602a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc073c60 0x7f65fc1a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f6602a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc1a59d0 con 0x7f65fc073c60 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f6602a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc1a5b40 con 0x7f65fc106150 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f6602a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc1a5d00 con 0x7f65fc101030 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f65fbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65fc101030 0x7f65fc0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f65fb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65fc106150 0x7f65fc073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f65fbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65fc101030 0x7f65fc0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:56270/0 (socket says 172.21.15.16:56270) 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f65fbe00700 1 -- 172.21.15.16:0/3868517078 learned_addr learned my addr 172.21.15.16:0/3868517078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.263+0000 7f6600a00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc073c60 0x7f65fc1a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.264+0000 7f65fbe00700 1 -- 172.21.15.16:0/3868517078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65fc106150 msgr2=0x7f65fc073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.264+0000 7f65fbe00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65fc106150 0x7f65fc073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.264+0000 7f65fbe00700 1 -- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc073c60 msgr2=0x7f65fc1a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.264+0000 7f65fbe00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc073c60 0x7f65fc1a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.264+0000 7f65fbe00700 1 -- 172.21.15.16:0/3868517078 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec0095f0 con 0x7f65fc101030 2024-04-05T23:34:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.264+0000 7f65fb400700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65fc106150 0x7f65fc073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.264+0000 7f65fbe00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65fc101030 0x7f65fc0731e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f65ec004760 tx=0x7f65ec004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.264+0000 7f65f8c00700 1 -- 172.21.15.16:0/3868517078 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec01d070 con 0x7f65fc101030 2024-04-05T23:34:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.264+0000 7f6602a00700 1 -- 172.21.15.16:0/3868517078 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65fc1a5f80 con 0x7f65fc101030 2024-04-05T23:34:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.265+0000 7f6602a00700 1 -- 172.21.15.16:0/3868517078 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65fc1a63b0 con 0x7f65fc101030 2024-04-05T23:34:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.265+0000 7f65f8c00700 1 -- 172.21.15.16:0/3868517078 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65ec022470 con 0x7f65fc101030 2024-04-05T23:34:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.265+0000 7f65f8c00700 1 -- 172.21.15.16:0/3868517078 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec017650 con 0x7f65fc101030 2024-04-05T23:34:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.268+0000 7f65f8c00700 1 -- 172.21.15.16:0/3868517078 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f65ec00fa90 con 0x7f65fc101030 2024-04-05T23:34:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.269+0000 7f65f8c00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65e00a1430 0x7f65e00a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.269+0000 7f66165c0b80 1 Processor -- start 2024-04-05T23:34:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.269+0000 7f65f8c00700 1 -- 172.21.15.16:0/3868517078 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f65ec0cdf70 con 0x7f65fc101030 2024-04-05T23:34:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.269+0000 7f66165c0b80 1 -- start start 2024-04-05T23:34:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.269+0000 7f65fb400700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65e00a1430 0x7f65e00a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba0b80 0x561b60ba0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60b9b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60b9b5c0 0x561b60b9ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f66165c0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60b01b80 con 0x561b60ba14a0 2024-04-05T23:34:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f66165c0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60af2930 con 0x561b60ba0b80 2024-04-05T23:34:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f66165c0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60af02f0 con 0x561b60b9b5c0 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba0b80 0x561b60ba0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba0b80 0x561b60ba0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47514/0 (socket says 172.21.15.16:47514) 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fbe00700 1 -- 172.21.15.16:0/215076081 learned_addr learned my addr 172.21.15.16:0/215076081 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f6600a00700 1 --2- 172.21.15.16:0/215076081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60b9b5c0 0x561b60b9ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fb400700 1 --2- 172.21.15.16:0/215076081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60b9b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fbe00700 1 -- 172.21.15.16:0/215076081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60b9b5c0 msgr2=0x561b60b9ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fbe00700 1 --2- 172.21.15.16:0/215076081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60b9b5c0 0x561b60b9ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fbe00700 1 -- 172.21.15.16:0/215076081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 msgr2=0x561b60b9b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fbe00700 1 --2- 172.21.15.16:0/215076081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60b9b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fb400700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65e00a1430 0x7f65e00a38f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65fc074910 tx=0x7f65e8009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fbe00700 1 -- 172.21.15.16:0/215076081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b6096d350 con 0x561b60ba0b80 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.270+0000 7f65fbe00700 1 --2- 172.21.15.16:0/215076081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba0b80 0x561b60ba0f60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f65ec097390 tx=0x7f65ec09ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b06a86a181e4d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f65f1200700 1 -- 172.21.15.16:0/215076081 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec09bea0 con 0x561b60ba0b80 2024-04-05T23:34:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f65f1200700 1 -- 172.21.15.16:0/215076081 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65ec00fd40 con 0x561b60ba0b80 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f65f1200700 1 -- 172.21.15.16:0/215076081 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec09b350 con 0x561b60ba0b80 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f66165c0b80 1 -- 172.21.15.16:0/215076081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba0b80 msgr2=0x561b60ba0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f66165c0b80 1 --2- 172.21.15.16:0/215076081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba0b80 0x561b60ba0f60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f65ec097390 tx=0x7f65ec09ac60 comp rx=0 tx=0).stop 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f66165c0b80 1 -- 172.21.15.16:0/215076081 shutdown_connections 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f66165c0b80 1 --2- 172.21.15.16:0/215076081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60b9b5c0 0x561b60b9ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f66165c0b80 1 --2- 172.21.15.16:0/215076081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba0b80 0x561b60ba0f60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f66165c0b80 1 --2- 172.21.15.16:0/215076081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60b9b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f66165c0b80 1 -- 172.21.15.16:0/215076081 >> 172.21.15.16:0/215076081 conn(0x561b60b0f1c0 msgr2=0x561b60b95be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f66165c0b80 1 -- 172.21.15.16:0/215076081 shutdown_connections 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.271+0000 7f66165c0b80 1 -- 172.21.15.16:0/215076081 wait complete. 2024-04-05T23:34:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.272+0000 7f66165c0b80 1 -- start start 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.272+0000 7f66165c0b80 1 Processor -- start 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60ba74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60ba4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f66165c0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60aebbe0 con 0x561b60ba14a0 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f66165c0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60a87d20 con 0x561b60b9b5c0 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f66165c0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60a879c0 con 0x561b60ba0b80 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f66165c0b80 10 client.0 _ll_register_callbacks cb 0x561b608f3340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60ba74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60ba74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47524/0 (socket says 172.21.15.16:47524) 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fbe00700 1 -- 172.21.15.16:0/386833891 learned_addr learned my addr 172.21.15.16:0/386833891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fb400700 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f6600a00700 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60ba4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fbe00700 1 -- 172.21.15.16:0/386833891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 msgr2=0x561b60ba47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fbe00700 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fbe00700 1 -- 172.21.15.16:0/386833891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 msgr2=0x561b60ba4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fbe00700 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60ba4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fbe00700 1 -- 172.21.15.16:0/386833891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65fc1a63b0 con 0x561b60b9b5c0 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.273+0000 7f65fb400700 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f65fbe00700 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60ba74e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f65ec004730 tx=0x7f65ec09a920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28bb070cfaac5dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f65e5600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f65e5600700 1 -- 172.21.15.16:0/386833891 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec09bea0 con 0x561b60b9b5c0 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f65e5600700 1 -- 172.21.15.16:0/386833891 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f65ec09ae90 con 0x561b60b9b5c0 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f66165c0b80 1 -- 172.21.15.16:0/386833891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b60b8e1e0 con 0x561b60b9b5c0 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f65e5600700 1 -- 172.21.15.16:0/386833891 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65ec09b590 con 0x561b60b9b5c0 2024-04-05T23:34:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f66165c0b80 10 client.25682 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f66165c0b80 1 -- 172.21.15.16:0/386833891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561b60919b00 con 0x561b60b9b5c0 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f65e5600700 1 -- 172.21.15.16:0/386833891 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561b60b8e1e0 con 0x561b60b9b5c0 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f65e5600700 1 -- 172.21.15.16:0/386833891 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f65ec0cb7f0 con 0x561b60b9b5c0 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f66165c0b80 10 client.25682 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f66165c0b80 10 client.25682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f66165c0b80 20 client.25682 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f66165c0b80 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561b60bb4750 0x561b60bb6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f66165c0b80 4 client.25682 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.274+0000 7f66165c0b80 1 -- 172.21.15.16:0/386833891 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b60ae46a0 con 0x561b60bb4750 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.275+0000 7f65fb400700 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561b60bb4750 0x561b60bb6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.275+0000 7f65fb400700 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561b60bb4750 0x561b60bb6b70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea65c799d6934316 server_cookie=463402c7e876746c in_seq=0 out_seq=0 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.275+0000 7f65e5600700 10 client.25682 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f65e5600700 1 -- 172.21.15.16:0/386833891 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561b60ae46a0 con 0x561b60bb4750 2024-04-05T23:34:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f65e5600700 10 client.25682 handle_command_reply: tid=0 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f66165c0b80 1 client.25682 shutdown 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f66165c0b80 1 -- 172.21.15.16:0/386833891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 msgr2=0x561b60ba74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f66165c0b80 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60ba74e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f65ec004730 tx=0x7f65ec09a920 comp rx=0 tx=0).stop 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f66165c0b80 1 -- 172.21.15.16:0/386833891 shutdown_connections 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f66165c0b80 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561b60bb4750 0x561b60bb6b70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f66165c0b80 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60ba4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f66165c0b80 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60ba74e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f66165c0b80 1 --2- 172.21.15.16:0/386833891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f65e5600700 0 client.25682 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.276+0000 7f66165c0b80 1 -- 172.21.15.16:0/386833891 >> 172.21.15.16:0/386833891 conn(0x561b60b0f1c0 msgr2=0x561b60b95280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.277+0000 7f66165c0b80 1 -- 172.21.15.16:0/386833891 shutdown_connections 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.277+0000 7f66165c0b80 1 -- 172.21.15.16:0/386833891 wait complete. 2024-04-05T23:34:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.277+0000 7f66165c0b80 20 client.25682 trim_cache size 0 max 16384 2024-04-05T23:34:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.292+0000 7f66165c0b80 1 Processor -- start 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f66165c0b80 1 -- start start 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60b9b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60bb0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60babba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f66165c0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b6096f640 con 0x561b60ba14a0 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f66165c0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60ba12e0 con 0x561b60b9b5c0 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f66165c0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60b973d0 con 0x561b60ba0b80 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f65fbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60b9b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f65fbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60b9b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47540/0 (socket says 172.21.15.16:47540) 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f65fbe00700 1 -- 172.21.15.16:0/2243187256 learned_addr learned my addr 172.21.15.16:0/2243187256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f65fb400700 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60bb0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f6600a00700 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60babba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.293+0000 7f65fbe00700 1 -- 172.21.15.16:0/2243187256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 msgr2=0x561b60bb0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.294+0000 7f65fbe00700 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60bb0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.294+0000 7f65fbe00700 1 -- 172.21.15.16:0/2243187256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 msgr2=0x561b60babba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.294+0000 7f65fbe00700 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60babba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.294+0000 7f65fbe00700 1 -- 172.21.15.16:0/2243187256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b60ab1e40 con 0x561b60b9b5c0 2024-04-05T23:34:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.294+0000 7f6600a00700 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60babba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.294+0000 7f65fbe00700 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60b9b030 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f65ec09ae20 tx=0x7f65ec09ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=115e86dce456665a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.294+0000 7f65f1c00700 1 -- 172.21.15.16:0/2243187256 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec09bea0 con 0x561b60b9b5c0 2024-04-05T23:34:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.294+0000 7f65f1c00700 1 -- 172.21.15.16:0/2243187256 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65ec0a3000 con 0x561b60b9b5c0 2024-04-05T23:34:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.294+0000 7f65f1c00700 1 -- 172.21.15.16:0/2243187256 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0a3280 con 0x561b60b9b5c0 2024-04-05T23:34:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 -- 172.21.15.16:0/2243187256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 msgr2=0x561b60b9b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60b9b030 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f65ec09ae20 tx=0x7f65ec09ac60 comp rx=0 tx=0).stop 2024-04-05T23:34:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 -- 172.21.15.16:0/2243187256 shutdown_connections 2024-04-05T23:34:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60ba14a0 0x561b60babba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60b9b5c0 0x561b60b9b030 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 --2- 172.21.15.16:0/2243187256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60bb0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 -- 172.21.15.16:0/2243187256 >> 172.21.15.16:0/2243187256 conn(0x561b60b0f1c0 msgr2=0x561b60bb0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 -- 172.21.15.16:0/2243187256 shutdown_connections 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 -- 172.21.15.16:0/2243187256 wait complete. 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 -- start start 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.295+0000 7f66165c0b80 1 Processor -- start 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60b9b5c0 0x561b60ba7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f66165c0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba14a0 0x561b60ba8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f66165c0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60c076b0 con 0x561b60b9b5c0 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f66165c0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60c07820 con 0x561b60ba14a0 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f66165c0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b60c079e0 con 0x561b60ba0b80 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f66165c0b80 10 client.0 _ll_register_callbacks cb 0x561b608f3340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f65fb400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f6600a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba14a0 0x561b60ba8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f65fb400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:56304/0 (socket says 172.21.15.16:56304) 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f65fb400700 1 -- 172.21.15.16:0/100623601 learned_addr learned my addr 172.21.15.16:0/100623601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f65fbe00700 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60b9b5c0 0x561b60ba7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f6600a00700 1 -- 172.21.15.16:0/100623601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 msgr2=0x561b60ba8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f6600a00700 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f6600a00700 1 -- 172.21.15.16:0/100623601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60b9b5c0 msgr2=0x561b60ba7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f6600a00700 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60b9b5c0 0x561b60ba7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f6600a00700 1 -- 172.21.15.16:0/100623601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b60919b00 con 0x561b60ba14a0 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.296+0000 7f65fb400700 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.297+0000 7f6600a00700 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba14a0 0x561b60ba8540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x561b60b977f0 tx=0x7f65f4007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29a611dd8526308c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.297+0000 7f65e5600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.297+0000 7f65e5600700 1 -- 172.21.15.16:0/100623601 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f400e040 con 0x561b60ba14a0 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.297+0000 7f66165c0b80 1 -- 172.21.15.16:0/100623601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65ec00fd40 con 0x561b60ba14a0 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.297+0000 7f65e5600700 1 -- 172.21.15.16:0/100623601 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f65f4004bb0 con 0x561b60ba14a0 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.297+0000 7f65e5600700 1 -- 172.21.15.16:0/100623601 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65f4004d70 con 0x561b60ba14a0 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.297+0000 7f66165c0b80 10 client.25688 fetch_fsmap learned FSMap version 30 2024-04-05T23:34:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.297+0000 7f66165c0b80 1 -- 172.21.15.16:0/100623601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561b60971c70 con 0x561b60ba14a0 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.297+0000 7f65e5600700 1 -- 172.21.15.16:0/100623601 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65ec00fd40 con 0x561b60ba14a0 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f65e5600700 1 -- 172.21.15.16:0/100623601 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 30) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f65f4026070 con 0x561b60ba14a0 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f66165c0b80 10 client.25688 fetch_fsmap finished waiting for FSMap version 30 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f66165c0b80 10 client.25688 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f66165c0b80 20 client.25688 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f66165c0b80 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561b60c092b0 0x561b60c0b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f66165c0b80 4 client.25688 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f66165c0b80 1 -- 172.21.15.16:0/100623601 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561b6093ac90 con 0x561b60c092b0 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f65fbe00700 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561b60c092b0 0x561b60c0b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f65fbe00700 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561b60c092b0 0x561b60c0b6d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8133757cae00fb8f server_cookie=a767d47585be435f in_seq=0 out_seq=0 2024-04-05T23:34:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.298+0000 7f65e5600700 10 client.25688 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f65e5600700 1 -- 172.21.15.16:0/100623601 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561b6093ac90 con 0x561b60c092b0 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f65e5600700 10 client.25688 handle_command_reply: tid=0 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 client.25688 shutdown 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 -- 172.21.15.16:0/100623601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba14a0 msgr2=0x561b60ba8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba14a0 0x561b60ba8540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x561b60b977f0 tx=0x7f65f4007ed0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 -- 172.21.15.16:0/100623601 shutdown_connections 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561b60c092b0 0x561b60c0b6d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561b60b9b5c0 0x561b60ba7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561b60ba14a0 0x561b60ba8540 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 --2- 172.21.15.16:0/100623601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b60ba0b80 0x561b60ba8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f65e5600700 0 client.25688 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 -- 172.21.15.16:0/100623601 >> 172.21.15.16:0/100623601 conn(0x561b60b0f1c0 msgr2=0x561b60ba1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 -- 172.21.15.16:0/100623601 shutdown_connections 2024-04-05T23:34:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.357+0000 7f66165c0b80 1 -- 172.21.15.16:0/100623601 wait complete. 2024-04-05T23:34:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f66165c0b80 20 client.25688 trim_cache size 0 max 16384 2024-04-05T23:34:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:34:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:34:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:34:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 -- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65e00a1430 msgr2=0x7f65e00a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65e00a1430 0x7f65e00a38f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65fc074910 tx=0x7f65e8009260 comp rx=0 tx=0).stop 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 -- 172.21.15.16:0/3868517078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65fc101030 msgr2=0x7f65fc0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65fc101030 0x7f65fc0731e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f65ec004760 tx=0x7f65ec004790 comp rx=0 tx=0).stop 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 -- 172.21.15.16:0/3868517078 shutdown_connections 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f65e00a1430 0x7f65e00a38f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc073c60 0x7f65fc1a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65fc101030 0x7f65fc0731e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 --2- 172.21.15.16:0/3868517078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65fc106150 0x7f65fc073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 -- 172.21.15.16:0/3868517078 >> 172.21.15.16:0/3868517078 conn(0x7f65fc0fc8b0 msgr2=0x7f65fc0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 -- 172.21.15.16:0/3868517078 shutdown_connections 2024-04-05T23:34:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:42.358+0000 7f6602a00700 1 -- 172.21.15.16:0/3868517078 wait complete. 2024-04-05T23:34:42.979 INFO:tasks.workunit.client.0.smithi016.stdout: 1 5905 23.50 MB/sec warmup 2 sec latency 26.837 ms 2024-04-05T23:34:43.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 8160 20.37 MB/sec warmup 3 sec latency 14.091 ms 2024-04-05T23:34:44.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:43 smithi016 ceph-mon[38248]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-05T23:34:44.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:43 smithi130 ceph-mon[79417]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-05T23:34:44.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:43 smithi027 ceph-mon[78597]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-05T23:34:44.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 11231 19.58 MB/sec warmup 4 sec latency 5.890 ms 2024-04-05T23:34:45.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 14490 19.42 MB/sec warmup 5 sec latency 8.232 ms 2024-04-05T23:34:46.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:45 smithi016 ceph-mon[38248]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-04-05T23:34:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:45 smithi130 ceph-mon[79417]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-04-05T23:34:46.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:45 smithi027 ceph-mon[78597]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-04-05T23:34:46.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 17544 18.64 MB/sec warmup 6 sec latency 12.261 ms 2024-04-05T23:34:47.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 20186 17.66 MB/sec warmup 7 sec latency 10.190 ms 2024-04-05T23:34:48.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:47 smithi016 ceph-mon[38248]: pgmap v40: 97 pgs: 97 active+clean; 8.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 689 KiB/s wr, 130 op/s 2024-04-05T23:34:48.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:47 smithi130 ceph-mon[79417]: pgmap v40: 97 pgs: 97 active+clean; 8.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 689 KiB/s wr, 130 op/s 2024-04-05T23:34:48.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:47 smithi027 ceph-mon[78597]: pgmap v40: 97 pgs: 97 active+clean; 8.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 689 KiB/s wr, 130 op/s 2024-04-05T23:34:48.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21308 16.75 MB/sec warmup 8 sec latency 668.054 ms 2024-04-05T23:34:49.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:48 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:49.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:48 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:49.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:48 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:49.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21308 14.89 MB/sec warmup 9 sec latency 1668.177 ms 2024-04-05T23:34:50.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:49 smithi016 ceph-mon[38248]: pgmap v41: 97 pgs: 97 active+clean; 8.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 689 KiB/s wr, 126 op/s 2024-04-05T23:34:50.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:49 smithi130 ceph-mon[79417]: pgmap v41: 97 pgs: 97 active+clean; 8.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 689 KiB/s wr, 126 op/s 2024-04-05T23:34:50.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-05T23:34:50.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:49 smithi027 ceph-mon[78597]: pgmap v41: 97 pgs: 97 active+clean; 8.2 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 689 KiB/s wr, 126 op/s 2024-04-05T23:34:50.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.368+0000 7f05de000700 1 -- 172.21.15.16:0/1217518082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d8073310 msgr2=0x7f05d80736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.368+0000 7f05de000700 1 --2- 172.21.15.16:0/1217518082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d8073310 0x7f05d80736f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009990 tx=0x7f05c8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.368+0000 7f05de000700 1 -- 172.21.15.16:0/1217518082 shutdown_connections 2024-04-05T23:34:50.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.368+0000 7f05de000700 1 --2- 172.21.15.16:0/1217518082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d8073310 0x7f05d80736f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.368+0000 7f05de000700 1 --2- 172.21.15.16:0/1217518082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d810d670 0x7f05d810fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.368+0000 7f05de000700 1 --2- 172.21.15.16:0/1217518082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d8073c30 0x7f05d810d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.368+0000 7f05de000700 1 -- 172.21.15.16:0/1217518082 >> 172.21.15.16:0/1217518082 conn(0x7f05d80fc900 msgr2=0x7f05d80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:50.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.368+0000 7f05de000700 1 -- 172.21.15.16:0/1217518082 shutdown_connections 2024-04-05T23:34:50.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.368+0000 7f05de000700 1 -- 172.21.15.16:0/1217518082 wait complete. 2024-04-05T23:34:50.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.369+0000 7f05de000700 1 Processor -- start 2024-04-05T23:34:50.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05de000700 1 -- start start 2024-04-05T23:34:50.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05de000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8073310 0x7f05d81994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05de000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d8073c30 0x7f05d8199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05de000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d810d670 0x7f05d819ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05de000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d819e400 con 0x7f05d8073c30 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05de000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d819e570 con 0x7f05d8073310 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05de000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d819e730 con 0x7f05d810d670 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05d7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8073310 0x7f05d81994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05d7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8073310 0x7f05d81994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47554/0 (socket says 172.21.15.16:47554) 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05d7400700 1 -- 172.21.15.16:0/202415264 learned_addr learned my addr 172.21.15.16:0/202415264 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.370+0000 7f05cfe00700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d8073c30 0x7f05d8199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d7e00700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d810d670 0x7f05d819ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d7400700 1 -- 172.21.15.16:0/202415264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d810d670 msgr2=0x7f05d819ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d7400700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d810d670 0x7f05d819ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d7400700 1 -- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d8073c30 msgr2=0x7f05d8199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d7400700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d8073c30 0x7f05d8199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d7400700 1 -- 172.21.15.16:0/202415264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c80095f0 con 0x7f05d8073310 2024-04-05T23:34:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d7400700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8073310 0x7f05d81994e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f05c8005770 tx=0x7f05c80057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d4c00700 1 -- 172.21.15.16:0/202415264 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c801d070 con 0x7f05d8073310 2024-04-05T23:34:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d4c00700 1 -- 172.21.15.16:0/202415264 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05c8021410 con 0x7f05d8073310 2024-04-05T23:34:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05d4c00700 1 -- 172.21.15.16:0/202415264 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8017650 con 0x7f05d8073310 2024-04-05T23:34:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.371+0000 7f05de000700 1 -- 172.21.15.16:0/202415264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d819e9b0 con 0x7f05d8073310 2024-04-05T23:34:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.372+0000 7f05de000700 1 -- 172.21.15.16:0/202415264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d819ec30 con 0x7f05d8073310 2024-04-05T23:34:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.375+0000 7f05d4c00700 1 -- 172.21.15.16:0/202415264 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f05c800fb00 con 0x7f05d8073310 2024-04-05T23:34:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05f1a6ab80 1 Processor -- start 2024-04-05T23:34:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05f1a6ab80 1 -- start start 2024-04-05T23:34:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05d4c00700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f05bc0a13b0 0x7f05bc0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05d4c00700 1 -- 172.21.15.16:0/202415264 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f05c80ce220 con 0x7f05d8073310 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5adc180 0x5561e5adc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 0x5561e5ad6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5a3cbe0 con 0x5561e5adc180 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5a2d990 con 0x5561e5ad6ca0 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5a2b350 con 0x5561e5ad62e0 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.376+0000 7f05d7e00700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f05bc0a13b0 0x7f05bc0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5adc180 0x5561e5adc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5adc180 0x5561e5adc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59214/0 (socket says 172.21.15.16:59214) 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7400700 1 -- 172.21.15.16:0/748485922 learned_addr learned my addr 172.21.15.16:0/748485922 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05cfe00700 1 --2- 172.21.15.16:0/748485922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 0x5561e5ad6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7400700 1 -- 172.21.15.16:0/748485922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 msgr2=0x5561e5ad6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7400700 1 --2- 172.21.15.16:0/748485922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 0x5561e5ad6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7400700 1 -- 172.21.15.16:0/748485922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 msgr2=0x5561e5ae2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7400700 1 --2- 172.21.15.16:0/748485922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7400700 1 -- 172.21.15.16:0/748485922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561e58a83b0 con 0x5561e5adc180 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7e00700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f05bc0a13b0 0x7f05bc0a3870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f05d0007b50 tx=0x7f05d0009200 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05d7400700 1 --2- 172.21.15.16:0/748485922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5adc180 0x5561e5adc560 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f05c809ae40 tx=0x7f05c809ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2138399d5ff50e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.377+0000 7f05c7e00700 1 -- 172.21.15.16:0/748485922 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c80ce050 con 0x5561e5adc180 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05c7e00700 1 -- 172.21.15.16:0/748485922 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05c809fe90 con 0x5561e5adc180 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05c7e00700 1 -- 172.21.15.16:0/748485922 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c80a10d0 con 0x5561e5adc180 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/748485922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5adc180 msgr2=0x5561e5adc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/748485922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5adc180 0x5561e5adc560 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f05c809ae40 tx=0x7f05c809ad20 comp rx=0 tx=0).stop 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/748485922 shutdown_connections 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/748485922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5adc180 0x5561e5adc560 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/748485922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/748485922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 0x5561e5ad6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/748485922 >> 172.21.15.16:0/748485922 conn(0x5561e5a4a220 msgr2=0x5561e5ad1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:50.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/748485922 shutdown_connections 2024-04-05T23:34:50.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.378+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/748485922 wait complete. 2024-04-05T23:34:50.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.379+0000 7f05f1a6ab80 1 -- start start 2024-04-05T23:34:50.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.379+0000 7f05f1a6ab80 1 Processor -- start 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5ae5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 0x5561e5adff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5a26c40 con 0x5561e5ad62e0 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e59c2d80 con 0x5561e5ad6ca0 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e59c2a20 con 0x5561e5adc180 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05f1a6ab80 10 client.0 _ll_register_callbacks cb 0x5561e582e3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05d7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5ae5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05cfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05cfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47564/0 (socket says 172.21.15.16:47564) 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05cfe00700 1 -- 172.21.15.16:0/1541087050 learned_addr learned my addr 172.21.15.16:0/1541087050 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05d7e00700 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 0x5561e5adff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05cfe00700 1 -- 172.21.15.16:0/1541087050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 msgr2=0x5561e5adff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05cfe00700 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 0x5561e5adff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05cfe00700 1 -- 172.21.15.16:0/1541087050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 msgr2=0x5561e5ae5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05cfe00700 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5ae5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05cfe00700 1 -- 172.21.15.16:0/1541087050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d819e9b0 con 0x5561e5ad6ca0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.380+0000 7f05d7400700 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5ae5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.381+0000 7f05cfe00700 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae6150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f05c0012840 tx=0x7f05c0012b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1da6cf00ee50096 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.381+0000 7f05c4c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.381+0000 7f05c4c00700 1 -- 172.21.15.16:0/1541087050 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0017070 con 0x5561e5ad6ca0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.381+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1541087050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561e5ac9490 con 0x5561e5ad6ca0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.381+0000 7f05c4c00700 1 -- 172.21.15.16:0/1541087050 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f05c0004830 con 0x5561e5ad6ca0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.381+0000 7f05c4c00700 1 -- 172.21.15.16:0/1541087050 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05c0004a30 con 0x5561e5ad6ca0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.381+0000 7f05f1a6ab80 10 client.25700 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.381+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1541087050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5561e5854b60 con 0x5561e5ad6ca0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.382+0000 7f05c4c00700 1 -- 172.21.15.16:0/1541087050 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5561e5ac9490 con 0x5561e5ad6ca0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.382+0000 7f05c4c00700 1 -- 172.21.15.16:0/1541087050 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f05c000f4e0 con 0x5561e5ad6ca0 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.382+0000 7f05f1a6ab80 10 client.25700 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.382+0000 7f05f1a6ab80 10 client.25700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:50.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.382+0000 7f05f1a6ab80 20 client.25700 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:50.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.382+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5561e5af00f0 0x5561e5af2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.382+0000 7f05f1a6ab80 4 client.25700 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:50.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.382+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1541087050 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561e5a1f700 con 0x5561e5af00f0 2024-04-05T23:34:50.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.382+0000 7f05d7400700 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5561e5af00f0 0x5561e5af2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.383+0000 7f05d7400700 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5561e5af00f0 0x5561e5af2510 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e88e9a50b2338d73 server_cookie=db1d54d3d8e67287 in_seq=0 out_seq=0 2024-04-05T23:34:50.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.383+0000 7f05c4c00700 10 client.25700 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:50.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.383+0000 7f05c4c00700 1 -- 172.21.15.16:0/1541087050 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5561e5a1f700 con 0x5561e5af00f0 2024-04-05T23:34:50.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.383+0000 7f05c4c00700 10 client.25700 handle_command_reply: tid=0 2024-04-05T23:34:50.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 client.25700 shutdown 2024-04-05T23:34:50.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1541087050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 msgr2=0x5561e5ae6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae6150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f05c0012840 tx=0x7f05c0012b50 comp rx=0 tx=0).stop 2024-04-05T23:34:50.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1541087050 shutdown_connections 2024-04-05T23:34:50.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5561e5af00f0 0x5561e5af2510 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5ae5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 0x5561e5adff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1541087050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae6150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05c4c00700 0 client.25700 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:50.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1541087050 >> 172.21.15.16:0/1541087050 conn(0x5561e5a4a220 msgr2=0x5561e5acb460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:50.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1541087050 shutdown_connections 2024-04-05T23:34:50.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1541087050 wait complete. 2024-04-05T23:34:50.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.384+0000 7f05f1a6ab80 20 client.25700 trim_cache size 0 max 16384 2024-04-05T23:34:50.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.401+0000 7f05f1a6ab80 1 Processor -- start 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.401+0000 7f05f1a6ab80 1 -- start start 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 0x5561e5ad16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5adc180 0x5561e5acb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e58aa6a0 con 0x5561e5ad6ca0 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5a1f700 con 0x5561e5adc180 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5ad7080 con 0x5561e5ad62e0 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05cfe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05cfe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59236/0 (socket says 172.21.15.16:59236) 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05cfe00700 1 -- 172.21.15.16:0/312234216 learned_addr learned my addr 172.21.15.16:0/312234216 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05d7e00700 1 --2- 172.21.15.16:0/312234216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5adc180 0x5561e5acb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.402+0000 7f05d7400700 1 --2- 172.21.15.16:0/312234216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 0x5561e5ad16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.403+0000 7f05cfe00700 1 -- 172.21.15.16:0/312234216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 msgr2=0x5561e5ad16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.403+0000 7f05cfe00700 1 --2- 172.21.15.16:0/312234216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 0x5561e5ad16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.403+0000 7f05cfe00700 1 -- 172.21.15.16:0/312234216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5adc180 msgr2=0x5561e5acb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.403+0000 7f05cfe00700 1 --2- 172.21.15.16:0/312234216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5adc180 0x5561e5acb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.403+0000 7f05cfe00700 1 -- 172.21.15.16:0/312234216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561e59ecea0 con 0x5561e5ad6ca0 2024-04-05T23:34:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.403+0000 7f05cfe00700 1 --2- 172.21.15.16:0/312234216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae5f70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f05c0000c00 tx=0x7f05c000fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54955954dfe3aeee server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.403+0000 7f05cd200700 1 -- 172.21.15.16:0/312234216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0017070 con 0x5561e5ad6ca0 2024-04-05T23:34:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.403+0000 7f05cd200700 1 -- 172.21.15.16:0/312234216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05c0018040 con 0x5561e5ad6ca0 2024-04-05T23:34:50.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.403+0000 7f05cd200700 1 -- 172.21.15.16:0/312234216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0005290 con 0x5561e5ad6ca0 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/312234216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad6ca0 msgr2=0x5561e5ae5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/312234216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae5f70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f05c0000c00 tx=0x7f05c000fa90 comp rx=0 tx=0).stop 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/312234216 shutdown_connections 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/312234216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5adc180 0x5561e5acb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/312234216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5ad62e0 0x5561e5ad16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/312234216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad6ca0 0x5561e5ae5f70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/312234216 >> 172.21.15.16:0/312234216 conn(0x5561e5a4a220 msgr2=0x5561e5ae5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/312234216 shutdown_connections 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/312234216 wait complete. 2024-04-05T23:34:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.404+0000 7f05f1a6ab80 1 -- start start 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05f1a6ab80 1 Processor -- start 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5aec720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5aeabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05f1a6ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 0x5561e5aeb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5aeba10 con 0x5561e5ad62e0 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5b43f30 con 0x5561e5ad6ca0 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05f1a6ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5b44070 con 0x5561e5adc180 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05f1a6ab80 10 client.0 _ll_register_callbacks cb 0x7f05c0009e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05d7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5aec720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05d7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5aec720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59246/0 (socket says 172.21.15.16:59246) 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05d7400700 1 -- 172.21.15.16:0/1198563051 learned_addr learned my addr 172.21.15.16:0/1198563051 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05cfe00700 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5aeabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.405+0000 7f05d7e00700 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 0x5561e5aeb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05d7400700 1 -- 172.21.15.16:0/1198563051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 msgr2=0x5561e5aeb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:50.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05d7400700 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 0x5561e5aeb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05d7400700 1 -- 172.21.15.16:0/1198563051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 msgr2=0x5561e5aeabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05d7400700 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5aeabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05d7400700 1 -- 172.21.15.16:0/1198563051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561e5854b60 con 0x5561e5ad62e0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05d7400700 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5aec720 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f05c8005cb0 tx=0x7f05c80094b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e257e0c867ec40 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05c4c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05c4c00700 1 -- 172.21.15.16:0/1198563051 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c80b9040 con 0x5561e5ad62e0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05c4c00700 1 -- 172.21.15.16:0/1198563051 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f05c809b840 con 0x5561e5ad62e0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1198563051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05c809fe90 con 0x5561e5ad62e0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05c4c00700 1 -- 172.21.15.16:0/1198563051 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05c80a3530 con 0x5561e5ad62e0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05f1a6ab80 10 client.15672 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.406+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1198563051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5561e58accd0 con 0x5561e5ad62e0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.407+0000 7f05c4c00700 1 -- 172.21.15.16:0/1198563051 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05c809fe90 con 0x5561e5ad62e0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.407+0000 7f05c4c00700 1 -- 172.21.15.16:0/1198563051 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f05c809a630 con 0x5561e5ad62e0 2024-04-05T23:34:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.407+0000 7f05f1a6ab80 10 client.15672 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.407+0000 7f05f1a6ab80 10 client.15672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.407+0000 7f05f1a6ab80 20 client.15672 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.407+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5561e5b453f0 0x5561e5b47810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.407+0000 7f05f1a6ab80 4 client.15672 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.407+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1198563051 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5561e5875cf0 con 0x5561e5b453f0 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.407+0000 7f05cfe00700 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5561e5b453f0 0x5561e5b47810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05cfe00700 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5561e5b453f0 0x5561e5b47810 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=715115123f3a4b54 server_cookie=1d8e236647340f19 in_seq=0 out_seq=0 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05c4c00700 10 client.15672 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05c4c00700 1 -- 172.21.15.16:0/1198563051 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5561e5875cf0 con 0x5561e5b453f0 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05c4c00700 10 client.15672 handle_command_reply: tid=0 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 client.15672 shutdown 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1198563051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 msgr2=0x5561e5aec720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5aec720 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f05c8005cb0 tx=0x7f05c80094b0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1198563051 shutdown_connections 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5561e5b453f0 0x5561e5b47810 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561e5ad62e0 0x5561e5aec720 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561e5adc180 0x5561e5aeb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05c4c00700 0 client.15672 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 --2- 172.21.15.16:0/1198563051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561e5ad6ca0 0x5561e5aeabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1198563051 >> 172.21.15.16:0/1198563051 conn(0x5561e5a4a220 msgr2=0x5561e5adb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1198563051 shutdown_connections 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.408+0000 7f05f1a6ab80 1 -- 172.21.15.16:0/1198563051 wait complete. 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05f1a6ab80 20 client.15672 trim_cache size 0 max 16384 2024-04-05T23:34:50.406 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:34:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-04-05T23:34:50.407 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-04-05T23:34:50.407 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:34:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 -- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f05bc0a13b0 msgr2=0x7f05bc0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f05bc0a13b0 0x7f05bc0a3870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f05d0007b50 tx=0x7f05d0009200 comp rx=0 tx=0).stop 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 -- 172.21.15.16:0/202415264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8073310 msgr2=0x7f05d81994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8073310 0x7f05d81994e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f05c8005770 tx=0x7f05c80057a0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 -- 172.21.15.16:0/202415264 shutdown_connections 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f05bc0a13b0 0x7f05bc0a3870 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d810d670 0x7f05d819ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d8073310 0x7f05d81994e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 --2- 172.21.15.16:0/202415264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d8073c30 0x7f05d8199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 -- 172.21.15.16:0/202415264 >> 172.21.15.16:0/202415264 conn(0x7f05d80fc900 msgr2=0x7f05d80fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 -- 172.21.15.16:0/202415264 shutdown_connections 2024-04-05T23:34:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.409+0000 7f05de000700 1 -- 172.21.15.16:0/202415264 wait complete. 2024-04-05T23:34:50.420 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-05T23:34:50.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:34:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.627+0000 7f3926000700 1 -- 172.21.15.16:0/144378195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3920100f90 msgr2=0x7f3920101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.627+0000 7f3926000700 1 --2- 172.21.15.16:0/144378195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3920100f90 0x7f3920101370 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3910009990 tx=0x7f3910009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.627+0000 7f3926000700 1 -- 172.21.15.16:0/144378195 shutdown_connections 2024-04-05T23:34:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.627+0000 7f3926000700 1 --2- 172.21.15.16:0/144378195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39201060b0 0x7f392010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.627+0000 7f3926000700 1 --2- 172.21.15.16:0/144378195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3920100f90 0x7f3920101370 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.627+0000 7f3926000700 1 --2- 172.21.15.16:0/144378195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3920101940 0x7f3920105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.627+0000 7f3926000700 1 -- 172.21.15.16:0/144378195 >> 172.21.15.16:0/144378195 conn(0x7f39200fc830 msgr2=0x7f39200fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:50.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.627+0000 7f3926000700 1 -- 172.21.15.16:0/144378195 shutdown_connections 2024-04-05T23:34:50.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.627+0000 7f3926000700 1 -- 172.21.15.16:0/144378195 wait complete. 2024-04-05T23:34:50.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.628+0000 7f3926000700 1 Processor -- start 2024-04-05T23:34:50.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.628+0000 7f3926000700 1 -- start start 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f3926000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3920100f90 0x7f39201972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f3926000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39201060b0 0x7f3920197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f3926000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392019bb60 0x7f392019bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f3926000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392019c5e0 con 0x7f392019bb60 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f3926000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392019c750 con 0x7f3920100f90 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f3926000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392019c910 con 0x7f39201060b0 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f3917e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39201060b0 0x7f3920197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f391fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392019bb60 0x7f392019bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f391f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3920100f90 0x7f39201972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f3917e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39201060b0 0x7f3920197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:56380/0 (socket says 172.21.15.16:56380) 2024-04-05T23:34:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.629+0000 7f3917e00700 1 -- 172.21.15.16:0/3855392460 learned_addr learned my addr 172.21.15.16:0/3855392460 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f3917e00700 1 -- 172.21.15.16:0/3855392460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3920100f90 msgr2=0x7f39201972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f3917e00700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3920100f90 0x7f39201972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f3917e00700 1 -- 172.21.15.16:0/3855392460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392019bb60 msgr2=0x7f392019bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f3917e00700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392019bb60 0x7f392019bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f3917e00700 1 -- 172.21.15.16:0/3855392460 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39100095f0 con 0x7f39201060b0 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f391f400700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3920100f90 0x7f39201972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f3917e00700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39201060b0 0x7f3920197810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f390800e970 tx=0x7f390800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f391cc00700 1 -- 172.21.15.16:0/3855392460 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908009790 con 0x7f39201060b0 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f391cc00700 1 -- 172.21.15.16:0/3855392460 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f390800c7e0 con 0x7f39201060b0 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f391cc00700 1 -- 172.21.15.16:0/3855392460 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39080054e0 con 0x7f39201060b0 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.630+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3920073170 con 0x7f39201060b0 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.631+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3920073630 con 0x7f39201060b0 2024-04-05T23:34:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.631+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39201092d0 con 0x7f39201060b0 2024-04-05T23:34:50.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.634+0000 7f391cc00700 1 -- 172.21.15.16:0/3855392460 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f390800c950 con 0x7f39201060b0 2024-04-05T23:34:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.635+0000 7f391cc00700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f39040a1430 0x7f39040a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.635+0000 7f391cc00700 1 -- 172.21.15.16:0/3855392460 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f3908014070 con 0x7f39201060b0 2024-04-05T23:34:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.636+0000 7f391f400700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f39040a1430 0x7f39040a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.636+0000 7f391f400700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f39040a1430 0x7f39040a38f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3910006010 tx=0x7f391000b260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:50.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.638+0000 7f391cc00700 1 -- 172.21.15.16:0/3855392460 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f39080965e0 con 0x7f39201060b0 2024-04-05T23:34:50.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.907+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3920102460 con 0x7f39201060b0 2024-04-05T23:34:50.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.908+0000 7f391cc00700 1 -- 172.21.15.16:0/3855392460 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+10903 (secure 0 0 0) 0x7f39080b1ab0 con 0x7f39201060b0 2024-04-05T23:34:50.900 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:34:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:34:47.757632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_25534":{"gid":25534,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/1942303111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1942303111},{"type":"v1","addr":"172.21.15.130:6835","nonce":1942303111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f39040a1430 msgr2=0x7f39040a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f39040a1430 0x7f39040a38f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3910006010 tx=0x7f391000b260 comp rx=0 tx=0).stop 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39201060b0 msgr2=0x7f3920197810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39201060b0 0x7f3920197810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f390800e970 tx=0x7f390800ec80 comp rx=0 tx=0).stop 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 shutdown_connections 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f39040a1430 0x7f39040a38f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392019bb60 0x7f392019bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3920100f90 0x7f39201972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 --2- 172.21.15.16:0/3855392460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39201060b0 0x7f3920197810 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 >> 172.21.15.16:0/3855392460 conn(0x7f39200fc830 msgr2=0x7f39200fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 shutdown_connections 2024-04-05T23:34:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:50.910+0000 7f3926000700 1 -- 172.21.15.16:0/3855392460 wait complete. 2024-04-05T23:34:50.904 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 31 2024-04-05T23:34:50.923 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-05T23:34:50.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21398 13.42 MB/sec warmup 10 sec latency 2570.698 ms 2024-04-05T23:34:51.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.133+0000 7fcf46200700 1 -- 172.21.15.16:0/207386728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401061e0 msgr2=0x7fcf4010dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.133+0000 7fcf46200700 1 --2- 172.21.15.16:0/207386728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401061e0 0x7fcf4010dd60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800b0a0 tx=0x7fcf3800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.133+0000 7fcf46200700 1 -- 172.21.15.16:0/207386728 shutdown_connections 2024-04-05T23:34:51.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.133+0000 7fcf46200700 1 --2- 172.21.15.16:0/207386728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401061e0 0x7fcf4010dd60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.133+0000 7fcf46200700 1 --2- 172.21.15.16:0/207386728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf401010c0 0x7fcf401014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.133+0000 7fcf46200700 1 --2- 172.21.15.16:0/207386728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf40101a70 0x7fcf40105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.133+0000 7fcf46200700 1 -- 172.21.15.16:0/207386728 >> 172.21.15.16:0/207386728 conn(0x7fcf400fc920 msgr2=0x7fcf400fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.133+0000 7fcf46200700 1 -- 172.21.15.16:0/207386728 shutdown_connections 2024-04-05T23:34:51.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.133+0000 7fcf46200700 1 -- 172.21.15.16:0/207386728 wait complete. 2024-04-05T23:34:51.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.134+0000 7fcf46200700 1 Processor -- start 2024-04-05T23:34:51.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.134+0000 7fcf46200700 1 -- start start 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf46200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401010c0 0x7fcf400731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf46200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf401061e0 0x7fcf40073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf46200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf40073c60 0x7fcf401a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf46200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf401a5c50 con 0x7fcf401010c0 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf46200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf401a5dc0 con 0x7fcf40073c60 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf46200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf401a5f80 con 0x7fcf401061e0 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401010c0 0x7fcf400731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401010c0 0x7fcf400731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59276/0 (socket says 172.21.15.16:59276) 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3f400700 1 -- 172.21.15.16:0/3498177461 learned_addr learned my addr 172.21.15.16:0/3498177461 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf401061e0 0x7fcf40073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3fe00700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf40073c60 0x7fcf401a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3f400700 1 -- 172.21.15.16:0/3498177461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf401061e0 msgr2=0x7fcf40073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3f400700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf401061e0 0x7fcf40073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3f400700 1 -- 172.21.15.16:0/3498177461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf40073c60 msgr2=0x7fcf401a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3f400700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf40073c60 0x7fcf401a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3f400700 1 -- 172.21.15.16:0/3498177461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38009d00 con 0x7fcf401010c0 2024-04-05T23:34:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.135+0000 7fcf3fe00700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf40073c60 0x7fcf401a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.136+0000 7fcf3f400700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401010c0 0x7fcf400731e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000e9a0 tx=0x7fcf3000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.136+0000 7fcf37e00700 1 -- 172.21.15.16:0/3498177461 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30009790 con 0x7fcf401010c0 2024-04-05T23:34:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.136+0000 7fcf37e00700 1 -- 172.21.15.16:0/3498177461 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf3000cbd0 con 0x7fcf401010c0 2024-04-05T23:34:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.136+0000 7fcf37e00700 1 -- 172.21.15.16:0/3498177461 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30018680 con 0x7fcf401010c0 2024-04-05T23:34:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.136+0000 7fcf46200700 1 -- 172.21.15.16:0/3498177461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf401a6260 con 0x7fcf401010c0 2024-04-05T23:34:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.136+0000 7fcf46200700 1 -- 172.21.15.16:0/3498177461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf401a6720 con 0x7fcf401010c0 2024-04-05T23:34:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.141+0000 7fcf37e00700 1 -- 172.21.15.16:0/3498177461 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7fcf3000cd40 con 0x7fcf401010c0 2024-04-05T23:34:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.141+0000 7fcf59d56b80 1 Processor -- start 2024-04-05T23:34:51.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf59d56b80 1 -- start start 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b3a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b342e0 0x556d26b34760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b34ca0 0x556d26b40f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26a9abe0 con 0x556d26b3a180 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26a8b990 con 0x556d26b342e0 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26a89350 con 0x556d26b34ca0 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf37e00700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcf240a13b0 0x7fcf240a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf37e00700 1 -- 172.21.15.16:0/3498177461 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fcf30014070 con 0x7fcf401010c0 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf3ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b34ca0 0x556d26b40f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf3f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b342e0 0x556d26b34760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf3fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b3a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf3ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b34ca0 0x556d26b40f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:56412/0 (socket says 172.21.15.16:56412) 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf3fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b3a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59284/0 (socket says 172.21.15.16:59284) 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.142+0000 7fcf3f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b342e0 0x556d26b34760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47624/0 (socket says 172.21.15.16:47624) 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3ea00700 1 -- 172.21.15.16:0/3333227779 learned_addr learned my addr 172.21.15.16:0/3333227779 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3fe00700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcf240a13b0 0x7fcf240a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3ea00700 1 -- 172.21.15.16:0/3333227779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b342e0 msgr2=0x556d26b34760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/3333227779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b342e0 0x556d26b34760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3ea00700 1 -- 172.21.15.16:0/3333227779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 msgr2=0x556d26b3a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/3333227779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b3a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3ea00700 1 -- 172.21.15.16:0/3333227779 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d269063b0 con 0x556d26b34ca0 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3f400700 1 --2- 172.21.15.16:0/3333227779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b342e0 0x556d26b34760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/3333227779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b34ca0 0x556d26b40f10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009c60 tx=0x7fcf2c009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4d28776a0302622 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.143+0000 7fcf3fe00700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcf240a13b0 0x7fcf240a3870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf38009cd0 tx=0x7fcf380099e0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf2be00700 1 -- 172.21.15.16:0/3333227779 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c009730 con 0x556d26b34ca0 2024-04-05T23:34:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf2be00700 1 -- 172.21.15.16:0/3333227779 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf2c015070 con 0x556d26b34ca0 2024-04-05T23:34:51.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf2be00700 1 -- 172.21.15.16:0/3333227779 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c003cc0 con 0x556d26b34ca0 2024-04-05T23:34:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3333227779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b34ca0 msgr2=0x556d26b40f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3333227779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b34ca0 0x556d26b40f10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009c60 tx=0x7fcf2c009f70 comp rx=0 tx=0).stop 2024-04-05T23:34:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3333227779 shutdown_connections 2024-04-05T23:34:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3333227779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b3a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3333227779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b34ca0 0x556d26b40f10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3333227779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b342e0 0x556d26b34760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3333227779 >> 172.21.15.16:0/3333227779 conn(0x556d26aa8220 msgr2=0x556d26b2f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3333227779 shutdown_connections 2024-04-05T23:34:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.144+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3333227779 wait complete. 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.145+0000 7fcf59d56b80 1 -- start start 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.145+0000 7fcf59d56b80 1 Processor -- start 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b40e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b3e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 0x556d26b3e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26a84c40 con 0x556d26b34ca0 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26a20d80 con 0x556d26b3a180 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26a20a20 con 0x556d26b342e0 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf59d56b80 10 client.0 _ll_register_callbacks cb 0x556d2688c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b3e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b3e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59300/0 (socket says 172.21.15.16:59300) 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3ea00700 1 -- 172.21.15.16:0/3203663260 learned_addr learned my addr 172.21.15.16:0/3203663260 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3f400700 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b40e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3fe00700 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 0x556d26b3e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3ea00700 1 -- 172.21.15.16:0/3203663260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 msgr2=0x556d26b40e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b40e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3ea00700 1 -- 172.21.15.16:0/3203663260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 msgr2=0x556d26b3e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 0x556d26b3e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.146+0000 7fcf3ea00700 1 -- 172.21.15.16:0/3203663260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf401a6720 con 0x556d26b34ca0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.147+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b3e0e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c005910 tx=0x7fcf2c005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d69b64c18f0941 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.147+0000 7fcf28c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.147+0000 7fcf28c00700 1 -- 172.21.15.16:0/3203663260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c020070 con 0x556d26b34ca0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.147+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3203663260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d26b27490 con 0x556d26b34ca0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.147+0000 7fcf28c00700 1 -- 172.21.15.16:0/3203663260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fcf2c017070 con 0x556d26b34ca0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.147+0000 7fcf28c00700 1 -- 172.21.15.16:0/3203663260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf2c003680 con 0x556d26b34ca0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.147+0000 7fcf59d56b80 10 client.15690 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.147+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3203663260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7fcf2c009890 con 0x556d26b34ca0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.147+0000 7fcf28c00700 1 -- 172.21.15.16:0/3203663260 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf2c031080 con 0x556d26b34ca0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.148+0000 7fcf28c00700 1 -- 172.21.15.16:0/3203663260 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7fcf2c011670 con 0x556d26b34ca0 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.148+0000 7fcf59d56b80 10 client.15690 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.148+0000 7fcf59d56b80 10 client.15690 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-05T23:34:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.148+0000 7fcf59d56b80 20 client.15690 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.148+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556d26b4e390 0x556d26b507b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.148+0000 7fcf59d56b80 4 client.15690 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.148+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3203663260 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d26a7d700 con 0x556d26b4e390 2024-04-05T23:34:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.148+0000 7fcf3f400700 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556d26b4e390 0x556d26b507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.149+0000 7fcf3f400700 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556d26b4e390 0x556d26b507b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=acd18a592e60472f server_cookie=303de4419db23437 in_seq=0 out_seq=0 2024-04-05T23:34:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.149+0000 7fcf28c00700 10 client.15690 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:34:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.149+0000 7fcf28c00700 1 -- 172.21.15.16:0/3203663260 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556d26a7d700 con 0x556d26b4e390 2024-04-05T23:34:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.149+0000 7fcf28c00700 10 client.15690 handle_command_reply: tid=0 2024-04-05T23:34:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.149+0000 7fcf59d56b80 1 client.15690 shutdown 2024-04-05T23:34:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3203663260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 msgr2=0x556d26b3e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b3e0e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c005910 tx=0x7fcf2c005270 comp rx=0 tx=0).stop 2024-04-05T23:34:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3203663260 shutdown_connections 2024-04-05T23:34:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 0x556d26b3e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b40e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556d26b4e390 0x556d26b507b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/3203663260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b3e0e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf28c00700 0 client.15690 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:34:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3203663260 >> 172.21.15.16:0/3203663260 conn(0x556d26aa8220 msgr2=0x556d26b29460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3203663260 shutdown_connections 2024-04-05T23:34:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 1 -- 172.21.15.16:0/3203663260 wait complete. 2024-04-05T23:34:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.150+0000 7fcf59d56b80 20 client.15690 trim_cache size 0 max 16384 2024-04-05T23:34:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.163+0000 7fcf59d56b80 1 Processor -- start 2024-04-05T23:34:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.163+0000 7fcf59d56b80 1 -- start start 2024-04-05T23:34:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b3e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b29560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 0x556d26b376e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d269086a0 con 0x556d26b34ca0 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26b3c650 con 0x556d26b3a180 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26b2b1a0 con 0x556d26b342e0 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b29560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b29560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59304/0 (socket says 172.21.15.16:59304) 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3ea00700 1 -- 172.21.15.16:0/2717597312 learned_addr learned my addr 172.21.15.16:0/2717597312 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3fe00700 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 0x556d26b376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3f400700 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b3e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3ea00700 1 -- 172.21.15.16:0/2717597312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 msgr2=0x556d26b3e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b3e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3ea00700 1 -- 172.21.15.16:0/2717597312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 msgr2=0x556d26b376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 0x556d26b376e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3ea00700 1 -- 172.21.15.16:0/2717597312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d268b2b60 con 0x556d26b34ca0 2024-04-05T23:34:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.164+0000 7fcf3f400700 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b3e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.165+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b29560 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009b80 tx=0x7fcf2c0053d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb5849023516625 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.165+0000 7fcf35200700 1 -- 172.21.15.16:0/2717597312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c020070 con 0x556d26b34ca0 2024-04-05T23:34:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.165+0000 7fcf35200700 1 -- 172.21.15.16:0/2717597312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556d26b27490 con 0x556d26b34ca0 2024-04-05T23:34:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.165+0000 7fcf35200700 1 -- 172.21.15.16:0/2717597312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c004a60 con 0x556d26b34ca0 2024-04-05T23:34:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.165+0000 7fcf59d56b80 1 -- 172.21.15.16:0/2717597312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 msgr2=0x556d26b29560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.165+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b29560 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009b80 tx=0x7fcf2c0053d0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.166+0000 7fcf59d56b80 1 -- 172.21.15.16:0/2717597312 shutdown_connections 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.166+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b3a180 0x556d26b376e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.166+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b3e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.166+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/2717597312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b34ca0 0x556d26b29560 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.166+0000 7fcf59d56b80 1 -- 172.21.15.16:0/2717597312 >> 172.21.15.16:0/2717597312 conn(0x556d26aa8220 msgr2=0x556d26b3d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.166+0000 7fcf59d56b80 1 -- 172.21.15.16:0/2717597312 shutdown_connections 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.166+0000 7fcf59d56b80 1 -- 172.21.15.16:0/2717597312 wait complete. 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.166+0000 7fcf59d56b80 1 -- start start 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.166+0000 7fcf59d56b80 1 Processor -- start 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b414e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b34ca0 0x556d26b41a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf59d56b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b41f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26b45910 con 0x556d26b3a180 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26b45a80 con 0x556d26b34ca0 2024-04-05T23:34:51.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf59d56b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d26ba1170 con 0x556d26b342e0 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf59d56b80 10 client.0 _ll_register_callbacks cb 0x7fcf30005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b41f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b41f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59310/0 (socket says 172.21.15.16:59310) 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3fe00700 1 -- 172.21.15.16:0/1253284876 learned_addr learned my addr 172.21.15.16:0/1253284876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3ea00700 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b34ca0 0x556d26b41a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3f400700 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3fe00700 1 -- 172.21.15.16:0/1253284876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 msgr2=0x556d26b414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3fe00700 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3fe00700 1 -- 172.21.15.16:0/1253284876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b34ca0 msgr2=0x556d26b41a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3fe00700 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b34ca0 0x556d26b41a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.167+0000 7fcf3fe00700 1 -- 172.21.15.16:0/1253284876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c009890 con 0x556d26b3a180 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.168+0000 7fcf3fe00700 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b41f60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800b0a0 tx=0x7fcf3800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611de4d1debd04eb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.168+0000 7fcf28c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.168+0000 7fcf28c00700 1 -- 172.21.15.16:0/1253284876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3800e070 con 0x556d26b3a180 2024-04-05T23:34:51.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.168+0000 7fcf28c00700 1 -- 172.21.15.16:0/1253284876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fcf380050a0 con 0x556d26b3a180 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.168+0000 7fcf59d56b80 1 -- 172.21.15.16:0/1253284876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf2c015070 con 0x556d26b3a180 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.168+0000 7fcf28c00700 1 -- 172.21.15.16:0/1253284876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf38005260 con 0x556d26b3a180 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.168+0000 7fcf59d56b80 10 client.15696 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.168+0000 7fcf59d56b80 1 -- 172.21.15.16:0/1253284876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556d26a4aea0 con 0x556d26b3a180 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.169+0000 7fcf28c00700 1 -- 172.21.15.16:0/1253284876 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf2c015070 con 0x556d26b3a180 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.169+0000 7fcf28c00700 1 -- 172.21.15.16:0/1253284876 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7fcf3802a040 con 0x556d26b3a180 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.169+0000 7fcf59d56b80 10 client.15696 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.169+0000 7fcf59d56b80 10 client.15696 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.169+0000 7fcf59d56b80 20 client.15696 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.169+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556d26ba2990 0x556d26ba4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.169+0000 7fcf59d56b80 4 client.15696 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.169+0000 7fcf59d56b80 1 -- 172.21.15.16:0/1253284876 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556d268d3cf0 con 0x556d26ba2990 2024-04-05T23:34:51.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.169+0000 7fcf3f400700 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556d26ba2990 0x556d26ba4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf3f400700 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556d26ba2990 0x556d26ba4db0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6b8f3b15e2a50cef server_cookie=f9d04f2a4ebd742b in_seq=0 out_seq=0 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf28c00700 10 client.15696 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf28c00700 1 -- 172.21.15.16:0/1253284876 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556d268d3cf0 con 0x556d26ba2990 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf28c00700 10 client.15696 handle_command_reply: tid=0 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 client.15696 shutdown 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 -- 172.21.15.16:0/1253284876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 msgr2=0x556d26b41f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b41f60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800b0a0 tx=0x7fcf3800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 -- 172.21.15.16:0/1253284876 shutdown_connections 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x556d26ba2990 0x556d26ba4db0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d26b3a180 0x556d26b41f60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556d26b342e0 0x556d26b414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf28c00700 0 client.15696 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 --2- 172.21.15.16:0/1253284876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556d26b34ca0 0x556d26b41a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 -- 172.21.15.16:0/1253284876 >> 172.21.15.16:0/1253284876 conn(0x556d26aa8220 msgr2=0x556d26b39190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 -- 172.21.15.16:0/1253284876 shutdown_connections 2024-04-05T23:34:51.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 1 -- 172.21.15.16:0/1253284876 wait complete. 2024-04-05T23:34:51.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.170+0000 7fcf59d56b80 20 client.15696 trim_cache size 0 max 16384 2024-04-05T23:34:51.167 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:51.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 -- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcf240a13b0 msgr2=0x7fcf240a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcf240a13b0 0x7fcf240a3870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf38009cd0 tx=0x7fcf380099e0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 -- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401010c0 msgr2=0x7fcf400731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401010c0 0x7fcf400731e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000e9a0 tx=0x7fcf3000ed60 comp rx=0 tx=0).stop 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 -- 172.21.15.16:0/3498177461 shutdown_connections 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fcf240a13b0 0x7fcf240a3870 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf401010c0 0x7fcf400731e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf40073c60 0x7fcf401a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 --2- 172.21.15.16:0/3498177461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf401061e0 0x7fcf40073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 -- 172.21.15.16:0/3498177461 >> 172.21.15.16:0/3498177461 conn(0x7fcf400fc920 msgr2=0x7fcf400febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 -- 172.21.15.16:0/3498177461 shutdown_connections 2024-04-05T23:34:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.171+0000 7fcf46200700 1 -- 172.21.15.16:0/3498177461 wait complete. 2024-04-05T23:34:51.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-05T23:34:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.390+0000 7f697e400700 1 -- 172.21.15.16:0/3252536479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 msgr2=0x7f697810d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.390+0000 7f697e400700 1 --2- 172.21.15.16:0/3252536479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 0x7f697810d130 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6964009990 tx=0x7f6964009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.390+0000 7f697e400700 1 -- 172.21.15.16:0/3252536479 shutdown_connections 2024-04-05T23:34:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.390+0000 7f697e400700 1 --2- 172.21.15.16:0/3252536479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f697810d670 0x7f697810fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.390+0000 7f697e400700 1 --2- 172.21.15.16:0/3252536479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978073310 0x7f69780736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.390+0000 7f697e400700 1 --2- 172.21.15.16:0/3252536479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 0x7f697810d130 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.390+0000 7f697e400700 1 -- 172.21.15.16:0/3252536479 >> 172.21.15.16:0/3252536479 conn(0x7f69780fc900 msgr2=0x7f69780fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.390+0000 7f697e400700 1 -- 172.21.15.16:0/3252536479 shutdown_connections 2024-04-05T23:34:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.391+0000 7f697e400700 1 -- 172.21.15.16:0/3252536479 wait complete. 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.391+0000 7f697e400700 1 Processor -- start 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f697e400700 1 -- start start 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f697e400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978073310 0x7f6978199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f697e400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 0x7f6978199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f697e400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f697810d670 0x7f697819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f697e400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697819e480 con 0x7f6978073c30 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f697e400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697819e5f0 con 0x7f697810d670 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f697e400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697819e7b0 con 0x7f6978073310 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f6976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 0x7f6978199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f6977400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978073310 0x7f6978199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f6976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 0x7f6978199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59320/0 (socket says 172.21.15.16:59320) 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f6976a00700 1 -- 172.21.15.16:0/3981940160 learned_addr learned my addr 172.21.15.16:0/3981940160 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.392+0000 7f6977e00700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f697810d670 0x7f697819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f6976a00700 1 -- 172.21.15.16:0/3981940160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978073310 msgr2=0x7f6978199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f6976a00700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978073310 0x7f6978199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f6976a00700 1 -- 172.21.15.16:0/3981940160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f697810d670 msgr2=0x7f697819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f6976a00700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f697810d670 0x7f697819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f6976a00700 1 -- 172.21.15.16:0/3981940160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69640095f0 con 0x7f6978073c30 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f6976a00700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 0x7f6978199aa0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6964005910 tx=0x7f6964004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f696fe00700 1 -- 172.21.15.16:0/3981940160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696401d070 con 0x7f6978073c30 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f696fe00700 1 -- 172.21.15.16:0/3981940160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6964022470 con 0x7f6978073c30 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f697e400700 1 -- 172.21.15.16:0/3981940160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697819ea30 con 0x7f6978073c30 2024-04-05T23:34:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.393+0000 7f696fe00700 1 -- 172.21.15.16:0/3981940160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964017650 con 0x7f6978073c30 2024-04-05T23:34:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.394+0000 7f697e400700 1 -- 172.21.15.16:0/3981940160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697819ecb0 con 0x7f6978073c30 2024-04-05T23:34:51.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.398+0000 7f696fe00700 1 -- 172.21.15.16:0/3981940160 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f69640225e0 con 0x7f6978073c30 2024-04-05T23:34:51.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f6991ec7b80 1 Processor -- start 2024-04-05T23:34:51.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f6991ec7b80 1 -- start start 2024-04-05T23:34:51.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f696fe00700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f695c0a13b0 0x7f695c0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f696fe00700 1 -- 172.21.15.16:0/3981940160 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f69640ce030 con 0x7f6978073c30 2024-04-05T23:34:51.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f793180 0x55ae5f793560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 0x55ae5f78d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78dca0 0x55ae5f799f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f6f3be0 con 0x55ae5f793180 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f6e4990 con 0x55ae5f78d2e0 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.399+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f6e2350 con 0x55ae5f78dca0 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977e00700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f695c0a13b0 0x7f695c0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f793180 0x55ae5f793560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f793180 0x55ae5f793560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59330/0 (socket says 172.21.15.16:59330) 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977400700 1 -- 172.21.15.16:0/539684694 learned_addr learned my addr 172.21.15.16:0/539684694 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6976a00700 1 --2- 172.21.15.16:0/539684694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 0x55ae5f78d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977e00700 1 --2- 172.21.15.16:0/539684694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78dca0 0x55ae5f799f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977400700 1 -- 172.21.15.16:0/539684694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78dca0 msgr2=0x55ae5f799f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977400700 1 --2- 172.21.15.16:0/539684694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78dca0 0x55ae5f799f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977400700 1 -- 172.21.15.16:0/539684694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 msgr2=0x55ae5f78d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977400700 1 --2- 172.21.15.16:0/539684694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 0x55ae5f78d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977400700 1 -- 172.21.15.16:0/539684694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae5f55f3b0 con 0x55ae5f793180 2024-04-05T23:34:51.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977e00700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f695c0a13b0 0x7f695c0a3870 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6970009ec0 tx=0x7f697000fc40 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.400+0000 7f6977400700 1 --2- 172.21.15.16:0/539684694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f793180 0x55ae5f793560 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6968009d80 tx=0x7f6968011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29c5f4be5fad87e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.401+0000 7f696cc00700 1 -- 172.21.15.16:0/539684694 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968004230 con 0x55ae5f793180 2024-04-05T23:34:51.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.401+0000 7f696cc00700 1 -- 172.21.15.16:0/539684694 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6968004390 con 0x55ae5f793180 2024-04-05T23:34:51.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.401+0000 7f696cc00700 1 -- 172.21.15.16:0/539684694 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69680197e0 con 0x55ae5f793180 2024-04-05T23:34:51.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.401+0000 7f6991ec7b80 1 -- 172.21.15.16:0/539684694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f793180 msgr2=0x55ae5f793560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.401+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/539684694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f793180 0x55ae5f793560 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6968009d80 tx=0x7f6968011720 comp rx=0 tx=0).stop 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.402+0000 7f6991ec7b80 1 -- 172.21.15.16:0/539684694 shutdown_connections 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.402+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/539684694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f793180 0x55ae5f793560 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.402+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/539684694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78dca0 0x55ae5f799f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.402+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/539684694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 0x55ae5f78d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.402+0000 7f6991ec7b80 1 -- 172.21.15.16:0/539684694 >> 172.21.15.16:0/539684694 conn(0x55ae5f701220 msgr2=0x55ae5f788570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.402+0000 7f6991ec7b80 1 -- 172.21.15.16:0/539684694 shutdown_connections 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.402+0000 7f6991ec7b80 1 -- 172.21.15.16:0/539684694 wait complete. 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.402+0000 7f6991ec7b80 1 -- start start 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.402+0000 7f6991ec7b80 1 Processor -- start 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 0x55ae5f799e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f797150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 0x55ae5f797780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f6ddc40 con 0x55ae5f78dca0 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f679d80 con 0x55ae5f793180 2024-04-05T23:34:51.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f679a20 con 0x55ae5f78d2e0 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6991ec7b80 10 client.0 _ll_register_callbacks cb 0x55ae5f4e53a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f797150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6977400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 0x55ae5f799e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f797150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59338/0 (socket says 172.21.15.16:59338) 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6976a00700 1 -- 172.21.15.16:0/2790458463 learned_addr learned my addr 172.21.15.16:0/2790458463 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.403+0000 7f6977e00700 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 0x55ae5f797780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6976a00700 1 -- 172.21.15.16:0/2790458463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 msgr2=0x55ae5f799e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6976a00700 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 0x55ae5f799e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6976a00700 1 -- 172.21.15.16:0/2790458463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 msgr2=0x55ae5f797780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6976a00700 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 0x55ae5f797780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6976a00700 1 -- 172.21.15.16:0/2790458463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69680099e0 con 0x55ae5f78dca0 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6976a00700 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f797150 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f696409af20 tx=0x7f696409b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7669d23a612e001 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6961600700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:51.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6961600700 1 -- 172.21.15.16:0/2790458463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69640a30f0 con 0x55ae5f78dca0 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6961600700 1 -- 172.21.15.16:0/2790458463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f696409b690 con 0x55ae5f78dca0 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6991ec7b80 1 -- 172.21.15.16:0/2790458463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae5f780490 con 0x55ae5f78dca0 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6961600700 1 -- 172.21.15.16:0/2790458463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69640a10c0 con 0x55ae5f78dca0 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6991ec7b80 10 client.15708 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.404+0000 7f6991ec7b80 1 -- 172.21.15.16:0/2790458463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ae5f50bb60 con 0x55ae5f78dca0 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.405+0000 7f6961600700 1 -- 172.21.15.16:0/2790458463 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69640b7080 con 0x55ae5f78dca0 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.405+0000 7f6961600700 1 -- 172.21.15.16:0/2790458463 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f69640cba10 con 0x55ae5f78dca0 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.405+0000 7f6991ec7b80 10 client.15708 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.405+0000 7f6991ec7b80 10 client.15708 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.d 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.405+0000 7f6991ec7b80 20 client.15708 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.405+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55ae5f7a7130 0x55ae5f7a9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.405+0000 7f6991ec7b80 4 client.15708 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.405+0000 7f6991ec7b80 1 -- 172.21.15.16:0/2790458463 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae5f6d6700 con 0x55ae5f7a7130 2024-04-05T23:34:51.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.405+0000 7f6977400700 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55ae5f7a7130 0x55ae5f7a9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.406+0000 7f6977400700 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55ae5f7a7130 0x55ae5f7a9520 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f958ed6a9535f230 server_cookie=fa87eeb9fa7802e2 in_seq=0 out_seq=0 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.406+0000 7f6961600700 10 client.15708 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.406+0000 7f6961600700 1 -- 172.21.15.16:0/2790458463 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ae5f6d6700 con 0x55ae5f7a7130 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.406+0000 7f6961600700 10 client.15708 handle_command_reply: tid=0 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.406+0000 7f6991ec7b80 1 client.15708 shutdown 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.406+0000 7f6991ec7b80 1 -- 172.21.15.16:0/2790458463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 msgr2=0x55ae5f797150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.406+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f797150 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f696409af20 tx=0x7f696409b4d0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.406+0000 7f6991ec7b80 1 -- 172.21.15.16:0/2790458463 shutdown_connections 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.407+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55ae5f7a7130 0x55ae5f7a9520 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.407+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 0x55ae5f797780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.407+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 0x55ae5f799e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.407+0000 7f6961600700 0 client.15708 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:34:51.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.407+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/2790458463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f797150 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.407+0000 7f6991ec7b80 1 -- 172.21.15.16:0/2790458463 >> 172.21.15.16:0/2790458463 conn(0x55ae5f701220 msgr2=0x55ae5f782460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.407+0000 7f6991ec7b80 1 -- 172.21.15.16:0/2790458463 shutdown_connections 2024-04-05T23:34:51.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.407+0000 7f6991ec7b80 1 -- 172.21.15.16:0/2790458463 wait complete. 2024-04-05T23:34:51.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.407+0000 7f6991ec7b80 20 client.15708 trim_cache size 0 max 16384 2024-04-05T23:34:51.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.420+0000 7f6991ec7b80 1 Processor -- start 2024-04-05T23:34:51.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6991ec7b80 1 -- start start 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 0x55ae5f782560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f799990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f793180 0x55ae5f78fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f5616a0 con 0x55ae5f78dca0 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f78e560 con 0x55ae5f78d2e0 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f798090 con 0x55ae5f793180 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f799990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6977400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 0x55ae5f782560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f799990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59342/0 (socket says 172.21.15.16:59342) 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6977400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 0x55ae5f782560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47686/0 (socket says 172.21.15.16:47686) 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6976a00700 1 -- 172.21.15.16:0/1969858950 learned_addr learned my addr 172.21.15.16:0/1969858950 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6977e00700 1 --2- 172.21.15.16:0/1969858950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f793180 0x55ae5f78fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6976a00700 1 -- 172.21.15.16:0/1969858950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f793180 msgr2=0x55ae5f78fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.421+0000 7f6976a00700 1 --2- 172.21.15.16:0/1969858950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f793180 0x55ae5f78fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.422+0000 7f6976a00700 1 -- 172.21.15.16:0/1969858950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 msgr2=0x55ae5f782560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.422+0000 7f6976a00700 1 --2- 172.21.15.16:0/1969858950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 0x55ae5f782560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.422+0000 7f6976a00700 1 -- 172.21.15.16:0/1969858950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae5f6a3ea0 con 0x55ae5f78dca0 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.422+0000 7f6976a00700 1 --2- 172.21.15.16:0/1969858950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f799990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6964006010 tx=0x7f6964005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7243f4fdce5030 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.422+0000 7f696d600700 1 -- 172.21.15.16:0/1969858950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69640b9170 con 0x55ae5f78dca0 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.422+0000 7f696d600700 1 -- 172.21.15.16:0/1969858950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ae5f780490 con 0x55ae5f78dca0 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.422+0000 7f696d600700 1 -- 172.21.15.16:0/1969858950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69640a10b0 con 0x55ae5f78dca0 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1969858950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 msgr2=0x55ae5f799990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1969858950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f799990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6964006010 tx=0x7f6964005c00 comp rx=0 tx=0).stop 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1969858950 shutdown_connections 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1969858950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f793180 0x55ae5f78fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1969858950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f78d2e0 0x55ae5f782560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1969858950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f799990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1969858950 >> 172.21.15.16:0/1969858950 conn(0x55ae5f701220 msgr2=0x55ae5f790680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1969858950 shutdown_connections 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1969858950 wait complete. 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 -- start start 2024-04-05T23:34:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.423+0000 7f6991ec7b80 1 Processor -- start 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 0x55ae5f79a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f79aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6991ec7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 0x55ae5f7f9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f79b220 con 0x55ae5f78dca0 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f79b390 con 0x55ae5f793180 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6991ec7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae5f7fa4e0 con 0x55ae5f78d2e0 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6991ec7b80 10 client.0 _ll_register_callbacks cb 0x7f696409ad60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f79aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6977e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 0x55ae5f7f9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f79aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59352/0 (socket says 172.21.15.16:59352) 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6976a00700 1 -- 172.21.15.16:0/1192849879 learned_addr learned my addr 172.21.15.16:0/1192849879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6977400700 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 0x55ae5f79a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6976a00700 1 -- 172.21.15.16:0/1192849879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 msgr2=0x55ae5f79a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6976a00700 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 0x55ae5f79a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6976a00700 1 -- 172.21.15.16:0/1192849879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 msgr2=0x55ae5f7f9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6976a00700 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 0x55ae5f7f9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.424+0000 7f6976a00700 1 -- 172.21.15.16:0/1192849879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae5f50bb60 con 0x55ae5f78dca0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6976a00700 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f79aa20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f696409af20 tx=0x7f69640cbb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7605fb3f2ddc75 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6961600700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6961600700 1 -- 172.21.15.16:0/1192849879 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69640a10b0 con 0x55ae5f78dca0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6961600700 1 -- 172.21.15.16:0/1192849879 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f696409b690 con 0x55ae5f78dca0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1192849879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6968004390 con 0x55ae5f78dca0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6961600700 1 -- 172.21.15.16:0/1192849879 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f696409af90 con 0x55ae5f78dca0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6991ec7b80 10 client.15714 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1192849879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ae5f563cd0 con 0x55ae5f78dca0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6961600700 1 -- 172.21.15.16:0/1192849879 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6968004390 con 0x55ae5f78dca0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6961600700 1 -- 172.21.15.16:0/1192849879 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f6964062d90 con 0x55ae5f78dca0 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6991ec7b80 10 client.15714 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6991ec7b80 10 client.15714 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.d 2024-04-05T23:34:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.425+0000 7f6991ec7b80 20 client.15714 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:51.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.426+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55ae5f7fb7b0 0x55ae5f7fdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.426+0000 7f6991ec7b80 4 client.15714 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:51.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.426+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1192849879 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ae5f52ccf0 con 0x55ae5f7fb7b0 2024-04-05T23:34:51.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.426+0000 7f6977400700 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55ae5f7fb7b0 0x55ae5f7fdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.426+0000 7f6961600700 10 client.15714 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.426+0000 7f6977400700 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55ae5f7fb7b0 0x55ae5f7fdba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e36ac9f9623d4d6 server_cookie=69d3fe0812473b1c in_seq=0 out_seq=0 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6961600700 1 -- 172.21.15.16:0/1192849879 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ae5f52ccf0 con 0x55ae5f7fb7b0 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6961600700 10 client.15714 handle_command_reply: tid=0 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 client.15714 shutdown 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1192849879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 msgr2=0x55ae5f79aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f79aa20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f696409af20 tx=0x7f69640cbb20 comp rx=0 tx=0).stop 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1192849879 shutdown_connections 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x55ae5f7fb7b0 0x55ae5f7fdba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ae5f793180 0x55ae5f7f9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae5f78d2e0 0x55ae5f79a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 --2- 172.21.15.16:0/1192849879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ae5f78dca0 0x55ae5f79aa20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6961600700 0 client.15714 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:34:51.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1192849879 >> 172.21.15.16:0/1192849879 conn(0x55ae5f701220 msgr2=0x55ae5f795a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1192849879 shutdown_connections 2024-04-05T23:34:51.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 1 -- 172.21.15.16:0/1192849879 wait complete. 2024-04-05T23:34:51.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.427+0000 7f6991ec7b80 20 client.15714 trim_cache size 0 max 16384 2024-04-05T23:34:51.424 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 -- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f695c0a13b0 msgr2=0x7f695c0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f695c0a13b0 0x7f695c0a3870 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6970009ec0 tx=0x7f697000fc40 comp rx=0 tx=0).stop 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 -- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 msgr2=0x7f6978199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 0x7f6978199aa0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6964005910 tx=0x7f6964004730 comp rx=0 tx=0).stop 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 -- 172.21.15.16:0/3981940160 shutdown_connections 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f695c0a13b0 0x7f695c0a3870 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f697810d670 0x7f697819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6978073310 0x7f6978199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 --2- 172.21.15.16:0/3981940160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6978073c30 0x7f6978199aa0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 -- 172.21.15.16:0/3981940160 >> 172.21.15.16:0/3981940160 conn(0x7f69780fc900 msgr2=0x7f69780fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 -- 172.21.15.16:0/3981940160 shutdown_connections 2024-04-05T23:34:51.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.428+0000 7f697e400700 1 -- 172.21.15.16:0/3981940160 wait complete. 2024-04-05T23:34:51.439 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-05T23:34:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.648+0000 7f8730600700 1 -- 172.21.15.16:0/993269773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 msgr2=0x7f8728073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.648+0000 7f8730600700 1 --2- 172.21.15.16:0/993269773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 0x7f8728073660 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f871c009990 tx=0x7f871c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.648+0000 7f8730600700 1 -- 172.21.15.16:0/993269773 shutdown_connections 2024-04-05T23:34:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.648+0000 7f8730600700 1 --2- 172.21.15.16:0/993269773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 0x7f8728073660 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.648+0000 7f8730600700 1 --2- 172.21.15.16:0/993269773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728111e40 0x7f8728114230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.648+0000 7f8730600700 1 --2- 172.21.15.16:0/993269773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8728073ba0 0x7f8728111900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.648+0000 7f8730600700 1 -- 172.21.15.16:0/993269773 >> 172.21.15.16:0/993269773 conn(0x7f87280fc900 msgr2=0x7f87280fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.648+0000 7f8730600700 1 -- 172.21.15.16:0/993269773 shutdown_connections 2024-04-05T23:34:51.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.648+0000 7f8730600700 1 -- 172.21.15.16:0/993269773 wait complete. 2024-04-05T23:34:51.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.649+0000 7f8730600700 1 Processor -- start 2024-04-05T23:34:51.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.650+0000 7f8730600700 1 -- start start 2024-04-05T23:34:51.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.650+0000 7f8730600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 0x7f87281a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.650+0000 7f8730600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728073ba0 0x7f87281a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.650+0000 7f8730600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8728111e40 0x7f872819d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.650+0000 7f8730600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872819de90 con 0x7f8728073280 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.650+0000 7f8730600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872819e000 con 0x7f8728073ba0 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.650+0000 7f8730600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872819e1c0 con 0x7f8728111e40 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.650+0000 7f872dc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 0x7f87281a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872dc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 0x7f87281a36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59374/0 (socket says 172.21.15.16:59374) 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872dc00700 1 -- 172.21.15.16:0/2359326892 learned_addr learned my addr 172.21.15.16:0/2359326892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872d200700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728073ba0 0x7f87281a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872e600700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8728111e40 0x7f872819d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872dc00700 1 -- 172.21.15.16:0/2359326892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8728111e40 msgr2=0x7f872819d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872dc00700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8728111e40 0x7f872819d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872dc00700 1 -- 172.21.15.16:0/2359326892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728073ba0 msgr2=0x7f87281a3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872dc00700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728073ba0 0x7f87281a3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872dc00700 1 -- 172.21.15.16:0/2359326892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8720009c40 con 0x7f8728073280 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872d200700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728073ba0 0x7f87281a3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f872dc00700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 0x7f87281a36a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f871c005910 tx=0x7f871c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f8726000700 1 -- 172.21.15.16:0/2359326892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c01d070 con 0x7f8728073280 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f8730600700 1 -- 172.21.15.16:0/2359326892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c0095f0 con 0x7f8728073280 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.651+0000 7f8726000700 1 -- 172.21.15.16:0/2359326892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f871c022470 con 0x7f8728073280 2024-04-05T23:34:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.652+0000 7f8730600700 1 -- 172.21.15.16:0/2359326892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87281aa4b0 con 0x7f8728073280 2024-04-05T23:34:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.652+0000 7f8726000700 1 -- 172.21.15.16:0/2359326892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c0175f0 con 0x7f8728073280 2024-04-05T23:34:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.655+0000 7f8726000700 1 -- 172.21.15.16:0/2359326892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f871c0225e0 con 0x7f8728073280 2024-04-05T23:34:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.656+0000 7f87440adb80 1 Processor -- start 2024-04-05T23:34:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.656+0000 7f8726000700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87100a13b0 0x7f87100a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.656+0000 7f87440adb80 1 -- start start 2024-04-05T23:34:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.656+0000 7f8726000700 1 -- 172.21.15.16:0/2359326892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f871c0ce1a0 con 0x7f8728073280 2024-04-05T23:34:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.656+0000 7f872e600700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87100a13b0 0x7f87100a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.656+0000 7f87440adb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260cad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.656+0000 7f87440adb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260ca7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.656+0000 7f87440adb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260cb3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f87440adb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260c0dbe0 con 0x55d260ca7ca0 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f87440adb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260bfe990 con 0x55d260ca72e0 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f87440adb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260bfc350 con 0x55d260cad180 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872dc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260cad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872dc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260cad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:56520/0 (socket says 172.21.15.16:56520) 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872dc00700 1 -- 172.21.15.16:0/2747593498 learned_addr learned my addr 172.21.15.16:0/2747593498 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872e600700 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260cb3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872d200700 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260ca7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872dc00700 1 -- 172.21.15.16:0/2747593498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 msgr2=0x55d260ca7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872dc00700 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260ca7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872dc00700 1 -- 172.21.15.16:0/2747593498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 msgr2=0x55d260cb3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872dc00700 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260cb3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872dc00700 1 -- 172.21.15.16:0/2747593498 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d260a793b0 con 0x55d260cad180 2024-04-05T23:34:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.657+0000 7f872d200700 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260ca7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.658+0000 7f872e600700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87100a13b0 0x7f87100a3870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f87281a43a0 tx=0x7f8720007bc0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.658+0000 7f872dc00700 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260cad560 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f871c097600 tx=0x7f871c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9489b774e8b24aba server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.658+0000 7f871aa00700 1 -- 172.21.15.16:0/2747593498 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c09b800 con 0x55d260cad180 2024-04-05T23:34:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.658+0000 7f871aa00700 1 -- 172.21.15.16:0/2747593498 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f871c09b100 con 0x55d260cad180 2024-04-05T23:34:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.658+0000 7f871aa00700 1 -- 172.21.15.16:0/2747593498 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c09a5f0 con 0x55d260cad180 2024-04-05T23:34:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.658+0000 7f87440adb80 1 -- 172.21.15.16:0/2747593498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 msgr2=0x55d260cad560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.658+0000 7f87440adb80 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260cad560 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f871c097600 tx=0x7f871c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.659+0000 7f87440adb80 1 -- 172.21.15.16:0/2747593498 shutdown_connections 2024-04-05T23:34:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.659+0000 7f87440adb80 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260cb3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.659+0000 7f87440adb80 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260cad560 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.659+0000 7f87440adb80 1 --2- 172.21.15.16:0/2747593498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260ca7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.659+0000 7f87440adb80 1 -- 172.21.15.16:0/2747593498 >> 172.21.15.16:0/2747593498 conn(0x55d260c1b220 msgr2=0x55d260ca2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.659+0000 7f87440adb80 1 -- 172.21.15.16:0/2747593498 shutdown_connections 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.659+0000 7f87440adb80 1 -- 172.21.15.16:0/2747593498 wait complete. 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.659+0000 7f87440adb80 1 -- start start 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.659+0000 7f87440adb80 1 Processor -- start 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f87440adb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260cb6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f87440adb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f87440adb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cb1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f87440adb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260bf7c40 con 0x55d260cad180 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f87440adb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260b93d80 con 0x55d260ca72e0 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f87440adb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260b93a20 con 0x55d260ca7ca0 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f87440adb80 10 client.0 _ll_register_callbacks cb 0x55d2609ff3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f872e600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cb1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f872dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260cb6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f872e600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cb1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59400/0 (socket says 172.21.15.16:59400) 2024-04-05T23:34:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f872e600700 1 -- 172.21.15.16:0/2159530694 learned_addr learned my addr 172.21.15.16:0/2159530694 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.660+0000 7f872d200700 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f872e600700 1 -- 172.21.15.16:0/2159530694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 msgr2=0x55d260cb7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f872e600700 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f872e600700 1 -- 172.21.15.16:0/2159530694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 msgr2=0x55d260cb6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f872e600700 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260cb6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f872e600700 1 -- 172.21.15.16:0/2159530694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87281aa4b0 con 0x55d260cad180 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f872dc00700 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260cb6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f872e600700 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cb1010 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8720017e50 tx=0x7f8720004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642519cfeef05372 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f870f400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f870f400700 1 -- 172.21.15.16:0/2159530694 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872000e050 con 0x55d260cad180 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f870f400700 1 -- 172.21.15.16:0/2159530694 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f872001cc00 con 0x55d260cad180 2024-04-05T23:34:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f87440adb80 1 -- 172.21.15.16:0/2159530694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d260c9a490 con 0x55d260cad180 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f870f400700 1 -- 172.21.15.16:0/2159530694 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8720009e70 con 0x55d260cad180 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f87440adb80 10 client.15726 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.661+0000 7f87440adb80 1 -- 172.21.15.16:0/2159530694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d260a25b60 con 0x55d260cad180 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.662+0000 7f870f400700 1 -- 172.21.15.16:0/2159530694 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8720032020 con 0x55d260cad180 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.662+0000 7f870f400700 1 -- 172.21.15.16:0/2159530694 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f872001ad00 con 0x55d260cad180 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.662+0000 7f87440adb80 10 client.15726 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.662+0000 7f87440adb80 10 client.15726 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.662+0000 7f87440adb80 20 client.15726 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.662+0000 7f87440adb80 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55d260cc1170 0x55d260cc3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.662+0000 7f87440adb80 4 client.15726 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.662+0000 7f87440adb80 1 -- 172.21.15.16:0/2159530694 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d260bf0700 con 0x55d260cc1170 2024-04-05T23:34:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.662+0000 7f872dc00700 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55d260cc1170 0x55d260cc3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.663+0000 7f870f400700 10 client.15726 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.663+0000 7f872dc00700 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55d260cc1170 0x55d260cc3590 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6d2d079a99e9fa64 server_cookie=550feda7bd39086a in_seq=0 out_seq=0 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.663+0000 7f870f400700 1 -- 172.21.15.16:0/2159530694 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d260bf0700 con 0x55d260cc1170 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.663+0000 7f870f400700 10 client.15726 handle_command_reply: tid=0 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.663+0000 7f87440adb80 1 client.15726 shutdown 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 -- 172.21.15.16:0/2159530694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 msgr2=0x55d260cb1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cb1010 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8720017e50 tx=0x7f8720004b40 comp rx=0 tx=0).stop 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 -- 172.21.15.16:0/2159530694 shutdown_connections 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55d260cc1170 0x55d260cc3590 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cb1010 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f870f400700 0 client.15726 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:34:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260cb6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 --2- 172.21.15.16:0/2159530694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 -- 172.21.15.16:0/2159530694 >> 172.21.15.16:0/2159530694 conn(0x55d260c1b220 msgr2=0x55d260c9c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 -- 172.21.15.16:0/2159530694 shutdown_connections 2024-04-05T23:34:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 1 -- 172.21.15.16:0/2159530694 wait complete. 2024-04-05T23:34:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.664+0000 7f87440adb80 20 client.15726 trim_cache size 0 max 16384 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f87440adb80 1 Processor -- start 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f87440adb80 1 -- start start 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f87440adb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260caec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f87440adb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260c9c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f87440adb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260ca9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f87440adb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260a7b6a0 con 0x55d260ca7ca0 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f87440adb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260cbdca0 con 0x55d260ca72e0 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f87440adb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260caae40 con 0x55d260cad180 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f872d200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260c9c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f872d200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260c9c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59402/0 (socket says 172.21.15.16:59402) 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f872d200700 1 -- 172.21.15.16:0/588153876 learned_addr learned my addr 172.21.15.16:0/588153876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f872e600700 1 --2- 172.21.15.16:0/588153876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260ca9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.678+0000 7f872dc00700 1 --2- 172.21.15.16:0/588153876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260caec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.679+0000 7f872d200700 1 -- 172.21.15.16:0/588153876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 msgr2=0x55d260ca9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.679+0000 7f872d200700 1 --2- 172.21.15.16:0/588153876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260ca9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.679+0000 7f872d200700 1 -- 172.21.15.16:0/588153876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 msgr2=0x55d260caec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.679+0000 7f872d200700 1 --2- 172.21.15.16:0/588153876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260caec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.679+0000 7f872d200700 1 -- 172.21.15.16:0/588153876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d260bbdea0 con 0x55d260ca7ca0 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.679+0000 7f872d200700 1 --2- 172.21.15.16:0/588153876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260c9c560 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8714009d80 tx=0x7f8714012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a30bd5082fbe50 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.679+0000 7f871b400700 1 -- 172.21.15.16:0/588153876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8714017070 con 0x55d260ca7ca0 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.679+0000 7f871b400700 1 -- 172.21.15.16:0/588153876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8714004510 con 0x55d260ca7ca0 2024-04-05T23:34:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.679+0000 7f871b400700 1 -- 172.21.15.16:0/588153876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871400f680 con 0x55d260ca7ca0 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 -- 172.21.15.16:0/588153876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 msgr2=0x55d260c9c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 --2- 172.21.15.16:0/588153876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260c9c560 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8714009d80 tx=0x7f8714012730 comp rx=0 tx=0).stop 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 -- 172.21.15.16:0/588153876 shutdown_connections 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 --2- 172.21.15.16:0/588153876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260cad180 0x55d260ca9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 --2- 172.21.15.16:0/588153876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260caec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 --2- 172.21.15.16:0/588153876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260ca7ca0 0x55d260c9c560 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 -- 172.21.15.16:0/588153876 >> 172.21.15.16:0/588153876 conn(0x55d260c1b220 msgr2=0x55d260ca2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 -- 172.21.15.16:0/588153876 shutdown_connections 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 -- 172.21.15.16:0/588153876 wait complete. 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.680+0000 7f87440adb80 1 -- start start 2024-04-05T23:34:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f87440adb80 1 Processor -- start 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f87440adb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260cb9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f87440adb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f87440adb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cba3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f87440adb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260d14100 con 0x55d260cad180 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f87440adb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260d14270 con 0x55d260ca72e0 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f87440adb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d260d14430 con 0x55d260ca7ca0 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f87440adb80 10 client.0 _ll_register_callbacks cb 0x7f8720017a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f872d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f872d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:56546/0 (socket says 172.21.15.16:56546) 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f872d200700 1 -- 172.21.15.16:0/900191643 learned_addr learned my addr 172.21.15.16:0/900191643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f872e600700 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cba3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.681+0000 7f872dc00700 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260cb9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f872e600700 1 -- 172.21.15.16:0/900191643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 msgr2=0x55d260cb9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f872e600700 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f872e600700 1 -- 172.21.15.16:0/900191643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 msgr2=0x55d260cb9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f872e600700 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260cb9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f872e600700 1 -- 172.21.15.16:0/900191643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8714009a30 con 0x55d260cad180 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f872d200700 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f872e600700 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cba3d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8720017e20 tx=0x7f8720019510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d080b61423b748 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f870f400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f870f400700 1 -- 172.21.15.16:0/900191643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872000e030 con 0x55d260cad180 2024-04-05T23:34:51.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f870f400700 1 -- 172.21.15.16:0/900191643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f8720033080 con 0x55d260cad180 2024-04-05T23:34:51.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f87440adb80 1 -- 172.21.15.16:0/900191643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f871c09b100 con 0x55d260cad180 2024-04-05T23:34:51.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.682+0000 7f870f400700 1 -- 172.21.15.16:0/900191643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87200098a0 con 0x55d260cad180 2024-04-05T23:34:51.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f87440adb80 10 client.15732 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:51.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f87440adb80 1 -- 172.21.15.16:0/900191643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d260a7dcd0 con 0x55d260cad180 2024-04-05T23:34:51.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f870f400700 1 -- 172.21.15.16:0/900191643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8720044020 con 0x55d260cad180 2024-04-05T23:34:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f870f400700 1 -- 172.21.15.16:0/900191643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f872002c610 con 0x55d260cad180 2024-04-05T23:34:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f87440adb80 10 client.15732 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f87440adb80 10 client.15732 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-05T23:34:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f87440adb80 20 client.15732 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f87440adb80 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55d260d15b70 0x55d260d17f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f87440adb80 4 client.15732 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f87440adb80 1 -- 172.21.15.16:0/900191643 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d260a46cf0 con 0x55d260d15b70 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.683+0000 7f872dc00700 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55d260d15b70 0x55d260d17f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f872dc00700 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55d260d15b70 0x55d260d17f90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d9e4005aa82042e server_cookie=32faaefc0402406f in_seq=0 out_seq=0 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f870f400700 10 client.15732 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f870f400700 1 -- 172.21.15.16:0/900191643 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d260a46cf0 con 0x55d260d15b70 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f870f400700 10 client.15732 handle_command_reply: tid=0 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 client.15732 shutdown 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 -- 172.21.15.16:0/900191643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 msgr2=0x55d260cba3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cba3d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8720017e20 tx=0x7f8720019510 comp rx=0 tx=0).stop 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 -- 172.21.15.16:0/900191643 shutdown_connections 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x55d260d15b70 0x55d260d17f90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d260cad180 0x55d260cba3d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d260ca72e0 0x55d260cb9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f870f400700 0 client.15732 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:34:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 --2- 172.21.15.16:0/900191643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d260ca7ca0 0x55d260cb9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 -- 172.21.15.16:0/900191643 >> 172.21.15.16:0/900191643 conn(0x55d260c1b220 msgr2=0x55d260cb67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 -- 172.21.15.16:0/900191643 shutdown_connections 2024-04-05T23:34:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.684+0000 7f87440adb80 1 -- 172.21.15.16:0/900191643 wait complete. 2024-04-05T23:34:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f87440adb80 20 client.15732 trim_cache size 0 max 16384 2024-04-05T23:34:51.685 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 -- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87100a13b0 msgr2=0x7f87100a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87100a13b0 0x7f87100a3870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f87281a43a0 tx=0x7f8720007bc0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 -- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 msgr2=0x7f87281a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 0x7f87281a36a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f871c005910 tx=0x7f871c004730 comp rx=0 tx=0).stop 2024-04-05T23:34:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 -- 172.21.15.16:0/2359326892 shutdown_connections 2024-04-05T23:34:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f87100a13b0 0x7f87100a3870 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8728073280 0x7f87281a36a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8728111e40 0x7f872819d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 --2- 172.21.15.16:0/2359326892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8728073ba0 0x7f87281a3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 -- 172.21.15.16:0/2359326892 >> 172.21.15.16:0/2359326892 conn(0x7f87280fc900 msgr2=0x7f87280fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 -- 172.21.15.16:0/2359326892 shutdown_connections 2024-04-05T23:34:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.685+0000 7f8730600700 1 -- 172.21.15.16:0/2359326892 wait complete. 2024-04-05T23:34:51.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-05T23:34:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.906+0000 7f83ce600700 1 -- 172.21.15.16:0/2919370663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81061c0 msgr2=0x7f83c810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.906+0000 7f83ce600700 1 --2- 172.21.15.16:0/2919370663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81061c0 0x7f83c810dd40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f83c000b0a0 tx=0x7f83c000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.906+0000 7f83ce600700 1 -- 172.21.15.16:0/2919370663 shutdown_connections 2024-04-05T23:34:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.906+0000 7f83ce600700 1 --2- 172.21.15.16:0/2919370663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81061c0 0x7f83c810dd40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.906+0000 7f83ce600700 1 --2- 172.21.15.16:0/2919370663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83c81010a0 0x7f83c8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.906+0000 7f83ce600700 1 --2- 172.21.15.16:0/2919370663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c8101a50 0x7f83c8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.906+0000 7f83ce600700 1 -- 172.21.15.16:0/2919370663 >> 172.21.15.16:0/2919370663 conn(0x7f83c80fc900 msgr2=0x7f83c80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.906+0000 7f83ce600700 1 -- 172.21.15.16:0/2919370663 shutdown_connections 2024-04-05T23:34:51.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.906+0000 7f83ce600700 1 -- 172.21.15.16:0/2919370663 wait complete. 2024-04-05T23:34:51.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.907+0000 7f83ce600700 1 Processor -- start 2024-04-05T23:34:51.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.907+0000 7f83ce600700 1 -- start start 2024-04-05T23:34:51.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.907+0000 7f83ce600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81010a0 0x7f83c8199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.907+0000 7f83ce600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83c81061c0 0x7f83c8199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.907+0000 7f83ce600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c819df20 0x7f83c819e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.907+0000 7f83ce600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c819e9a0 con 0x7f83c81010a0 2024-04-05T23:34:51.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.907+0000 7f83ce600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c819eb10 con 0x7f83c819df20 2024-04-05T23:34:51.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.907+0000 7f83ce600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c819ecd0 con 0x7f83c81061c0 2024-04-05T23:34:51.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81010a0 0x7f83c8199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81010a0 0x7f83c8199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59418/0 (socket says 172.21.15.16:59418) 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7400700 1 -- 172.21.15.16:0/3007382233 learned_addr learned my addr 172.21.15.16:0/3007382233 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7e00700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c819df20 0x7f83c819e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c6a00700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83c81061c0 0x7f83c8199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7400700 1 -- 172.21.15.16:0/3007382233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83c81061c0 msgr2=0x7f83c8199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7400700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83c81061c0 0x7f83c8199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7400700 1 -- 172.21.15.16:0/3007382233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c819df20 msgr2=0x7f83c819e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7400700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c819df20 0x7f83c819e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7400700 1 -- 172.21.15.16:0/3007382233 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c0009d00 con 0x7f83c81010a0 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83c7400700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81010a0 0x7f83c8199640 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f83b800e9a0 tx=0x7f83b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.908+0000 7f83bfe00700 1 -- 172.21.15.16:0/3007382233 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b8009790 con 0x7f83c81010a0 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.909+0000 7f83bfe00700 1 -- 172.21.15.16:0/3007382233 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83b800cbd0 con 0x7f83c81010a0 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.909+0000 7f83ce600700 1 -- 172.21.15.16:0/3007382233 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c81a6230 con 0x7f83c81010a0 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.909+0000 7f83bfe00700 1 -- 172.21.15.16:0/3007382233 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b8018680 con 0x7f83c81010a0 2024-04-05T23:34:51.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.909+0000 7f83ce600700 1 -- 172.21.15.16:0/3007382233 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c81a6710 con 0x7f83c81010a0 2024-04-05T23:34:51.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.912+0000 7f83bfe00700 1 -- 172.21.15.16:0/3007382233 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f83b800cd40 con 0x7f83c81010a0 2024-04-05T23:34:51.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.913+0000 7f83e2092b80 1 Processor -- start 2024-04-05T23:34:51.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.913+0000 7f83bfe00700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83ac0a13b0 0x7f83ac0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.913+0000 7f83e2092b80 1 -- start start 2024-04-05T23:34:51.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.913+0000 7f83bfe00700 1 -- 172.21.15.16:0/3007382233 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f83b8014070 con 0x7f83c81010a0 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefed180 0x55aeeefed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7e00700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83ac0a13b0 0x7f83ac0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe72e0 0x55aeeefe7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83e2092b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeeef4dbe0 con 0x55aeeefed180 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83e2092b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeeef3e990 con 0x55aeeefe7ca0 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83e2092b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeeef3c350 con 0x55aeeefe72e0 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefed180 0x55aeeefed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefed180 0x55aeeefed560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59428/0 (socket says 172.21.15.16:59428) 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7400700 1 -- 172.21.15.16:0/232705747 learned_addr learned my addr 172.21.15.16:0/232705747 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c6a00700 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe72e0 0x55aeeefe7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7e00700 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7400700 1 -- 172.21.15.16:0/232705747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe72e0 msgr2=0x55aeeefe7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7400700 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe72e0 0x55aeeefe7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7400700 1 -- 172.21.15.16:0/232705747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe7ca0 msgr2=0x55aeeeff3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7400700 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.914+0000 7f83c7400700 1 -- 172.21.15.16:0/232705747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeeedb93b0 con 0x55aeeefed180 2024-04-05T23:34:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.915+0000 7f83c7e00700 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.915+0000 7f83c7e00700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83ac0a13b0 0x7f83ac0a3870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f83c000b780 tx=0x7f83c00095c0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.915+0000 7f83c7400700 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefed180 0x55aeeefed560 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f83b809a050 tx=0x7f83b8099aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fff32f5d6ff354 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.915+0000 7f83bcc00700 1 -- 172.21.15.16:0/232705747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b809def0 con 0x55aeeefed180 2024-04-05T23:34:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.915+0000 7f83bcc00700 1 -- 172.21.15.16:0/232705747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83b80a2540 con 0x55aeeefed180 2024-04-05T23:34:51.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.915+0000 7f83bcc00700 1 -- 172.21.15.16:0/232705747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b80a2380 con 0x55aeeefed180 2024-04-05T23:34:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 -- 172.21.15.16:0/232705747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefed180 msgr2=0x55aeeefed560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefed180 0x55aeeefed560 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f83b809a050 tx=0x7f83b8099aa0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 -- 172.21.15.16:0/232705747 shutdown_connections 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefed180 0x55aeeefed560 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 --2- 172.21.15.16:0/232705747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe72e0 0x55aeeefe7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 -- 172.21.15.16:0/232705747 >> 172.21.15.16:0/232705747 conn(0x55aeeef5b220 msgr2=0x55aeeefe2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 -- 172.21.15.16:0/232705747 shutdown_connections 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 -- 172.21.15.16:0/232705747 wait complete. 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.916+0000 7f83e2092b80 1 -- start start 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83e2092b80 1 Processor -- start 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 0x55aeeeff3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 0x55aeeeff17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83e2092b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeeef37c40 con 0x55aeeefe7ca0 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83e2092b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeeeed3d80 con 0x55aeeefe72e0 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83e2092b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeeeed3a20 con 0x55aeeefed180 2024-04-05T23:34:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83e2092b80 10 client.0 _ll_register_callbacks cb 0x55aeeed3f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83c6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83c6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59434/0 (socket says 172.21.15.16:59434) 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83c6a00700 1 -- 172.21.15.16:0/71191453 learned_addr learned my addr 172.21.15.16:0/71191453 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83c7400700 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 0x55aeeeff3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.917+0000 7f83c7e00700 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 0x55aeeeff17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83c6a00700 1 -- 172.21.15.16:0/71191453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 msgr2=0x55aeeeff17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83c6a00700 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 0x55aeeeff17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83c6a00700 1 -- 172.21.15.16:0/71191453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 msgr2=0x55aeeeff3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83c6a00700 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 0x55aeeeff3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83c6a00700 1 -- 172.21.15.16:0/71191453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c81a6710 con 0x55aeeefe7ca0 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83c6a00700 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff1130 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009b80 tx=0x7f83b4009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8619be79efebd25d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83b1600700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83b1600700 1 -- 172.21.15.16:0/71191453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4016070 con 0x55aeeefe7ca0 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83e2092b80 1 -- 172.21.15.16:0/71191453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aeeefda490 con 0x55aeeefe7ca0 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.918+0000 7f83b1600700 1 -- 172.21.15.16:0/71191453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f83b4003dc0 con 0x55aeeefe7ca0 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83b1600700 1 -- 172.21.15.16:0/71191453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83b4011440 con 0x55aeeefe7ca0 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83e2092b80 10 client.15744 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83e2092b80 1 -- 172.21.15.16:0/71191453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55aeeed65b60 con 0x55aeeefe7ca0 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83b1600700 1 -- 172.21.15.16:0/71191453 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83b4027080 con 0x55aeeefe7ca0 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83b1600700 1 -- 172.21.15.16:0/71191453 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f83b401b5e0 con 0x55aeeefe7ca0 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83e2092b80 10 client.15744 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83e2092b80 10 client.15744 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83e2092b80 20 client.15744 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83e2092b80 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55aeef001510 0x55aeef003930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83e2092b80 4 client.15744 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83e2092b80 1 -- 172.21.15.16:0/71191453 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aeeef30700 con 0x55aeef001510 2024-04-05T23:34:51.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.919+0000 7f83c7400700 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55aeef001510 0x55aeef003930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.920+0000 7f83c7400700 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55aeef001510 0x55aeef003930 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b7cdfdaa8a5c5c83 server_cookie=ee000f6553e92a1b in_seq=0 out_seq=0 2024-04-05T23:34:51.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.920+0000 7f83b1600700 10 client.15744 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83b1600700 1 -- 172.21.15.16:0/71191453 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55aeeef30700 con 0x55aeef001510 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83b1600700 10 client.15744 handle_command_reply: tid=0 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 client.15744 shutdown 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 -- 172.21.15.16:0/71191453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 msgr2=0x55aeeeff1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff1130 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009b80 tx=0x7f83b4009e90 comp rx=0 tx=0).stop 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 -- 172.21.15.16:0/71191453 shutdown_connections 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55aeef001510 0x55aeef003930 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 0x55aeeeff17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83b1600700 0 client.15744 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:34:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 0x55aeeeff3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 --2- 172.21.15.16:0/71191453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff1130 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 -- 172.21.15.16:0/71191453 >> 172.21.15.16:0/71191453 conn(0x55aeeef5b220 msgr2=0x55aeeefdc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 -- 172.21.15.16:0/71191453 shutdown_connections 2024-04-05T23:34:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.921+0000 7f83e2092b80 1 -- 172.21.15.16:0/71191453 wait complete. 2024-04-05T23:34:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.922+0000 7f83e2092b80 20 client.15744 trim_cache size 0 max 16384 2024-04-05T23:34:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.935+0000 7f83e2092b80 1 Processor -- start 2024-04-05T23:34:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.935+0000 7f83e2092b80 1 -- start start 2024-04-05T23:34:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.935+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 0x55aeeefdc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.935+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeefe2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.935+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 0x55aeeefe2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.935+0000 7f83e2092b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeeedbb6a0 con 0x55aeeefe7ca0 2024-04-05T23:34:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.935+0000 7f83e2092b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeeefe3900 con 0x55aeeefe72e0 2024-04-05T23:34:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.935+0000 7f83e2092b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeeefe2ac0 con 0x55aeeefed180 2024-04-05T23:34:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeefe2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeefe2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59446/0 (socket says 172.21.15.16:59446) 2024-04-05T23:34:51.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c6a00700 1 -- 172.21.15.16:0/484352798 learned_addr learned my addr 172.21.15.16:0/484352798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c7400700 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 0x55aeeefdc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c7e00700 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 0x55aeeefe2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c6a00700 1 -- 172.21.15.16:0/484352798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 msgr2=0x55aeeefe2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c6a00700 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 0x55aeeefe2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c6a00700 1 -- 172.21.15.16:0/484352798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 msgr2=0x55aeeefdc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c6a00700 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 0x55aeeefdc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c6a00700 1 -- 172.21.15.16:0/484352798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeeeefdea0 con 0x55aeeefe7ca0 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c7400700 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 0x55aeeefdc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83c6a00700 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeefe2040 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f83b401b910 tx=0x7f83b4009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8a2a83d1b44559 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.936+0000 7f83bd600700 1 -- 172.21.15.16:0/484352798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4016040 con 0x55aeeefe7ca0 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83bd600700 1 -- 172.21.15.16:0/484352798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55aeeefda490 con 0x55aeeefe7ca0 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83bd600700 1 -- 172.21.15.16:0/484352798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b401b6c0 con 0x55aeeefe7ca0 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83e2092b80 1 -- 172.21.15.16:0/484352798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 msgr2=0x55aeeefe2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83e2092b80 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeefe2040 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f83b401b910 tx=0x7f83b4009f00 comp rx=0 tx=0).stop 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83e2092b80 1 -- 172.21.15.16:0/484352798 shutdown_connections 2024-04-05T23:34:51.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83e2092b80 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefed180 0x55aeeefe2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83e2092b80 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefe72e0 0x55aeeefdc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83e2092b80 1 --2- 172.21.15.16:0/484352798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe7ca0 0x55aeeefe2040 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83e2092b80 1 -- 172.21.15.16:0/484352798 >> 172.21.15.16:0/484352798 conn(0x55aeeef5b220 msgr2=0x55aeeefefa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83e2092b80 1 -- 172.21.15.16:0/484352798 shutdown_connections 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.937+0000 7f83e2092b80 1 -- 172.21.15.16:0/484352798 wait complete. 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.938+0000 7f83e2092b80 1 -- start start 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.938+0000 7f83e2092b80 1 Processor -- start 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.938+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe72e0 0x55aeeeff66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.938+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.938+0000 7f83e2092b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefed180 0x55aeef054340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.938+0000 7f83e2092b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeef054880 con 0x55aeeefe72e0 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.938+0000 7f83e2092b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeef0549c0 con 0x55aeeefed180 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.938+0000 7f83e2092b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeef054b00 con 0x55aeeefe7ca0 2024-04-05T23:34:51.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.938+0000 7f83e2092b80 10 client.0 _ll_register_callbacks cb 0x7f83b4004630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:51.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c6a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefed180 0x55aeef054340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c6a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:56604/0 (socket says 172.21.15.16:56604) 2024-04-05T23:34:51.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe72e0 0x55aeeeff66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c6a00700 1 -- 172.21.15.16:0/929436561 learned_addr learned my addr 172.21.15.16:0/929436561 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:51.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c7400700 1 -- 172.21.15.16:0/929436561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe7ca0 msgr2=0x55aeeeff6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c7400700 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c7400700 1 -- 172.21.15.16:0/929436561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefed180 msgr2=0x55aeef054340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c7400700 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefed180 0x55aeef054340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c7400700 1 -- 172.21.15.16:0/929436561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aeeedb93b0 con 0x55aeeefe72e0 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83c7400700 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe72e0 0x55aeeeff66e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f83b800e970 tx=0x7f83b809a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fb89864817f8be server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83b1600700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.939+0000 7f83b1600700 1 -- 172.21.15.16:0/929436561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b80b8040 con 0x55aeeefe72e0 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83b1600700 1 -- 172.21.15.16:0/929436561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f83b809def0 con 0x55aeeefe72e0 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83e2092b80 1 -- 172.21.15.16:0/929436561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f83b80a2540 con 0x55aeeefe72e0 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83b1600700 1 -- 172.21.15.16:0/929436561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83b80a22b0 con 0x55aeeefe72e0 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83e2092b80 10 client.15750 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83e2092b80 1 -- 172.21.15.16:0/929436561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55aeeed65b60 con 0x55aeeefe72e0 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83b1600700 1 -- 172.21.15.16:0/929436561 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83b80a2540 con 0x55aeeefe72e0 2024-04-05T23:34:51.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83b1600700 1 -- 172.21.15.16:0/929436561 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f83b80b81a0 con 0x55aeeefe72e0 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83e2092b80 10 client.15750 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83e2092b80 10 client.15750 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83e2092b80 20 client.15750 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83e2092b80 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55aeef055e90 0x55aeef058280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83e2092b80 4 client.15750 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83e2092b80 1 -- 172.21.15.16:0/929436561 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55aeeed86cf0 con 0x55aeef055e90 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.940+0000 7f83c6a00700 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55aeef055e90 0x55aeef058280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83b1600700 10 client.15750 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83c6a00700 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55aeef055e90 0x55aeef058280 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2318c4285189d2dc server_cookie=21bb86bd36284f88 in_seq=0 out_seq=0 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83b1600700 1 -- 172.21.15.16:0/929436561 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55aeeed86cf0 con 0x55aeef055e90 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83b1600700 10 client.15750 handle_command_reply: tid=0 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83e2092b80 1 client.15750 shutdown 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83e2092b80 1 -- 172.21.15.16:0/929436561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe72e0 msgr2=0x55aeeeff66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83e2092b80 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe72e0 0x55aeeeff66e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f83b800e970 tx=0x7f83b809a900 comp rx=0 tx=0).stop 2024-04-05T23:34:51.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83e2092b80 1 -- 172.21.15.16:0/929436561 shutdown_connections 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83e2092b80 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aeeefe72e0 0x55aeeeff66e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83e2092b80 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55aeef055e90 0x55aeef058280 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83e2092b80 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aeeefed180 0x55aeef054340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83e2092b80 1 --2- 172.21.15.16:0/929436561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aeeefe7ca0 0x55aeeeff6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83b1600700 0 client.15750 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.941+0000 7f83e2092b80 1 -- 172.21.15.16:0/929436561 >> 172.21.15.16:0/929436561 conn(0x55aeeef5b220 msgr2=0x55aeeefef9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83e2092b80 1 -- 172.21.15.16:0/929436561 shutdown_connections 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83e2092b80 1 -- 172.21.15.16:0/929436561 wait complete. 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83e2092b80 20 client.15750 trim_cache size 0 max 16384 2024-04-05T23:34:51.938 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 -- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83ac0a13b0 msgr2=0x7f83ac0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83ac0a13b0 0x7f83ac0a3870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f83c000b780 tx=0x7f83c00095c0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 -- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81010a0 msgr2=0x7f83c8199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81010a0 0x7f83c8199640 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f83b800e9a0 tx=0x7f83b800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 -- 172.21.15.16:0/3007382233 shutdown_connections 2024-04-05T23:34:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83ac0a13b0 0x7f83ac0a3870 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c81010a0 0x7f83c8199640 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c819df20 0x7f83c819e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 --2- 172.21.15.16:0/3007382233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83c81061c0 0x7f83c8199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.942+0000 7f83ce600700 1 -- 172.21.15.16:0/3007382233 >> 172.21.15.16:0/3007382233 conn(0x7f83c80fc900 msgr2=0x7f83c80fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.943+0000 7f83ce600700 1 -- 172.21.15.16:0/3007382233 shutdown_connections 2024-04-05T23:34:51.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:51.943+0000 7f83ce600700 1 -- 172.21.15.16:0/3007382233 wait complete. 2024-04-05T23:34:51.951 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:51 smithi016 ceph-mon[38248]: pgmap v42: 97 pgs: 97 active+clean; 8.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 721 KiB/s wr, 128 op/s 2024-04-05T23:34:51.951 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:51 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3855392460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:34:51.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-05T23:34:51.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21398 12.20 MB/sec warmup 11 sec latency 1053.417 ms 2024-04-05T23:34:52.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:51 smithi130 ceph-mon[79417]: pgmap v42: 97 pgs: 97 active+clean; 8.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 721 KiB/s wr, 128 op/s 2024-04-05T23:34:52.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:51 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3855392460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:34:52.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.164+0000 7f11c8a00700 1 -- 172.21.15.16:0/1890522045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c4101a50 msgr2=0x7f11c4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.164+0000 7f11c8a00700 1 --2- 172.21.15.16:0/1890522045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c4101a50 0x7f11c4105a90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11b00098f0 tx=0x7f11b0009c00 comp rx=0 tx=0).stop 2024-04-05T23:34:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.164+0000 7f11c8a00700 1 -- 172.21.15.16:0/1890522045 shutdown_connections 2024-04-05T23:34:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.164+0000 7f11c8a00700 1 --2- 172.21.15.16:0/1890522045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c41061c0 0x7f11c410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.164+0000 7f11c8a00700 1 --2- 172.21.15.16:0/1890522045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c41010a0 0x7f11c4101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.164+0000 7f11c8a00700 1 --2- 172.21.15.16:0/1890522045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c4101a50 0x7f11c4105a90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.164+0000 7f11c8a00700 1 -- 172.21.15.16:0/1890522045 >> 172.21.15.16:0/1890522045 conn(0x7f11c40fc900 msgr2=0x7f11c40fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.164+0000 7f11c8a00700 1 -- 172.21.15.16:0/1890522045 shutdown_connections 2024-04-05T23:34:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.164+0000 7f11c8a00700 1 -- 172.21.15.16:0/1890522045 wait complete. 2024-04-05T23:34:52.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.165+0000 7f11c8a00700 1 Processor -- start 2024-04-05T23:34:52.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.165+0000 7f11c8a00700 1 -- start start 2024-04-05T23:34:52.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.165+0000 7f11c8a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c41010a0 0x7f11c4199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.165+0000 7f11c8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c41061c0 0x7f11c4199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.165+0000 7f11c8a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c419de50 0x7f11c419e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.165+0000 7f11c8a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c419e8d0 con 0x7f11c419de50 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.165+0000 7f11c8a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c419ea40 con 0x7f11c41061c0 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.165+0000 7f11c8a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c419ec00 con 0x7f11c41010a0 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c419de50 0x7f11c419e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c419de50 0x7f11c419e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52862/0 (socket says 172.21.15.16:52862) 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c1e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c41010a0 0x7f11c4199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c2800700 1 -- 172.21.15.16:0/2900313237 learned_addr learned my addr 172.21.15.16:0/2900313237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c1400700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c41061c0 0x7f11c4199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c2800700 1 -- 172.21.15.16:0/2900313237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c41010a0 msgr2=0x7f11c4199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c2800700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c41010a0 0x7f11c4199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c2800700 1 -- 172.21.15.16:0/2900313237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c41061c0 msgr2=0x7f11c4199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c2800700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c41061c0 0x7f11c4199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c2800700 1 -- 172.21.15.16:0/2900313237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc009d00 con 0x7f11c419de50 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.166+0000 7f11c2800700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c419de50 0x7f11c419e2d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00e3c0 tx=0x7f11bc00e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.167+0000 7f11baa00700 1 -- 172.21.15.16:0/2900313237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc0090d0 con 0x7f11c419de50 2024-04-05T23:34:52.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.167+0000 7f11c8a00700 1 -- 172.21.15.16:0/2900313237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b0009520 con 0x7f11c419de50 2024-04-05T23:34:52.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.167+0000 7f11baa00700 1 -- 172.21.15.16:0/2900313237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11bc00f040 con 0x7f11c419de50 2024-04-05T23:34:52.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.167+0000 7f11baa00700 1 -- 172.21.15.16:0/2900313237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc007c20 con 0x7f11c419de50 2024-04-05T23:34:52.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.167+0000 7f11c8a00700 1 -- 172.21.15.16:0/2900313237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c41a64b0 con 0x7f11c419de50 2024-04-05T23:34:52.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.170+0000 7f11baa00700 1 -- 172.21.15.16:0/2900313237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f11bc003a00 con 0x7f11c419de50 2024-04-05T23:34:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.171+0000 7f11dc51fb80 1 Processor -- start 2024-04-05T23:34:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.171+0000 7f11baa00700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f11a80a13b0 0x7f11a80a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.171+0000 7f11dc51fb80 1 -- start start 2024-04-05T23:34:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.171+0000 7f11baa00700 1 -- 172.21.15.16:0/2900313237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f11bc0ccec0 con 0x7f11c419de50 2024-04-05T23:34:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.171+0000 7f11c1400700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f11a80a13b0 0x7f11a80a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197f4180 0x55de197f4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 0x55de197ee760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197faf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de19754be0 con 0x55de197eeca0 2024-04-05T23:34:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de19745990 con 0x55de197f4180 2024-04-05T23:34:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de19743350 con 0x55de197ee2e0 2024-04-05T23:34:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11c2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197faf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11c2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197faf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52874/0 (socket says 172.21.15.16:52874) 2024-04-05T23:34:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11c2800700 1 -- 172.21.15.16:0/1719423642 learned_addr learned my addr 172.21.15.16:0/1719423642 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11c1e00700 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197f4180 0x55de197f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11c1400700 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 0x55de197ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11c1400700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f11a80a13b0 0x7f11a80a3870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f11b0005e90 tx=0x7f11b0019040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11c1e00700 1 -- 172.21.15.16:0/1719423642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 msgr2=0x55de197ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.172+0000 7f11c1e00700 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 0x55de197ee760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.173+0000 7f11c1e00700 1 -- 172.21.15.16:0/1719423642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 msgr2=0x55de197faf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.173+0000 7f11c1e00700 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197faf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.173+0000 7f11c1e00700 1 -- 172.21.15.16:0/1719423642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de195c03b0 con 0x55de197f4180 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.173+0000 7f11c1400700 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 0x55de197ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.173+0000 7f11c2800700 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197faf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.173+0000 7f11c1e00700 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197f4180 0x55de197f4560 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009cb0 tx=0x7f11b4011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb7a0de746689afa server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.173+0000 7f11af000700 1 -- 172.21.15.16:0/1719423642 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4011d20 con 0x55de197f4180 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.173+0000 7f11af000700 1 -- 172.21.15.16:0/1719423642 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11b4011e80 con 0x55de197f4180 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.173+0000 7f11af000700 1 -- 172.21.15.16:0/1719423642 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4005220 con 0x55de197f4180 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 -- 172.21.15.16:0/1719423642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197f4180 msgr2=0x55de197f4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197f4180 0x55de197f4560 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009cb0 tx=0x7f11b4011440 comp rx=0 tx=0).stop 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 -- 172.21.15.16:0/1719423642 shutdown_connections 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197faf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197f4180 0x55de197f4560 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/1719423642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 0x55de197ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 -- 172.21.15.16:0/1719423642 >> 172.21.15.16:0/1719423642 conn(0x55de19762220 msgr2=0x55de197e9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 -- 172.21.15.16:0/1719423642 shutdown_connections 2024-04-05T23:34:52.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 -- 172.21.15.16:0/1719423642 wait complete. 2024-04-05T23:34:52.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.174+0000 7f11dc51fb80 1 -- start start 2024-04-05T23:34:52.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11dc51fb80 1 Processor -- start 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 0x55de197fad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 0x55de197f8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197f4180 0x55de197f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1973ec40 con 0x55de197f4180 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de196dad80 con 0x55de197eeca0 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de196daa20 con 0x55de197ee2e0 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11dc51fb80 10 client.0 _ll_register_callbacks cb 0x55de195463a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11c2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197f4180 0x55de197f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11c2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197f4180 0x55de197f85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52878/0 (socket says 172.21.15.16:52878) 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.175+0000 7f11c2800700 1 -- 172.21.15.16:0/3680637102 learned_addr learned my addr 172.21.15.16:0/3680637102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11c1e00700 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 0x55de197fad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11c1400700 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 0x55de197f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11c2800700 1 -- 172.21.15.16:0/3680637102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 msgr2=0x55de197fad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11c2800700 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 0x55de197fad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11c2800700 1 -- 172.21.15.16:0/3680637102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 msgr2=0x55de197f8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11c2800700 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 0x55de197f8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11c2800700 1 -- 172.21.15.16:0/3680637102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c41a64b0 con 0x55de197f4180 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11c2800700 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197f4180 0x55de197f85f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0095b0 tx=0x7f11bc099990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b105903d846ded0c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11a3400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11a3400700 1 -- 172.21.15.16:0/3680637102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc099de0 con 0x55de197f4180 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11a3400700 1 -- 172.21.15.16:0/3680637102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f11bc09dc10 con 0x55de197f4180 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.176+0000 7f11dc51fb80 1 -- 172.21.15.16:0/3680637102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de19737700 con 0x55de197f4180 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11a3400700 1 -- 172.21.15.16:0/3680637102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11bc0a21d0 con 0x55de197f4180 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11dc51fb80 10 client.15762 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11dc51fb80 1 -- 172.21.15.16:0/3680637102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f11b40098e0 con 0x55de197f4180 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11a3400700 1 -- 172.21.15.16:0/3680637102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11bc063020 con 0x55de197f4180 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11a3400700 1 -- 172.21.15.16:0/3680637102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f11bc09a040 con 0x55de197f4180 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11dc51fb80 10 client.15762 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11dc51fb80 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11dc51fb80 20 client.15762 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de19808430 0x55de1980a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11dc51fb80 4 client.15762 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11dc51fb80 1 -- 172.21.15.16:0/3680637102 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de1958dcf0 con 0x55de19808430 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.177+0000 7f11c1e00700 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de19808430 0x55de1980a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.178+0000 7f11c1e00700 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de19808430 0x55de1980a850 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b74b70e91698a78e server_cookie=523b39f993cac991 in_seq=0 out_seq=0 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.178+0000 7f11a3400700 10 client.15762 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11a3400700 1 -- 172.21.15.16:0/3680637102 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55de1958dcf0 con 0x55de19808430 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11a3400700 10 client.15762 handle_command_reply: tid=0 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 client.15762 shutdown 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 -- 172.21.15.16:0/3680637102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197f4180 msgr2=0x55de197f85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197f4180 0x55de197f85f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0095b0 tx=0x7f11bc099990 comp rx=0 tx=0).stop 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 -- 172.21.15.16:0/3680637102 shutdown_connections 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de19808430 0x55de1980a850 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197f4180 0x55de197f85f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197ee2e0 0x55de197fad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/3680637102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 0x55de197f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 -- 172.21.15.16:0/3680637102 >> 172.21.15.16:0/3680637102 conn(0x55de19762220 msgr2=0x55de197e3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11a3400700 0 client.15762 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 -- 172.21.15.16:0/3680637102 shutdown_connections 2024-04-05T23:34:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 1 -- 172.21.15.16:0/3680637102 wait complete. 2024-04-05T23:34:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.179+0000 7f11dc51fb80 20 client.15762 trim_cache size 0 max 16384 2024-04-05T23:34:52.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.193+0000 7f11dc51fb80 1 Processor -- start 2024-04-05T23:34:52.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.193+0000 7f11dc51fb80 1 -- start start 2024-04-05T23:34:52.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197ee2e0 0x55de197f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197ff510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 0x55de197e3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de195c26a0 con 0x55de197eeca0 2024-04-05T23:34:52.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de197ffa50 con 0x55de197ee2e0 2024-04-05T23:34:52.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de197e3aa0 con 0x55de197f4180 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c1400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197ff510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c1400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197ff510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52894/0 (socket says 172.21.15.16:52894) 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c1400700 1 -- 172.21.15.16:0/156524526 learned_addr learned my addr 172.21.15.16:0/156524526 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c1e00700 1 --2- 172.21.15.16:0/156524526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197ee2e0 0x55de197f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c2800700 1 --2- 172.21.15.16:0/156524526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 0x55de197e3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c1400700 1 -- 172.21.15.16:0/156524526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 msgr2=0x55de197e3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c1400700 1 --2- 172.21.15.16:0/156524526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 0x55de197e3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c1400700 1 -- 172.21.15.16:0/156524526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197ee2e0 msgr2=0x55de197f39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c1400700 1 --2- 172.21.15.16:0/156524526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197ee2e0 0x55de197f39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.194+0000 7f11c1400700 1 -- 172.21.15.16:0/156524526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de1956cb60 con 0x55de197eeca0 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.195+0000 7f11c1400700 1 --2- 172.21.15.16:0/156524526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197ff510 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f11b000c010 tx=0x7f11b001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de52acaf92476f6a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:52.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.195+0000 7f11afa00700 1 -- 172.21.15.16:0/156524526 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b000f7e0 con 0x55de197eeca0 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.195+0000 7f11afa00700 1 -- 172.21.15.16:0/156524526 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11b002b040 con 0x55de197eeca0 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.195+0000 7f11afa00700 1 -- 172.21.15.16:0/156524526 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b000bb90 con 0x55de197eeca0 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 -- 172.21.15.16:0/156524526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 msgr2=0x55de197ff510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/156524526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197ff510 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f11b000c010 tx=0x7f11b001e000 comp rx=0 tx=0).stop 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 -- 172.21.15.16:0/156524526 shutdown_connections 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/156524526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 0x55de197e3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/156524526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197ee2e0 0x55de197f39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/156524526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197eeca0 0x55de197ff510 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 -- 172.21.15.16:0/156524526 >> 172.21.15.16:0/156524526 conn(0x55de19762220 msgr2=0x55de197e9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 -- 172.21.15.16:0/156524526 shutdown_connections 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 -- 172.21.15.16:0/156524526 wait complete. 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 -- start start 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.196+0000 7f11dc51fb80 1 Processor -- start 2024-04-05T23:34:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197ee2e0 0x55de198559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 0x55de19855f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11dc51fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 0x55de198501f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de198507c0 con 0x55de197ee2e0 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de19850930 con 0x55de197eeca0 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11dc51fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de19850af0 con 0x55de197f4180 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11dc51fb80 10 client.0 _ll_register_callbacks cb 0x55de195463a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197ee2e0 0x55de198559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197ee2e0 0x55de198559d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52904/0 (socket says 172.21.15.16:52904) 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c1e00700 1 -- 172.21.15.16:0/786028254 learned_addr learned my addr 172.21.15.16:0/786028254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c1400700 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 0x55de19855f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c2800700 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 0x55de198501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c1e00700 1 -- 172.21.15.16:0/786028254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 msgr2=0x55de198501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c1e00700 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 0x55de198501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c1e00700 1 -- 172.21.15.16:0/786028254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 msgr2=0x55de19855f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c1e00700 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 0x55de19855f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c1e00700 1 -- 172.21.15.16:0/786028254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c410e5b0 con 0x55de197ee2e0 2024-04-05T23:34:52.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.197+0000 7f11c2800700 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 0x55de198501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.198+0000 7f11c1e00700 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197ee2e0 0x55de198559d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009cb0 tx=0x7f11b4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b88dc261e595cf server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.198+0000 7f11a3400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.198+0000 7f11a3400700 1 -- 172.21.15.16:0/786028254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4029040 con 0x55de197ee2e0 2024-04-05T23:34:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.198+0000 7f11a3400700 1 -- 172.21.15.16:0/786028254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f11b4024420 con 0x55de197ee2e0 2024-04-05T23:34:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.198+0000 7f11dc51fb80 1 -- 172.21.15.16:0/786028254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de197e1490 con 0x55de197ee2e0 2024-04-05T23:34:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.198+0000 7f11a3400700 1 -- 172.21.15.16:0/786028254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11b40245e0 con 0x55de197ee2e0 2024-04-05T23:34:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.198+0000 7f11dc51fb80 10 client.15768 fetch_fsmap learned FSMap version 31 2024-04-05T23:34:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.198+0000 7f11dc51fb80 1 -- 172.21.15.16:0/786028254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55de19704ea0 con 0x55de197ee2e0 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.199+0000 7f11a3400700 1 -- 172.21.15.16:0/786028254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55de197e1490 con 0x55de197ee2e0 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.199+0000 7f11a3400700 1 -- 172.21.15.16:0/786028254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 31) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f11b4038070 con 0x55de197ee2e0 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.199+0000 7f11dc51fb80 10 client.15768 fetch_fsmap finished waiting for FSMap version 31 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.199+0000 7f11dc51fb80 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.199+0000 7f11dc51fb80 20 client.15768 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.199+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1985d840 0x55de1985fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.199+0000 7f11dc51fb80 4 client.15768 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.199+0000 7f11dc51fb80 1 -- 172.21.15.16:0/786028254 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55de195c2ce0 con 0x55de1985d840 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.199+0000 7f11c1400700 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1985d840 0x55de1985fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11c1400700 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1985d840 0x55de1985fc60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60950d0d30c51812 server_cookie=40140b62479fb8a7 in_seq=0 out_seq=0 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11a3400700 10 client.15768 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11a3400700 1 -- 172.21.15.16:0/786028254 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55de195c2ce0 con 0x55de1985d840 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11a3400700 10 client.15768 handle_command_reply: tid=0 2024-04-05T23:34:52.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 client.15768 shutdown 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 -- 172.21.15.16:0/786028254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197ee2e0 msgr2=0x55de198559d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197ee2e0 0x55de198559d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009cb0 tx=0x7f11b4009bf0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 -- 172.21.15.16:0/786028254 shutdown_connections 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1985d840 0x55de1985fc60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de197ee2e0 0x55de198559d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de197f4180 0x55de198501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11a3400700 0 client.15768 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 --2- 172.21.15.16:0/786028254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de197eeca0 0x55de19855f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 -- 172.21.15.16:0/786028254 >> 172.21.15.16:0/786028254 conn(0x55de19762220 msgr2=0x55de19803c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 -- 172.21.15.16:0/786028254 shutdown_connections 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.200+0000 7f11dc51fb80 1 -- 172.21.15.16:0/786028254 wait complete. 2024-04-05T23:34:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11dc51fb80 20 client.15768 trim_cache size 0 max 16384 2024-04-05T23:34:52.198 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:34:52.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 -- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f11a80a13b0 msgr2=0x7f11a80a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:52.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f11a80a13b0 0x7f11a80a3870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f11b0005e90 tx=0x7f11b0019040 comp rx=0 tx=0).stop 2024-04-05T23:34:52.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 -- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c419de50 msgr2=0x7f11c419e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:52.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c419de50 0x7f11c419e2d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00e3c0 tx=0x7f11bc00e6d0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 -- 172.21.15.16:0/2900313237 shutdown_connections 2024-04-05T23:34:52.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f11a80a13b0 0x7f11a80a3870 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11c419de50 0x7f11c419e2d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c41010a0 0x7f11c4199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 --2- 172.21.15.16:0/2900313237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c41061c0 0x7f11c4199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:52.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 -- 172.21.15.16:0/2900313237 >> 172.21.15.16:0/2900313237 conn(0x7f11c40fc900 msgr2=0x7f11c40fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:52.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 -- 172.21.15.16:0/2900313237 shutdown_connections 2024-04-05T23:34:52.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:52.201+0000 7f11c8a00700 1 -- 172.21.15.16:0/2900313237 wait complete. 2024-04-05T23:34:52.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:51 smithi027 ceph-mon[78597]: pgmap v42: 97 pgs: 97 active+clean; 8.3 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 721 KiB/s wr, 128 op/s 2024-04-05T23:34:52.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:51 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3855392460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:34:52.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21398 11.19 MB/sec warmup 12 sec latency 2053.487 ms 2024-04-05T23:34:53.213 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-05T23:34:53.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-05T23:34:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.436+0000 7f2877600700 1 -- 172.21.15.16:0/3574459445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f287010d670 msgr2=0x7f287010fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.436+0000 7f2877600700 1 --2- 172.21.15.16:0/3574459445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f287010d670 0x7f287010fae0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f286800b0a0 tx=0x7f286800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.436+0000 7f2877600700 1 -- 172.21.15.16:0/3574459445 shutdown_connections 2024-04-05T23:34:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.436+0000 7f2877600700 1 --2- 172.21.15.16:0/3574459445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f287010d670 0x7f287010fae0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.436+0000 7f2877600700 1 --2- 172.21.15.16:0/3574459445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2870073310 0x7f28700736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.436+0000 7f2877600700 1 --2- 172.21.15.16:0/3574459445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2870073c30 0x7f287010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.436+0000 7f2877600700 1 -- 172.21.15.16:0/3574459445 >> 172.21.15.16:0/3574459445 conn(0x7f28700fc920 msgr2=0x7f28700fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.436+0000 7f2877600700 1 -- 172.21.15.16:0/3574459445 shutdown_connections 2024-04-05T23:34:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.436+0000 7f2877600700 1 -- 172.21.15.16:0/3574459445 wait complete. 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.437+0000 7f2877600700 1 Processor -- start 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.437+0000 7f2877600700 1 -- start start 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2877600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870073310 0x7f2870199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2877600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2870073c30 0x7f2870199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2877600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f287010d670 0x7f287019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2877600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287019e590 con 0x7f2870073310 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2877600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287019e700 con 0x7f287010d670 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2877600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287019e8c0 con 0x7f2870073c30 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2874c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870073310 0x7f2870199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2874c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870073310 0x7f2870199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52930/0 (socket says 172.21.15.16:52930) 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2874c00700 1 -- 172.21.15.16:0/494371850 learned_addr learned my addr 172.21.15.16:0/494371850 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2875600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f287010d670 0x7f287019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f286fe00700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2870073c30 0x7f2870199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2875600700 1 -- 172.21.15.16:0/494371850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2870073c30 msgr2=0x7f2870199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2875600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2870073c30 0x7f2870199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2875600700 1 -- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870073310 msgr2=0x7f2870199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2875600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870073310 0x7f2870199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2875600700 1 -- 172.21.15.16:0/494371850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868009d00 con 0x7f287010d670 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.438+0000 7f2874c00700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870073310 0x7f2870199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.439+0000 7f2875600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f287010d670 0x7f287019df50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f286800b070 tx=0x7f2868007860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.439+0000 7f286d600700 1 -- 172.21.15.16:0/494371850 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286800e040 con 0x7f287010d670 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.439+0000 7f286d600700 1 -- 172.21.15.16:0/494371850 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f286801d470 con 0x7f287010d670 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.439+0000 7f286d600700 1 -- 172.21.15.16:0/494371850 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868005280 con 0x7f287010d670 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.439+0000 7f2877600700 1 -- 172.21.15.16:0/494371850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287019eb40 con 0x7f287010d670 2024-04-05T23:34:53.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.439+0000 7f2877600700 1 -- 172.21.15.16:0/494371850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28701a63e0 con 0x7f287010d670 2024-04-05T23:34:53.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.443+0000 7f286d600700 1 -- 172.21.15.16:0/494371850 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f286801d5e0 con 0x7f287010d670 2024-04-05T23:34:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f288b0f3b80 1 Processor -- start 2024-04-05T23:34:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f288b0f3b80 1 -- start start 2024-04-05T23:34:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f286d600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28580a1430 0x7f28580a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70725e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f286d600700 1 -- 172.21.15.16:0/494371850 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f28680ce280 con 0x7f287010d670 2024-04-05T23:34:53.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df706c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7078f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df6fd2c60 con 0x563df706cd20 2024-04-05T23:34:53.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df6fc3a10 con 0x563df7072200 2024-04-05T23:34:53.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df6fc13d0 con 0x563df706c360 2024-04-05T23:34:53.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f286fe00700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28580a1430 0x7f28580a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.444+0000 7f2874c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f286fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df706c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f286fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df706c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39660/0 (socket says 172.21.15.16:39660) 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f2874c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:49456/0 (socket says 172.21.15.16:49456) 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f286fe00700 1 -- 172.21.15.16:0/1808112071 learned_addr learned my addr 172.21.15.16:0/1808112071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f2875600700 1 --2- 172.21.15.16:0/1808112071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7078f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f2874c00700 1 -- 172.21.15.16:0/1808112071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 msgr2=0x563df706c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f2874c00700 1 --2- 172.21.15.16:0/1808112071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df706c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f2874c00700 1 -- 172.21.15.16:0/1808112071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 msgr2=0x563df7078f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f2874c00700 1 --2- 172.21.15.16:0/1808112071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7078f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f2874c00700 1 -- 172.21.15.16:0/1808112071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563df6e3e430 con 0x563df7072200 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f286fe00700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28580a1430 0x7f28580a38f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f285c013a30 tx=0x7f285c009bd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.445+0000 7f2874c00700 1 --2- 172.21.15.16:0/1808112071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70725e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2864011570 tx=0x7f2864011880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=429b81cad45f6ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f2861c00700 1 -- 172.21.15.16:0/1808112071 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864004230 con 0x563df7072200 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f2861c00700 1 -- 172.21.15.16:0/1808112071 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2864004390 con 0x563df7072200 2024-04-05T23:34:53.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f2861c00700 1 -- 172.21.15.16:0/1808112071 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28640197e0 con 0x563df7072200 2024-04-05T23:34:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f288b0f3b80 1 -- 172.21.15.16:0/1808112071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 msgr2=0x563df70725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/1808112071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70725e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2864011570 tx=0x7f2864011880 comp rx=0 tx=0).stop 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f288b0f3b80 1 -- 172.21.15.16:0/1808112071 shutdown_connections 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/1808112071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7078f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/1808112071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70725e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/1808112071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df706c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f288b0f3b80 1 -- 172.21.15.16:0/1808112071 >> 172.21.15.16:0/1808112071 conn(0x563df6fe02a0 msgr2=0x563df70675f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f288b0f3b80 1 -- 172.21.15.16:0/1808112071 shutdown_connections 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.446+0000 7f288b0f3b80 1 -- 172.21.15.16:0/1808112071 wait complete. 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.447+0000 7f288b0f3b80 1 -- start start 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.447+0000 7f288b0f3b80 1 Processor -- start 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706c360 0x563df7078f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7076210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df7072200 0x563df7076950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df6fbccc0 con 0x563df706cd20 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df6f58e00 con 0x563df706c360 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df6f58aa0 con 0x563df7072200 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f288b0f3b80 10 client.0 _ll_register_callbacks cb 0x563df6dc4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f286fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7076210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f286fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7076210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52942/0 (socket says 172.21.15.16:52942) 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f286fe00700 1 -- 172.21.15.16:0/991006057 learned_addr learned my addr 172.21.15.16:0/991006057 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.448+0000 7f2874c00700 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706c360 0x563df7078f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f2875600700 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df7072200 0x563df7076950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f286fe00700 1 -- 172.21.15.16:0/991006057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df7072200 msgr2=0x563df7076950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f286fe00700 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df7072200 0x563df7076950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f286fe00700 1 -- 172.21.15.16:0/991006057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706c360 msgr2=0x563df7078f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f286fe00700 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706c360 0x563df7078f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f286fe00700 1 -- 172.21.15.16:0/991006057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2864009d60 con 0x563df706cd20 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f286fe00700 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7076210 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f285c009a00 tx=0x7f285c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6aef5f756748f37 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f2856000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f2856000700 1 -- 172.21.15.16:0/991006057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c009830 con 0x563df706cd20 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f288b0f3b80 1 -- 172.21.15.16:0/991006057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563df705f510 con 0x563df706cd20 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.449+0000 7f2856000700 1 -- 172.21.15.16:0/991006057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f285c004830 con 0x563df706cd20 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f2856000700 1 -- 172.21.15.16:0/991006057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f285c011e70 con 0x563df706cd20 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f288b0f3b80 10 client.15780 fetch_fsmap learned FSMap version 32 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f288b0f3b80 1 -- 172.21.15.16:0/991006057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x563df6deabe0 con 0x563df706cd20 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f2856000700 1 -- 172.21.15.16:0/991006057 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563df705f510 con 0x563df706cd20 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f2856000700 1 -- 172.21.15.16:0/991006057 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f285c027030 con 0x563df706cd20 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f288b0f3b80 10 client.15780 fetch_fsmap finished waiting for FSMap version 32 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f288b0f3b80 10 client.15780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f288b0f3b80 20 client.15780 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563df70865a0 0x563df70889c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f288b0f3b80 4 client.15780 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:53.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.450+0000 7f288b0f3b80 1 -- 172.21.15.16:0/991006057 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563df6fb5780 con 0x563df70865a0 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.451+0000 7f2874c00700 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563df70865a0 0x563df70889c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.451+0000 7f2874c00700 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563df70865a0 0x563df70889c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d66224a6312bae7 server_cookie=dd8bd37817a53e13 in_seq=0 out_seq=0 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.451+0000 7f2856000700 10 client.15780 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f2856000700 1 -- 172.21.15.16:0/991006057 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563df6fb5780 con 0x563df70865a0 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f2856000700 10 client.15780 handle_command_reply: tid=0 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 client.15780 shutdown 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 -- 172.21.15.16:0/991006057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 msgr2=0x563df7076210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7076210 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f285c009a00 tx=0x7f285c009260 comp rx=0 tx=0).stop 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 -- 172.21.15.16:0/991006057 shutdown_connections 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563df70865a0 0x563df70889c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df7072200 0x563df7076950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706c360 0x563df7078f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f2856000700 0 client.15780 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:53.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/991006057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df7076210 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 -- 172.21.15.16:0/991006057 >> 172.21.15.16:0/991006057 conn(0x563df6fe02a0 msgr2=0x563df70614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 -- 172.21.15.16:0/991006057 shutdown_connections 2024-04-05T23:34:53.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 1 -- 172.21.15.16:0/991006057 wait complete. 2024-04-05T23:34:53.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.452+0000 7f288b0f3b80 20 client.15780 trim_cache size 0 max 16384 2024-04-05T23:34:53.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.470+0000 7f288b0f3b80 1 Processor -- start 2024-04-05T23:34:53.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.470+0000 7f288b0f3b80 1 -- start start 2024-04-05T23:34:53.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df7068980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df707d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df6e40720 con 0x563df706cd20 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df7077810 con 0x563df7072200 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df706d100 con 0x563df706c360 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f286fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df707d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f2875600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f286fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df707d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52954/0 (socket says 172.21.15.16:52954) 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f286fe00700 1 -- 172.21.15.16:0/3839433151 learned_addr learned my addr 172.21.15.16:0/3839433151 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f2874c00700 1 --2- 172.21.15.16:0/3839433151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df7068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f2875600700 1 -- 172.21.15.16:0/3839433151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 msgr2=0x563df7068980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f2875600700 1 --2- 172.21.15.16:0/3839433151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df7068980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f2875600700 1 -- 172.21.15.16:0/3839433151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 msgr2=0x563df707d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f2875600700 1 --2- 172.21.15.16:0/3839433151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df707d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f2875600700 1 -- 172.21.15.16:0/3839433151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563df708d610 con 0x563df7072200 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.471+0000 7f286fe00700 1 --2- 172.21.15.16:0/3839433151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df707d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.472+0000 7f2875600700 1 --2- 172.21.15.16:0/3839433151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70615e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f28680cb820 tx=0x7f28680099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=657dce9fb71b08f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.472+0000 7f2862600700 1 -- 172.21.15.16:0/3839433151 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286809b1a0 con 0x563df7072200 2024-04-05T23:34:53.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.472+0000 7f2862600700 1 -- 172.21.15.16:0/3839433151 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f286809efd0 con 0x563df7072200 2024-04-05T23:34:53.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.472+0000 7f2862600700 1 -- 172.21.15.16:0/3839433151 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286809b400 con 0x563df7072200 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.472+0000 7f288b0f3b80 1 -- 172.21.15.16:0/3839433151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 msgr2=0x563df70615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.472+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/3839433151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70615e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f28680cb820 tx=0x7f28680099e0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.473+0000 7f288b0f3b80 1 -- 172.21.15.16:0/3839433151 shutdown_connections 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.473+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/3839433151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df7072200 0x563df70615e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.473+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/3839433151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df7068980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.473+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/3839433151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df706cd20 0x563df707d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.473+0000 7f288b0f3b80 1 -- 172.21.15.16:0/3839433151 >> 172.21.15.16:0/3839433151 conn(0x563df6fe02a0 msgr2=0x563df70670c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.473+0000 7f288b0f3b80 1 -- 172.21.15.16:0/3839433151 shutdown_connections 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.473+0000 7f288b0f3b80 1 -- 172.21.15.16:0/3839433151 wait complete. 2024-04-05T23:34:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.473+0000 7f288b0f3b80 1 -- start start 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.473+0000 7f288b0f3b80 1 Processor -- start 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df707d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706cd20 0x563df707b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f288b0f3b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df7072200 0x563df707baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df707c030 con 0x563df7072200 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df707c1a0 con 0x563df706cd20 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f288b0f3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563df707c360 con 0x563df706c360 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f288b0f3b80 10 client.0 _ll_register_callbacks cb 0x7f285c013350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2874c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df707d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2875600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df7072200 0x563df707baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2874c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df707d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39690/0 (socket says 172.21.15.16:39690) 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2874c00700 1 -- 172.21.15.16:0/486470217 learned_addr learned my addr 172.21.15.16:0/486470217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f286fe00700 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706cd20 0x563df707b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2874c00700 1 -- 172.21.15.16:0/486470217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706cd20 msgr2=0x563df707b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2874c00700 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706cd20 0x563df707b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2874c00700 1 -- 172.21.15.16:0/486470217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df7072200 msgr2=0x563df707baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2874c00700 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df7072200 0x563df707baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2874c00700 1 -- 172.21.15.16:0/486470217 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563df6deabe0 con 0x563df706c360 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.474+0000 7f2875600700 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df7072200 0x563df707baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.475+0000 7f2874c00700 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df707d240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2864005e90 tx=0x7f28640052d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b8835f4c8fc6620 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.475+0000 7f2856000700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.475+0000 7f2856000700 1 -- 172.21.15.16:0/486470217 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864029040 con 0x563df706c360 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.475+0000 7f288b0f3b80 1 -- 172.21.15.16:0/486470217 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2864004390 con 0x563df706c360 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.475+0000 7f2856000700 1 -- 172.21.15.16:0/486470217 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f286402b810 con 0x563df706c360 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.475+0000 7f2856000700 1 -- 172.21.15.16:0/486470217 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2864005700 con 0x563df706c360 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.475+0000 7f288b0f3b80 10 client.25855 fetch_fsmap learned FSMap version 32 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.475+0000 7f288b0f3b80 1 -- 172.21.15.16:0/486470217 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7f28701a63e0 con 0x563df706c360 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.476+0000 7f2856000700 1 -- 172.21.15.16:0/486470217 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2864004390 con 0x563df706c360 2024-04-05T23:34:53.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.476+0000 7f2856000700 1 -- 172.21.15.16:0/486470217 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 32) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f28640115a0 con 0x563df706c360 2024-04-05T23:34:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.476+0000 7f288b0f3b80 10 client.25855 fetch_fsmap finished waiting for FSMap version 32 2024-04-05T23:34:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.476+0000 7f288b0f3b80 10 client.25855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.476+0000 7f288b0f3b80 20 client.25855 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.476+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563df70db830 0x563df70ddc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.476+0000 7f288b0f3b80 4 client.25855 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-05T23:34:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.476+0000 7f288b0f3b80 1 -- 172.21.15.16:0/486470217 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563df6e0bd70 con 0x563df70db830 2024-04-05T23:34:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.476+0000 7f286fe00700 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563df70db830 0x563df70ddc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.477+0000 7f286fe00700 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563df70db830 0x563df70ddc50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53ad0cb7bd89319e server_cookie=3c3df1ba9c9b5226 in_seq=0 out_seq=0 2024-04-05T23:34:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.477+0000 7f2856000700 10 client.25855 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.477+0000 7f2856000700 1 -- 172.21.15.16:0/486470217 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563df6e0bd70 con 0x563df70db830 2024-04-05T23:34:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.477+0000 7f2856000700 10 client.25855 handle_command_reply: tid=0 2024-04-05T23:34:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.477+0000 7f288b0f3b80 1 client.25855 shutdown 2024-04-05T23:34:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 -- 172.21.15.16:0/486470217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 msgr2=0x563df707d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df707d240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2864005e90 tx=0x7f28640052d0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 -- 172.21.15.16:0/486470217 shutdown_connections 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563df70db830 0x563df70ddc50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563df7072200 0x563df707baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563df706c360 0x563df707d240 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 --2- 172.21.15.16:0/486470217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563df706cd20 0x563df707b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 -- 172.21.15.16:0/486470217 >> 172.21.15.16:0/486470217 conn(0x563df6fe02a0 msgr2=0x563df706b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f2856000700 0 client.25855 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 -- 172.21.15.16:0/486470217 shutdown_connections 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 1 -- 172.21.15.16:0/486470217 wait complete. 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f288b0f3b80 20 client.25855 trim_cache size 0 max 16384 2024-04-05T23:34:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:34:53.475 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-04-05T23:34:53.475 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "9987f0ed-9576-49e7-8f88-9336b0067c59", 2024-04-05T23:34:53.475 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-04-05T23:34:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f2877600700 1 -- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28580a1430 msgr2=0x7f28580a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.478+0000 7f2877600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28580a1430 0x7f28580a38f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f285c013a30 tx=0x7f285c009bd0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 -- 172.21.15.16:0/494371850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f287010d670 msgr2=0x7f287019df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f287010d670 0x7f287019df50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f286800b070 tx=0x7f2868007860 comp rx=0 tx=0).stop 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 -- 172.21.15.16:0/494371850 shutdown_connections 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28580a1430 0x7f28580a38f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2870073310 0x7f2870199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f287010d670 0x7f287019df50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 --2- 172.21.15.16:0/494371850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2870073c30 0x7f2870199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 -- 172.21.15.16:0/494371850 >> 172.21.15.16:0/494371850 conn(0x7f28700fc920 msgr2=0x7f28700fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 -- 172.21.15.16:0/494371850 shutdown_connections 2024-04-05T23:34:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.479+0000 7f2877600700 1 -- 172.21.15.16:0/494371850 wait complete. 2024-04-05T23:34:53.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-05T23:34:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.701+0000 7f612ba00700 1 -- 172.21.15.16:0/1069004691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61241010c0 msgr2=0x7f61241014a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.701+0000 7f612ba00700 1 --2- 172.21.15.16:0/1069004691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61241010c0 0x7f61241014a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6118009990 tx=0x7f6118009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.701+0000 7f612ba00700 1 -- 172.21.15.16:0/1069004691 shutdown_connections 2024-04-05T23:34:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.701+0000 7f612ba00700 1 --2- 172.21.15.16:0/1069004691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61241010c0 0x7f61241014a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.701+0000 7f612ba00700 1 --2- 172.21.15.16:0/1069004691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61241061e0 0x7f612410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.701+0000 7f612ba00700 1 --2- 172.21.15.16:0/1069004691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6124101a70 0x7f6124105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.701+0000 7f612ba00700 1 -- 172.21.15.16:0/1069004691 >> 172.21.15.16:0/1069004691 conn(0x7f61240fc920 msgr2=0x7f61240fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.701+0000 7f612ba00700 1 -- 172.21.15.16:0/1069004691 shutdown_connections 2024-04-05T23:34:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.701+0000 7f612ba00700 1 -- 172.21.15.16:0/1069004691 wait complete. 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.702+0000 7f612ba00700 1 Processor -- start 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f612ba00700 1 -- start start 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f612ba00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61241010c0 0x7f6124197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f612ba00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61241061e0 0x7f6124197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f612ba00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f612419bc10 0x7f612419c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f612ba00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612419c690 con 0x7f612419bc10 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f612ba00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612419c800 con 0x7f61241010c0 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f612ba00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612419c9c0 con 0x7f61241061e0 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f6129a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f612419bc10 0x7f612419c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f6129000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61241010c0 0x7f6124197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f6129a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f612419bc10 0x7f612419c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52992/0 (socket says 172.21.15.16:52992) 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f6129a00700 1 -- 172.21.15.16:0/1367897929 learned_addr learned my addr 172.21.15.16:0/1367897929 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.703+0000 7f6123e00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61241061e0 0x7f6124197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f6129a00700 1 -- 172.21.15.16:0/1367897929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61241061e0 msgr2=0x7f6124197870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f6129a00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61241061e0 0x7f6124197870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f6129a00700 1 -- 172.21.15.16:0/1367897929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61241010c0 msgr2=0x7f6124197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f6129a00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61241010c0 0x7f6124197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f6129a00700 1 -- 172.21.15.16:0/1367897929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61180095f0 con 0x7f612419bc10 2024-04-05T23:34:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f6129a00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f612419bc10 0x7f612419c090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f611c00ef30 tx=0x7f611c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f6121600700 1 -- 172.21.15.16:0/1367897929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c018070 con 0x7f612419bc10 2024-04-05T23:34:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f6121600700 1 -- 172.21.15.16:0/1367897929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f611c00f040 con 0x7f612419bc10 2024-04-05T23:34:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f6121600700 1 -- 172.21.15.16:0/1367897929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c00cd00 con 0x7f612419bc10 2024-04-05T23:34:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.704+0000 7f612ba00700 1 -- 172.21.15.16:0/1367897929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61241a3ec0 con 0x7f612419bc10 2024-04-05T23:34:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.705+0000 7f612ba00700 1 -- 172.21.15.16:0/1367897929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61241a4340 con 0x7f612419bc10 2024-04-05T23:34:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.709+0000 7f6121600700 1 -- 172.21.15.16:0/1367897929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f611c003a00 con 0x7f612419bc10 2024-04-05T23:34:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.710+0000 7f613f5e8b80 1 Processor -- start 2024-04-05T23:34:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.710+0000 7f613f5e8b80 1 -- start start 2024-04-05T23:34:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.710+0000 7f6121600700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f610c0a13b0 0x7f610c0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.710+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48c7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.710+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c12e0 0x55c7a48c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.710+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c1ca0 0x55c7a48cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.710+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a4827be0 con 0x55c7a48c12e0 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.710+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a4818990 con 0x55c7a48c7180 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.710+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a4816350 con 0x55c7a48c1ca0 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6121600700 1 -- 172.21.15.16:0/1367897929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f611c0cca80 con 0x7f612419bc10 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6129000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6129000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48c7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:49540/0 (socket says 172.21.15.16:49540) 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6129000700 1 -- 172.21.15.16:0/2488632819 learned_addr learned my addr 172.21.15.16:0/2488632819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6129a00700 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c1ca0 0x55c7a48cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6123e00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f610c0a13b0 0x7f610c0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6123e00700 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c12e0 0x55c7a48c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6129a00700 1 -- 172.21.15.16:0/2488632819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 msgr2=0x55c7a48c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6129a00700 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6129a00700 1 -- 172.21.15.16:0/2488632819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c12e0 msgr2=0x55c7a48c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6129a00700 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c12e0 0x55c7a48c1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6129a00700 1 -- 172.21.15.16:0/2488632819 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7a46933b0 con 0x55c7a48c1ca0 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.711+0000 7f6123e00700 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c12e0 0x55c7a48c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.712+0000 7f6129a00700 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c1ca0 0x55c7a48cdf10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f611c0ca020 tx=0x7f611c0994e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=739766dde36914fc server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.712+0000 7f6123e00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f610c0a13b0 0x7f610c0a3870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55c7a48b6ca0 tx=0x7f61100099b0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.712+0000 7f6116000700 1 -- 172.21.15.16:0/2488632819 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c077460 con 0x55c7a48c1ca0 2024-04-05T23:34:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.712+0000 7f6116000700 1 -- 172.21.15.16:0/2488632819 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f611c077280 con 0x55c7a48c1ca0 2024-04-05T23:34:53.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.712+0000 7f6116000700 1 -- 172.21.15.16:0/2488632819 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c00ce60 con 0x55c7a48c1ca0 2024-04-05T23:34:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 -- 172.21.15.16:0/2488632819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c1ca0 msgr2=0x55c7a48cdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c1ca0 0x55c7a48cdf10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f611c0ca020 tx=0x7f611c0994e0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 -- 172.21.15.16:0/2488632819 shutdown_connections 2024-04-05T23:34:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c1ca0 0x55c7a48cdf10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/2488632819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c12e0 0x55c7a48c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 -- 172.21.15.16:0/2488632819 >> 172.21.15.16:0/2488632819 conn(0x55c7a4835220 msgr2=0x55c7a48bc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 -- 172.21.15.16:0/2488632819 shutdown_connections 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 -- 172.21.15.16:0/2488632819 wait complete. 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 -- start start 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.713+0000 7f613f5e8b80 1 Processor -- start 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a48cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a4811c40 con 0x55c7a48c1ca0 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a47add80 con 0x55c7a48c7180 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a47ada20 con 0x55c7a48c12e0 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f613f5e8b80 10 client.0 _ll_register_callbacks cb 0x55c7a46193a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f6123e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a48cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f6129000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f6129a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f6129000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39726/0 (socket says 172.21.15.16:39726) 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.714+0000 7f6129000700 1 -- 172.21.15.16:0/1533239477 learned_addr learned my addr 172.21.15.16:0/1533239477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f6129000700 1 -- 172.21.15.16:0/1533239477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 msgr2=0x55c7a48cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f6129000700 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f6129000700 1 -- 172.21.15.16:0/1533239477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 msgr2=0x55c7a48cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f6129000700 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a48cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f6129000700 1 -- 172.21.15.16:0/1533239477 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61241a4340 con 0x55c7a48c12e0 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f6129a00700 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f6129000700 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48cb370 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6118000c00 tx=0x7f6118005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15be5be3320c5238 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f610aa00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f610aa00700 1 -- 172.21.15.16:0/1533239477 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611801d070 con 0x55c7a48c12e0 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1533239477 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c7a48b4490 con 0x55c7a48c12e0 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f610aa00700 1 -- 172.21.15.16:0/1533239477 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f6118004500 con 0x55c7a48c12e0 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f610aa00700 1 -- 172.21.15.16:0/1533239477 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f611800bee0 con 0x55c7a48c12e0 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f613f5e8b80 10 client.25867 fetch_fsmap learned FSMap version 32 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.715+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1533239477 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55c7a463fb60 con 0x55c7a48c12e0 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.716+0000 7f610aa00700 1 -- 172.21.15.16:0/1533239477 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c7a48b4490 con 0x55c7a48c12e0 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.716+0000 7f610aa00700 1 -- 172.21.15.16:0/1533239477 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 32) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f611800f7c0 con 0x55c7a48c12e0 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.716+0000 7f613f5e8b80 10 client.25867 fetch_fsmap finished waiting for FSMap version 32 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.716+0000 7f613f5e8b80 10 client.25867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.716+0000 7f613f5e8b80 20 client.25867 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.716+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c7a48db430 0x55c7a48dd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.716+0000 7f613f5e8b80 4 client.25867 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.716+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1533239477 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c7a480a700 con 0x55c7a48db430 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.716+0000 7f6123e00700 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c7a48db430 0x55c7a48dd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.717+0000 7f6123e00700 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c7a48db430 0x55c7a48dd850 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfe7545a190c27d2 server_cookie=a9d51f8d26c30127 in_seq=0 out_seq=0 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.717+0000 7f610aa00700 10 client.25867 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f610aa00700 1 -- 172.21.15.16:0/1533239477 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c7a480a700 con 0x55c7a48db430 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f610aa00700 10 client.25867 handle_command_reply: tid=0 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 client.25867 shutdown 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1533239477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 msgr2=0x55c7a48cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48cb370 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6118000c00 tx=0x7f6118005800 comp rx=0 tx=0).stop 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1533239477 shutdown_connections 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c7a48db430 0x55c7a48dd850 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c7180 0x55c7a48cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48cb370 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f610aa00700 0 client.25867 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1533239477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a48cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1533239477 >> 172.21.15.16:0/1533239477 conn(0x55c7a4835220 msgr2=0x55c7a48b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1533239477 shutdown_connections 2024-04-05T23:34:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1533239477 wait complete. 2024-04-05T23:34:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.718+0000 7f613f5e8b80 20 client.25867 trim_cache size 0 max 16384 2024-04-05T23:34:53.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.735+0000 7f613f5e8b80 1 Processor -- start 2024-04-05T23:34:53.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.735+0000 7f613f5e8b80 1 -- start start 2024-04-05T23:34:53.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48d2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c1ca0 0x55c7a48b6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c7180 0x55c7a48c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a46956a0 con 0x55c7a48c7180 2024-04-05T23:34:53.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a48bdb50 con 0x55c7a48c1ca0 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a48d2a50 con 0x55c7a48c12e0 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6129a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c7180 0x55c7a48c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6129a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c7180 0x55c7a48c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53012/0 (socket says 172.21.15.16:53012) 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6129a00700 1 -- 172.21.15.16:0/1522717720 learned_addr learned my addr 172.21.15.16:0/1522717720 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6123e00700 1 --2- 172.21.15.16:0/1522717720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c1ca0 0x55c7a48b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6129000700 1 --2- 172.21.15.16:0/1522717720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48d2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6129a00700 1 -- 172.21.15.16:0/1522717720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 msgr2=0x55c7a48d2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6129a00700 1 --2- 172.21.15.16:0/1522717720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48d2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6129a00700 1 -- 172.21.15.16:0/1522717720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c1ca0 msgr2=0x55c7a48b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6129a00700 1 --2- 172.21.15.16:0/1522717720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c1ca0 0x55c7a48b6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.736+0000 7f6129a00700 1 -- 172.21.15.16:0/1522717720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7a47d7ea0 con 0x55c7a48c7180 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.737+0000 7f6129a00700 1 --2- 172.21.15.16:0/1522717720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c7180 0x55c7a48c8d90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f611c099930 tx=0x7f611c004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4319be7e91ed177e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.737+0000 7f6116a00700 1 -- 172.21.15.16:0/1522717720 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c004ad0 con 0x55c7a48c7180 2024-04-05T23:34:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.737+0000 7f6116a00700 1 -- 172.21.15.16:0/1522717720 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f611c00ce60 con 0x55c7a48c7180 2024-04-05T23:34:53.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.737+0000 7f6116a00700 1 -- 172.21.15.16:0/1522717720 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c099090 con 0x55c7a48c7180 2024-04-05T23:34:53.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1522717720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c7180 msgr2=0x55c7a48c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1522717720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c7180 0x55c7a48c8d90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f611c099930 tx=0x7f611c004dc0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1522717720 shutdown_connections 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1522717720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c7180 0x55c7a48c8d90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1522717720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c12e0 0x55c7a48d2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/1522717720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c1ca0 0x55c7a48b6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1522717720 >> 172.21.15.16:0/1522717720 conn(0x55c7a4835220 msgr2=0x55c7a48bc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1522717720 shutdown_connections 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 -- 172.21.15.16:0/1522717720 wait complete. 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 -- start start 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.738+0000 7f613f5e8b80 1 Processor -- start 2024-04-05T23:34:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c12e0 0x55c7a49277f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a4927d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f613f5e8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c7180 0x55c7a4932680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a4928460 con 0x55c7a48c1ca0 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a4932bc0 con 0x55c7a48c12e0 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f613f5e8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7a4932d50 con 0x55c7a48c7180 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f613f5e8b80 10 client.0 _ll_register_callbacks cb 0x7f6118005350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6123e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a4927d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6123e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a4927d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53018/0 (socket says 172.21.15.16:53018) 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6123e00700 1 -- 172.21.15.16:0/4126337751 learned_addr learned my addr 172.21.15.16:0/4126337751 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6129a00700 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c7180 0x55c7a4932680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6129000700 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c12e0 0x55c7a49277f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6123e00700 1 -- 172.21.15.16:0/4126337751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c7180 msgr2=0x55c7a4932680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6123e00700 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c7180 0x55c7a4932680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6123e00700 1 -- 172.21.15.16:0/4126337751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c12e0 msgr2=0x55c7a49277f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6123e00700 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c12e0 0x55c7a49277f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.739+0000 7f6123e00700 1 -- 172.21.15.16:0/4126337751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c7a46933b0 con 0x55c7a48c1ca0 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.740+0000 7f6123e00700 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a4927d30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6124198880 tx=0x7f6110004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c7edff9a97f621 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.740+0000 7f610aa00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.740+0000 7f610aa00700 1 -- 172.21.15.16:0/4126337751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110023040 con 0x55c7a48c1ca0 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.740+0000 7f610aa00700 1 -- 172.21.15.16:0/4126337751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f611001e620 con 0x55c7a48c1ca0 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.740+0000 7f613f5e8b80 1 -- 172.21.15.16:0/4126337751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f611c077280 con 0x55c7a48c1ca0 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.740+0000 7f610aa00700 1 -- 172.21.15.16:0/4126337751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f611001e7e0 con 0x55c7a48c1ca0 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.740+0000 7f613f5e8b80 10 client.15801 fetch_fsmap learned FSMap version 32 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.740+0000 7f613f5e8b80 1 -- 172.21.15.16:0/4126337751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55c7a4697cd0 con 0x55c7a48c1ca0 2024-04-05T23:34:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.741+0000 7f610aa00700 1 -- 172.21.15.16:0/4126337751 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f611003e020 con 0x55c7a48c1ca0 2024-04-05T23:34:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.741+0000 7f610aa00700 1 -- 172.21.15.16:0/4126337751 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f611001d8f0 con 0x55c7a48c1ca0 2024-04-05T23:34:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.741+0000 7f613f5e8b80 10 client.15801 fetch_fsmap finished waiting for FSMap version 32 2024-04-05T23:34:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.741+0000 7f613f5e8b80 10 client.15801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.741+0000 7f613f5e8b80 20 client.15801 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.741+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c7a4934560 0x55c7a4936980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.741+0000 7f613f5e8b80 4 client.15801 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-05T23:34:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.741+0000 7f613f5e8b80 1 -- 172.21.15.16:0/4126337751 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c7a4660cf0 con 0x55c7a4934560 2024-04-05T23:34:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.741+0000 7f6129000700 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c7a4934560 0x55c7a4936980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:53.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f6129000700 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c7a4934560 0x55c7a4936980 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e21dced49f9c03 server_cookie=4106de76cfd43b0d in_seq=0 out_seq=0 2024-04-05T23:34:53.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f610aa00700 10 client.15801 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:53.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f610aa00700 1 -- 172.21.15.16:0/4126337751 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c7a4660cf0 con 0x55c7a4934560 2024-04-05T23:34:53.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f610aa00700 10 client.15801 handle_command_reply: tid=0 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 client.15801 shutdown 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 -- 172.21.15.16:0/4126337751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 msgr2=0x55c7a4927d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a4927d30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6124198880 tx=0x7f6110004750 comp rx=0 tx=0).stop 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 -- 172.21.15.16:0/4126337751 shutdown_connections 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c7a4934560 0x55c7a4936980 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7a48c7180 0x55c7a4932680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c7a48c12e0 0x55c7a49277f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f610aa00700 0 client.15801 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 --2- 172.21.15.16:0/4126337751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c7a48c1ca0 0x55c7a4927d30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 -- 172.21.15.16:0/4126337751 >> 172.21.15.16:0/4126337751 conn(0x55c7a4835220 msgr2=0x55c7a48cca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 -- 172.21.15.16:0/4126337751 shutdown_connections 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 1 -- 172.21.15.16:0/4126337751 wait complete. 2024-04-05T23:34:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.742+0000 7f613f5e8b80 20 client.15801 trim_cache size 0 max 16384 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "9987f0ed-9576-49e7-8f88-9336b0067c59": { 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "9987f0ed-9576-49e7-8f88-9336b0067c59", 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:34:53.740 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:34:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 -- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f610c0a13b0 msgr2=0x7f610c0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f610c0a13b0 0x7f610c0a3870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55c7a48b6ca0 tx=0x7f61100099b0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 -- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f612419bc10 msgr2=0x7f612419c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f612419bc10 0x7f612419c090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f611c00ef30 tx=0x7f611c00c3a0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 -- 172.21.15.16:0/1367897929 shutdown_connections 2024-04-05T23:34:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f610c0a13b0 0x7f610c0a3870 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f612419bc10 0x7f612419c090 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61241010c0 0x7f6124197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 --2- 172.21.15.16:0/1367897929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61241061e0 0x7f6124197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:53.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 -- 172.21.15.16:0/1367897929 >> 172.21.15.16:0/1367897929 conn(0x7f61240fc920 msgr2=0x7f61240fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:53.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 -- 172.21.15.16:0/1367897929 shutdown_connections 2024-04-05T23:34:53.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:34:53.743+0000 7f612ba00700 1 -- 172.21.15.16:0/1367897929 wait complete. 2024-04-05T23:34:53.754 INFO:tasks.cephfs.filesystem:scrub status for tag:9987f0ed-9576-49e7-8f88-9336b0067c59 - {'path': '/', 'tag': '9987f0ed-9576-49e7-8f88-9336b0067c59', 'options': 'recursive,force'} 2024-04-05T23:34:53.771 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: 2024-04-05T23:34:53.488+0000 7fa23c2fe700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000222 [3,head] /volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25303=0-4194304@2} caps={25303=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25303 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55e688a5d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000222:[<0x10000000221/BENCHS.LWP v2>,<0x10000000009/WORDPRO v524>,<0x10000000008/~dmtmp v32>,<0x10000000007/client0 v16>,<0x10000000006/clients v6>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v37>,<0x1/volumes v35>]//[]","memoryvalue":"(3)0x10000000222:[<0x10000000221/BENCHS.LWP v2>,<0x10000000009/WORDPRO v3990>,<0x10000000008/~dmtmp v54>,<0x10000000007/client0 v40>,<0x10000000006/clients v32>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v30>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v48>,<0x10000000001/sv_1 v60>,<0x10000000000/qa v27>,<0x1/volumes v24>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-05T23:34:53.771 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: 2024-04-05T23:34:53.488+0000 7fa23c2fe700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000223 [3,head] /volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25303=0-4194304@2} caps={25303=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25303 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55e688a5c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000223:[<0x10000000221/RESULTS.XLS v4>,<0x10000000009/WORDPRO v524>,<0x10000000008/~dmtmp v32>,<0x10000000007/client0 v16>,<0x10000000006/clients v6>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v37>,<0x1/volumes v35>]//[]","memoryvalue":"(3)0x10000000223:[<0x10000000221/RESULTS.XLS v4>,<0x10000000009/WORDPRO v3990>,<0x10000000008/~dmtmp v54>,<0x10000000007/client0 v40>,<0x10000000006/clients v32>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v30>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v48>,<0x10000000001/sv_1 v60>,<0x10000000000/qa v27>,<0x1/volumes v24>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-05T23:34:53.771 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: 2024-04-05T23:34:53.489+0000 7fa23c2fe700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000224 [3,head] /volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25303=0-4194304@2} caps={25303=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25303 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55e688aaa580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000224:[<0x10000000221/NEWS1_1.LWP v6>,<0x10000000009/WORDPRO v524>,<0x10000000008/~dmtmp v32>,<0x10000000007/client0 v16>,<0x10000000006/clients v6>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v37>,<0x1/volumes v35>]//[]","memoryvalue":"(3)0x10000000224:[<0x10000000221/NEWS1_1.LWP v6>,<0x10000000009/WORDPRO v3990>,<0x10000000008/~dmtmp v54>,<0x10000000007/client0 v40>,<0x10000000006/clients v32>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v30>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v48>,<0x10000000001/sv_1 v60>,<0x10000000000/qa v27>,<0x1/volumes v24>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-05T23:34:53.771 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: 2024-04-05T23:34:53.490+0000 7fa23c2fe700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000226 [3,head] /volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25303=0-4194304@2} caps={25303=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25303 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55e68886b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000226:[<0x10000000225/T1.XLS v2>,<0x10000000009/PM v684>,<0x10000000008/~dmtmp v32>,<0x10000000007/client0 v16>,<0x10000000006/clients v6>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v37>,<0x1/volumes v35>]//[]","memoryvalue":"(3)0x10000000226:[<0x10000000225/T1.XLS v2>,<0x10000000009/PM v3569>,<0x10000000008/~dmtmp v54>,<0x10000000007/client0 v40>,<0x10000000006/clients v32>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v30>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v48>,<0x10000000001/sv_1 v60>,<0x10000000000/qa v27>,<0x1/volumes v24>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-05T23:34:53.771 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: 2024-04-05T23:34:53.490+0000 7fa23c2fe700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000227 [3,head] /volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25303=0-4194304@2} caps={25303=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25303 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55e68886b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000227:[<0x10000000225/EVENTS.DOC v4>,<0x10000000009/PM v684>,<0x10000000008/~dmtmp v32>,<0x10000000007/client0 v16>,<0x10000000006/clients v6>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v37>,<0x1/volumes v35>]//[]","memoryvalue":"(3)0x10000000227:[<0x10000000225/EVENTS.DOC v4>,<0x10000000009/PM v3569>,<0x10000000008/~dmtmp v54>,<0x10000000007/client0 v40>,<0x10000000006/clients v32>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v30>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v48>,<0x10000000001/sv_1 v60>,<0x10000000000/qa v27>,<0x1/volumes v24>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-05T23:34:53.771 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: 2024-04-05T23:34:53.491+0000 7fa23c2fe700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000228 [3,head] /volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25303=0-4194304@2} caps={25303=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25303 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55e68886a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x10000000228:[<0x10000000225/MOVED.DOC v6>,<0x10000000009/PM v684>,<0x10000000008/~dmtmp v32>,<0x10000000007/client0 v16>,<0x10000000006/clients v6>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v37>,<0x1/volumes v35>]//[]","memoryvalue":"(3)0x10000000228:[<0x10000000225/MOVED.DOC v6>,<0x10000000009/PM v3569>,<0x10000000008/~dmtmp v54>,<0x10000000007/client0 v40>,<0x10000000006/clients v32>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v30>,<0x100000001fa/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa v48>,<0x10000000001/sv_1 v60>,<0x10000000000/qa v27>,<0x1/volumes v24>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-05T23:34:53.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:53 smithi016 ceph-mon[38248]: pgmap v43: 97 pgs: 97 active+clean; 11 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 1.1 MiB/s wr, 179 op/s 2024-04-05T23:34:53.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:53 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:53.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:53 smithi016 ceph-mon[38248]: scrub queued for path: / 2024-04-05T23:34:53.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:53 smithi016 ceph-mon[38248]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:53.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:53 smithi016 ceph-mon[38248]: scrub summary: active paths [/] 2024-04-05T23:34:53.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:53 smithi016 ceph-mon[38248]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:53.980 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21444 10.34 MB/sec warmup 13 sec latency 2564.201 ms 2024-04-05T23:34:54.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-mon[79417]: pgmap v43: 97 pgs: 97 active+clean; 11 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 1.1 MiB/s wr, 179 op/s 2024-04-05T23:34:54.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:54.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-mon[79417]: scrub queued for path: / 2024-04-05T23:34:54.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-mon[79417]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:54.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-mon[79417]: scrub summary: active paths [/] 2024-04-05T23:34:54.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:53 smithi130 ceph-mon[79417]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:54.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:53 smithi027 ceph-mon[78597]: pgmap v43: 97 pgs: 97 active+clean; 11 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 1.1 MiB/s wr, 179 op/s 2024-04-05T23:34:54.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:53 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:54.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:53 smithi027 ceph-mon[78597]: scrub queued for path: / 2024-04-05T23:34:54.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:53 smithi027 ceph-mon[78597]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:54.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:53 smithi027 ceph-mon[78597]: scrub summary: active paths [/] 2024-04-05T23:34:54.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:53 smithi027 ceph-mon[78597]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:54.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21444 9.60 MB/sec warmup 14 sec latency 1441.433 ms 2024-04-05T23:34:55.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: scrub summary: active paths [/] 2024-04-05T23:34:55.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: scrub summary: active paths [/] 2024-04-05T23:34:55.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: Scrub error on inode 0x10000000222 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: Scrub error on inode 0x10000000223 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: Scrub error on inode 0x10000000224 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:54 smithi016 ceph-mon[38248]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:55.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: scrub summary: active paths [/] 2024-04-05T23:34:55.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: scrub summary: active paths [/] 2024-04-05T23:34:55.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: Scrub error on inode 0x10000000222 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: Scrub error on inode 0x10000000223 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: Scrub error on inode 0x10000000224 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:55.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:54 smithi130 ceph-mon[79417]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:55.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: scrub summary: active paths [/] 2024-04-05T23:34:55.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: scrub summary: active paths [/] 2024-04-05T23:34:55.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: Scrub error on inode 0x10000000222 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: Scrub error on inode 0x10000000223 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: Scrub error on inode 0x10000000224 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:55.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.c log and `damage ls` output for details 2024-04-05T23:34:55.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:54 smithi027 ceph-mon[78597]: scrub summary: idle+waiting paths [/] 2024-04-05T23:34:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.519+0000 7f6204000700 1 -- 172.21.15.16:0/1734866789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61fc101040 msgr2=0x7f61fc101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.519+0000 7f6204000700 1 --2- 172.21.15.16:0/1734866789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61fc101040 0x7f61fc101420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009990 tx=0x7f61f0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.519+0000 7f6204000700 1 -- 172.21.15.16:0/1734866789 shutdown_connections 2024-04-05T23:34:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.519+0000 7f6204000700 1 --2- 172.21.15.16:0/1734866789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc106160 0x7f61fc10dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.519+0000 7f6204000700 1 --2- 172.21.15.16:0/1734866789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61fc101040 0x7f61fc101420 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.519+0000 7f6204000700 1 --2- 172.21.15.16:0/1734866789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61fc1019f0 0x7f61fc105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.519+0000 7f6204000700 1 -- 172.21.15.16:0/1734866789 >> 172.21.15.16:0/1734866789 conn(0x7f61fc0fc8e0 msgr2=0x7f61fc0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.519+0000 7f6204000700 1 -- 172.21.15.16:0/1734866789 shutdown_connections 2024-04-05T23:34:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.519+0000 7f6204000700 1 -- 172.21.15.16:0/1734866789 wait complete. 2024-04-05T23:34:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.520+0000 7f6204000700 1 Processor -- start 2024-04-05T23:34:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.520+0000 7f6204000700 1 -- start start 2024-04-05T23:34:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6204000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61fc101040 0x7f61fc0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6204000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc106160 0x7f61fc073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6204000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61fc073c60 0x7f61fc1a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6204000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc1a5a20 con 0x7f61fc106160 2024-04-05T23:34:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6204000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc1a5b60 con 0x7f61fc073c60 2024-04-05T23:34:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6204000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc1a5d20 con 0x7f61fc101040 2024-04-05T23:34:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6200c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc106160 0x7f61fc073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6201600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61fc101040 0x7f61fc0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6201600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61fc101040 0x7f61fc0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39748/0 (socket says 172.21.15.16:39748) 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6200c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc106160 0x7f61fc073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53034/0 (socket says 172.21.15.16:53034) 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6201600700 1 -- 172.21.15.16:0/1720116709 learned_addr learned my addr 172.21.15.16:0/1720116709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.521+0000 7f6202000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61fc073c60 0x7f61fc1a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f6201600700 1 -- 172.21.15.16:0/1720116709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61fc073c60 msgr2=0x7f61fc1a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f6201600700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61fc073c60 0x7f61fc1a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f6201600700 1 -- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc106160 msgr2=0x7f61fc073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f6201600700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc106160 0x7f61fc073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f6201600700 1 -- 172.21.15.16:0/1720116709 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f00095f0 con 0x7f61fc101040 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f6200c00700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc106160 0x7f61fc073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f6201600700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61fc101040 0x7f61fc0731e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f61f0000c00 tx=0x7f61f0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f61fa000700 1 -- 172.21.15.16:0/1720116709 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f001d070 con 0x7f61fc101040 2024-04-05T23:34:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f61fa000700 1 -- 172.21.15.16:0/1720116709 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61f0021410 con 0x7f61fc101040 2024-04-05T23:34:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f6204000700 1 -- 172.21.15.16:0/1720116709 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc1a5fa0 con 0x7f61fc101040 2024-04-05T23:34:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.522+0000 7f61fa000700 1 -- 172.21.15.16:0/1720116709 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0017650 con 0x7f61fc101040 2024-04-05T23:34:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.523+0000 7f6204000700 1 -- 172.21.15.16:0/1720116709 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc1a63c0 con 0x7f61fc101040 2024-04-05T23:34:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.526+0000 7f61fa000700 1 -- 172.21.15.16:0/1720116709 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 136106+0+0 (secure 0 0 0) 0x7f61f0021580 con 0x7f61fc101040 2024-04-05T23:34:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.527+0000 7f6217afdb80 1 Processor -- start 2024-04-05T23:34:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.527+0000 7f61fa000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e40a13b0 0x7f61e40a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.527+0000 7f6217afdb80 1 -- start start 2024-04-05T23:34:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.527+0000 7f61fa000700 1 -- 172.21.15.16:0/1720116709 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f61f00ce0d0 con 0x7f61fc101040 2024-04-05T23:34:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.527+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 0x561322159f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.527+0000 7f6202000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e40a13b0 0x7f61e40a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.527+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x561322154030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.527+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613221545c0 0x561322154a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6217afdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613220bab80 con 0x56132215a4a0 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6217afdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613220ab930 con 0x5613221545c0 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6217afdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613220a92f0 con 0x561322159b80 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6200c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x561322154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6200c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x561322154030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53036/0 (socket says 172.21.15.16:53036) 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6200c00700 1 -- 172.21.15.16:0/475610601 learned_addr learned my addr 172.21.15.16:0/475610601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6201600700 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 0x561322159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6202000700 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613221545c0 0x561322154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6200c00700 1 -- 172.21.15.16:0/475610601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 msgr2=0x561322159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6200c00700 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 0x561322159f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6200c00700 1 -- 172.21.15.16:0/475610601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613221545c0 msgr2=0x561322154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6200c00700 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613221545c0 0x561322154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6200c00700 1 -- 172.21.15.16:0/475610601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561321f26350 con 0x56132215a4a0 2024-04-05T23:34:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6202000700 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613221545c0 0x561322154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:34:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.528+0000 7f6202000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e40a13b0 0x7f61e40a3870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009ec0 tx=0x7f61f400fc40 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.529+0000 7f6200c00700 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x561322154030 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f61e8011540 tx=0x7f61e8011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970aa4d91b711833 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.529+0000 7f61ee600700 1 -- 172.21.15.16:0/475610601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e8004a30 con 0x56132215a4a0 2024-04-05T23:34:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.529+0000 7f61ee600700 1 -- 172.21.15.16:0/475610601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61e80054e0 con 0x56132215a4a0 2024-04-05T23:34:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.529+0000 7f61ee600700 1 -- 172.21.15.16:0/475610601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e800e7f0 con 0x56132215a4a0 2024-04-05T23:34:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.529+0000 7f6217afdb80 1 -- 172.21.15.16:0/475610601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 msgr2=0x561322154030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.529+0000 7f6217afdb80 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x561322154030 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f61e8011540 tx=0x7f61e8011850 comp rx=0 tx=0).stop 2024-04-05T23:34:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.530+0000 7f6217afdb80 1 -- 172.21.15.16:0/475610601 shutdown_connections 2024-04-05T23:34:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.530+0000 7f6217afdb80 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613221545c0 0x561322154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.530+0000 7f6217afdb80 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 0x561322159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.530+0000 7f6217afdb80 1 --2- 172.21.15.16:0/475610601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x561322154030 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.530+0000 7f6217afdb80 1 -- 172.21.15.16:0/475610601 >> 172.21.15.16:0/475610601 conn(0x5613220c81c0 msgr2=0x56132214ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.530+0000 7f6217afdb80 1 -- 172.21.15.16:0/475610601 shutdown_connections 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.530+0000 7f6217afdb80 1 -- 172.21.15.16:0/475610601 wait complete. 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.530+0000 7f6217afdb80 1 -- start start 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.530+0000 7f6217afdb80 1 Processor -- start 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322160410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 0x56132215d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6217afdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613220a4be0 con 0x56132215a4a0 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6217afdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561322040d20 con 0x561322159b80 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6217afdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613220409c0 con 0x5613221545c0 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6217afdb80 10 client.0 _ll_register_callbacks cb 0x561321eac340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6201600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6201600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322160410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39778/0 (socket says 172.21.15.16:39778) 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6201600700 1 -- 172.21.15.16:0/3403996098 learned_addr learned my addr 172.21.15.16:0/3403996098 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.531+0000 7f6202000700 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6200c00700 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 0x56132215d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6201600700 1 -- 172.21.15.16:0/3403996098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 msgr2=0x56132215d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6201600700 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 0x56132215d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6201600700 1 -- 172.21.15.16:0/3403996098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 msgr2=0x56132215dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6201600700 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6201600700 1 -- 172.21.15.16:0/3403996098 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e8009d10 con 0x5613221545c0 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6201600700 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322160410 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f61f0097a60 tx=0x7f61f0021b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6fc7e16f984f020 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f61e2a00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f61e2a00700 1 -- 172.21.15.16:0/3403996098 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f009a6e0 con 0x5613221545c0 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f61e2a00700 1 -- 172.21.15.16:0/3403996098 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f61f009ee20 con 0x5613221545c0 2024-04-05T23:34:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6217afdb80 1 -- 172.21.15.16:0/3403996098 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5613221471e0 con 0x5613221545c0 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f61e2a00700 1 -- 172.21.15.16:0/3403996098 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61f0060ea0 con 0x5613221545c0 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6217afdb80 10 client.25885 fetch_fsmap learned FSMap version 32 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.532+0000 7f6217afdb80 1 -- 172.21.15.16:0/3403996098 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561321ed2b00 con 0x5613221545c0 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.533+0000 7f61e2a00700 1 -- 172.21.15.16:0/3403996098 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61f0064080 con 0x5613221545c0 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.533+0000 7f61e2a00700 1 -- 172.21.15.16:0/3403996098 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 32) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f61f00cb950 con 0x5613221545c0 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.533+0000 7f6217afdb80 10 client.25885 fetch_fsmap finished waiting for FSMap version 32 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.533+0000 7f6217afdb80 10 client.25885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.533+0000 7f6217afdb80 20 client.25885 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.533+0000 7f6217afdb80 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56132216db60 0x56132216ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.533+0000 7f6217afdb80 4 client.25885 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.533+0000 7f6217afdb80 1 -- 172.21.15.16:0/3403996098 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56132209d6a0 con 0x56132216db60 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.533+0000 7f6200c00700 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56132216db60 0x56132216ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.534+0000 7f6200c00700 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56132216db60 0x56132216ff80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a8a1dbb792439a2 server_cookie=6591809dd0e17674 in_seq=0 out_seq=0 2024-04-05T23:34:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.534+0000 7f61e2a00700 10 client.25885 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.534+0000 7f61e2a00700 1 -- 172.21.15.16:0/3403996098 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56132209d6a0 con 0x56132216db60 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.534+0000 7f61e2a00700 10 client.25885 handle_command_reply: tid=0 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 client.25885 shutdown 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 -- 172.21.15.16:0/3403996098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 msgr2=0x561322160410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322160410 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f61f0097a60 tx=0x7f61f0021b60 comp rx=0 tx=0).stop 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 -- 172.21.15.16:0/3403996098 shutdown_connections 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56132216db60 0x56132216ff80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322160410 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f61e2a00700 0 client.25885 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 --2- 172.21.15.16:0/3403996098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 0x56132215d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 -- 172.21.15.16:0/3403996098 >> 172.21.15.16:0/3403996098 conn(0x5613220c81c0 msgr2=0x56132214e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 -- 172.21.15.16:0/3403996098 shutdown_connections 2024-04-05T23:34:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 1 -- 172.21.15.16:0/3403996098 wait complete. 2024-04-05T23:34:55.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.535+0000 7f6217afdb80 20 client.25885 trim_cache size 0 max 16384 2024-04-05T23:34:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.550+0000 7f6217afdb80 1 Processor -- start 2024-04-05T23:34:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6217afdb80 1 -- start start 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322164f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 0x56132214ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6217afdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561321f28640 con 0x56132215a4a0 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6217afdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613221583e0 con 0x561322159b80 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6217afdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561322150570 con 0x5613221545c0 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6202000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6202000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53054/0 (socket says 172.21.15.16:53054) 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6202000700 1 -- 172.21.15.16:0/767735314 learned_addr learned my addr 172.21.15.16:0/767735314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6200c00700 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 0x56132214ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.551+0000 7f6201600700 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322164f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f6201600700 1 -- 172.21.15.16:0/767735314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 msgr2=0x56132214ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f6201600700 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 0x56132214ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f6201600700 1 -- 172.21.15.16:0/767735314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 msgr2=0x56132215ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f6201600700 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f6201600700 1 -- 172.21.15.16:0/767735314 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56132206ae40 con 0x5613221545c0 2024-04-05T23:34:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f6202000700 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f6200c00700 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 0x56132214ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:34:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f6201600700 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322164f80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f61f00974f0 tx=0x7f61f00995a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5d7afe88f602e22 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f61ef000700 1 -- 172.21.15.16:0/767735314 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0060ea0 con 0x5613221545c0 2024-04-05T23:34:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f61ef000700 1 -- 172.21.15.16:0/767735314 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5613221471e0 con 0x5613221545c0 2024-04-05T23:34:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.552+0000 7f61ef000700 1 -- 172.21.15.16:0/767735314 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f00056a0 con 0x5613221545c0 2024-04-05T23:34:55.549 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 -- 172.21.15.16:0/767735314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 msgr2=0x561322164f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:55.549 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322164f80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f61f00974f0 tx=0x7f61f00995a0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 -- 172.21.15.16:0/767735314 shutdown_connections 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56132215a4a0 0x56132215ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5613221545c0 0x561322164f80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 --2- 172.21.15.16:0/767735314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561322159b80 0x56132214ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 -- 172.21.15.16:0/767735314 >> 172.21.15.16:0/767735314 conn(0x5613220c81c0 msgr2=0x5613221549a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 -- 172.21.15.16:0/767735314 shutdown_connections 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 -- 172.21.15.16:0/767735314 wait complete. 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.553+0000 7f6217afdb80 1 -- start start 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.554+0000 7f6217afdb80 1 Processor -- start 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.554+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5613221545c0 0x5613221556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.554+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 0x561322155c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.554+0000 7f6217afdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56132215a4a0 0x561322156160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.554+0000 7f6217afdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561322156740 con 0x5613221545c0 2024-04-05T23:34:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.554+0000 7f6217afdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613221568b0 con 0x56132215a4a0 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.554+0000 7f6217afdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561322156a70 con 0x561322159b80 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.554+0000 7f6217afdb80 10 client.0 _ll_register_callbacks cb 0x7f61f009abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6201600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5613221545c0 0x5613221556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6201600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5613221545c0 0x5613221556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53062/0 (socket says 172.21.15.16:53062) 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6201600700 1 -- 172.21.15.16:0/2160879124 learned_addr learned my addr 172.21.15.16:0/2160879124 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6200c00700 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 0x561322155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6202000700 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56132215a4a0 0x561322156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6201600700 1 -- 172.21.15.16:0/2160879124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 msgr2=0x561322155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6201600700 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 0x561322155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6201600700 1 -- 172.21.15.16:0/2160879124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56132215a4a0 msgr2=0x561322156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6201600700 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56132215a4a0 0x561322156160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6201600700 1 -- 172.21.15.16:0/2160879124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561321f26350 con 0x5613221545c0 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6201600700 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5613221545c0 0x5613221556e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f61f009af80 tx=0x7f61f0021b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4134774ab236e484 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f61e2a00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:34:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f61e2a00700 1 -- 172.21.15.16:0/2160879124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f007c070 con 0x5613221545c0 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f61e2a00700 1 -- 172.21.15.16:0/2160879124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f61f00a33e0 con 0x5613221545c0 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.555+0000 7f6217afdb80 1 -- 172.21.15.16:0/2160879124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61e80054e0 con 0x5613221545c0 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f61e2a00700 1 -- 172.21.15.16:0/2160879124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61f0060ea0 con 0x5613221545c0 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f6217afdb80 10 client.15819 fetch_fsmap learned FSMap version 32 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f6217afdb80 1 -- 172.21.15.16:0/2160879124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7f61f400fe70 con 0x5613221545c0 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f61e2a00700 1 -- 172.21.15.16:0/2160879124 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61f00a7080 con 0x5613221545c0 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f61e2a00700 1 -- 172.21.15.16:0/2160879124 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 32) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f61f007f020 con 0x5613221545c0 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f6217afdb80 10 client.15819 fetch_fsmap finished waiting for FSMap version 32 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f6217afdb80 10 client.15819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f6217afdb80 20 client.15819 populate_metadata read hostname 'smithi016' 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f6217afdb80 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5613221c21b0 0x5613221c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f6217afdb80 4 client.15819 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f6217afdb80 1 -- 172.21.15.16:0/2160879124 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561321ef3c90 con 0x5613221c21b0 2024-04-05T23:34:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.556+0000 7f6200c00700 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5613221c21b0 0x5613221c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:34:55.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.557+0000 7f6200c00700 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5613221c21b0 0x5613221c45a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36acf5c04fc14687 server_cookie=d9e0673f4fb6c6d5 in_seq=0 out_seq=0 2024-04-05T23:34:55.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.557+0000 7f61e2a00700 10 client.15819 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.630+0000 7f61e2a00700 1 -- 172.21.15.16:0/2160879124 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561321ef3c90 con 0x5613221c21b0 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.630+0000 7f61e2a00700 10 client.15819 handle_command_reply: tid=0 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.630+0000 7f6217afdb80 1 client.15819 shutdown 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 -- 172.21.15.16:0/2160879124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5613221545c0 msgr2=0x5613221556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5613221545c0 0x5613221556e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f61f009af80 tx=0x7f61f0021b60 comp rx=0 tx=0).stop 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 -- 172.21.15.16:0/2160879124 shutdown_connections 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5613221c21b0 0x5613221c45a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5613221545c0 0x5613221556e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56132215a4a0 0x561322156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f61e2a00700 0 client.15819 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 --2- 172.21.15.16:0/2160879124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561322159b80 0x561322155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 -- 172.21.15.16:0/2160879124 >> 172.21.15.16:0/2160879124 conn(0x5613220c81c0 msgr2=0x5613221696e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 -- 172.21.15.16:0/2160879124 shutdown_connections 2024-04-05T23:34:55.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 1 -- 172.21.15.16:0/2160879124 wait complete. 2024-04-05T23:34:55.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.631+0000 7f6217afdb80 20 client.15819 trim_cache size 0 max 16384 2024-04-05T23:34:55.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:34:55.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:34:55.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:34:55.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:34:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 -- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e40a13b0 msgr2=0x7f61e40a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e40a13b0 0x7f61e40a3870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009ec0 tx=0x7f61f400fc40 comp rx=0 tx=0).stop 2024-04-05T23:34:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 -- 172.21.15.16:0/1720116709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61fc101040 msgr2=0x7f61fc0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:34:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61fc101040 0x7f61fc0731e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f61f0000c00 tx=0x7f61f0004730 comp rx=0 tx=0).stop 2024-04-05T23:34:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 -- 172.21.15.16:0/1720116709 shutdown_connections 2024-04-05T23:34:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e40a13b0 0x7f61e40a3870 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61fc073c60 0x7f61fc1a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61fc101040 0x7f61fc0731e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 --2- 172.21.15.16:0/1720116709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc106160 0x7f61fc073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:34:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 -- 172.21.15.16:0/1720116709 >> 172.21.15.16:0/1720116709 conn(0x7f61fc0fc8e0 msgr2=0x7f61fc0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:34:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 -- 172.21.15.16:0/1720116709 shutdown_connections 2024-04-05T23:34:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:34:55.632+0000 7f6204000700 1 -- 172.21.15.16:0/1720116709 wait complete. 2024-04-05T23:34:55.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21466 8.96 MB/sec warmup 15 sec latency 2340.104 ms 2024-04-05T23:34:56.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:55 smithi016 ceph-mon[38248]: pgmap v44: 97 pgs: 97 active+clean; 11 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-04-05T23:34:56.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:55 smithi130 ceph-mon[79417]: pgmap v44: 97 pgs: 97 active+clean; 11 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-04-05T23:34:56.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:55 smithi027 ceph-mon[78597]: pgmap v44: 97 pgs: 97 active+clean; 11 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-04-05T23:34:56.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21466 8.40 MB/sec warmup 16 sec latency 1087.824 ms 2024-04-05T23:34:57.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21466 7.91 MB/sec warmup 17 sec latency 2087.971 ms 2024-04-05T23:34:58.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:57 smithi016 ceph-mon[38248]: pgmap v45: 97 pgs: 97 active+clean; 7.5 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 244 op/s 2024-04-05T23:34:58.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:57 smithi130 ceph-mon[79417]: pgmap v45: 97 pgs: 97 active+clean; 7.5 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 244 op/s 2024-04-05T23:34:58.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:57 smithi027 ceph-mon[78597]: pgmap v45: 97 pgs: 97 active+clean; 7.5 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 244 op/s 2024-04-05T23:34:58.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21466 7.47 MB/sec warmup 18 sec latency 3088.069 ms 2024-04-05T23:34:59.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:58 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:59.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:58 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:59.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:58 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:34:59.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21466 7.07 MB/sec warmup 19 sec latency 4088.202 ms 2024-04-05T23:35:00.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:34:59 smithi016 ceph-mon[38248]: pgmap v46: 97 pgs: 97 active+clean; 7.5 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 424 KiB/s wr, 122 op/s 2024-04-05T23:35:00.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:34:59 smithi130 ceph-mon[79417]: pgmap v46: 97 pgs: 97 active+clean; 7.5 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 424 KiB/s wr, 122 op/s 2024-04-05T23:35:00.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:34:59 smithi027 ceph-mon[78597]: pgmap v46: 97 pgs: 97 active+clean; 7.5 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 424 KiB/s wr, 122 op/s 2024-04-05T23:35:00.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21506 6.74 MB/sec warmup 20 sec latency 4663.797 ms 2024-04-05T23:35:01.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21991 6.47 MB/sec warmup 21 sec latency 269.171 ms 2024-04-05T23:35:02.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:01 smithi016 ceph-mon[38248]: pgmap v47: 97 pgs: 97 active+clean; 7.5 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 435 KiB/s wr, 135 op/s 2024-04-05T23:35:02.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:01 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a[38225]: 2024-04-05T23:35:01.806+0000 7f0cb0200700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-05T23:35:02.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:01 smithi130 ceph-mon[79417]: pgmap v47: 97 pgs: 97 active+clean; 7.5 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 435 KiB/s wr, 135 op/s 2024-04-05T23:35:02.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:01 smithi027 ceph-mon[78597]: pgmap v47: 97 pgs: 97 active+clean; 7.5 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 435 KiB/s wr, 135 op/s 2024-04-05T23:35:02.981 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21991 6.17 MB/sec warmup 22 sec latency 1269.251 ms 2024-04-05T23:35:03.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:02 smithi016 ceph-mon[38248]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-05T23:35:03.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:02 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] up:active 2024-04-05T23:35:03.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:02 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:03.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:02 smithi016 ceph-mon[38248]: scrub summary: idle 2024-04-05T23:35:03.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:02 smithi130 ceph-mon[79417]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-05T23:35:03.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:02 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] up:active 2024-04-05T23:35:03.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:02 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:03.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:02 smithi130 ceph-mon[79417]: scrub summary: idle 2024-04-05T23:35:03.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:02 smithi027 ceph-mon[78597]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-05T23:35:03.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:02 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] up:active 2024-04-05T23:35:03.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:02 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:03.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:02 smithi027 ceph-mon[78597]: scrub summary: idle 2024-04-05T23:35:03.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21991 5.91 MB/sec warmup 23 sec latency 2269.361 ms 2024-04-05T23:35:04.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:03 smithi016 ceph-mon[38248]: scrub summary: idle 2024-04-05T23:35:04.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:03 smithi016 ceph-mon[38248]: scrub summary: idle 2024-04-05T23:35:04.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:03 smithi016 ceph-mon[38248]: pgmap v48: 97 pgs: 97 active+clean; 7.6 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 424 KiB/s wr, 140 op/s 2024-04-05T23:35:04.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:03 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:04.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:03 smithi130 ceph-mon[79417]: scrub summary: idle 2024-04-05T23:35:04.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:03 smithi130 ceph-mon[79417]: scrub summary: idle 2024-04-05T23:35:04.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:03 smithi130 ceph-mon[79417]: pgmap v48: 97 pgs: 97 active+clean; 7.6 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 424 KiB/s wr, 140 op/s 2024-04-05T23:35:04.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:03 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:04.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:03 smithi027 ceph-mon[78597]: scrub summary: idle 2024-04-05T23:35:04.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:03 smithi027 ceph-mon[78597]: scrub summary: idle 2024-04-05T23:35:04.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:03 smithi027 ceph-mon[78597]: pgmap v48: 97 pgs: 97 active+clean; 7.6 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 424 KiB/s wr, 140 op/s 2024-04-05T23:35:04.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:03 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:04.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22046 5.67 MB/sec warmup 24 sec latency 3202.260 ms 2024-04-05T23:35:05.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22114 5.45 MB/sec warmup 25 sec latency 944.242 ms 2024-04-05T23:35:06.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:05 smithi016 ceph-mon[38248]: pgmap v49: 97 pgs: 97 active+clean; 7.6 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 67 KiB/s wr, 89 op/s 2024-04-05T23:35:06.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:05 smithi130 ceph-mon[79417]: pgmap v49: 97 pgs: 97 active+clean; 7.6 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 67 KiB/s wr, 89 op/s 2024-04-05T23:35:06.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:05 smithi027 ceph-mon[78597]: pgmap v49: 97 pgs: 97 active+clean; 7.6 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 67 KiB/s wr, 89 op/s 2024-04-05T23:35:06.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22114 5.24 MB/sec warmup 26 sec latency 1047.312 ms 2024-04-05T23:35:07.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22114 5.04 MB/sec warmup 27 sec latency 2047.442 ms 2024-04-05T23:35:08.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:07 smithi027 ceph-mon[78597]: pgmap v50: 97 pgs: 97 active+clean; 9.2 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 206 KiB/s wr, 124 op/s 2024-04-05T23:35:08.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:07 smithi027 ceph-mon[78597]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-05T23:35:08.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:07 smithi016 ceph-mon[38248]: pgmap v50: 97 pgs: 97 active+clean; 9.2 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 206 KiB/s wr, 124 op/s 2024-04-05T23:35:08.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:07 smithi016 ceph-mon[38248]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-05T23:35:08.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:07 smithi130 ceph-mon[79417]: pgmap v50: 97 pgs: 97 active+clean; 9.2 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 206 KiB/s wr, 124 op/s 2024-04-05T23:35:08.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:07 smithi130 ceph-mon[79417]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-05T23:35:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.795+0000 7fc623800700 1 -- 172.21.15.16:0/3030672028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c101030 msgr2=0x7fc61c101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.795+0000 7fc623800700 1 --2- 172.21.15.16:0/3030672028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c101030 0x7fc61c101410 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc610009990 tx=0x7fc610009ca0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.795+0000 7fc623800700 1 -- 172.21.15.16:0/3030672028 shutdown_connections 2024-04-05T23:35:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.795+0000 7fc623800700 1 --2- 172.21.15.16:0/3030672028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c101030 0x7fc61c101410 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.795+0000 7fc623800700 1 --2- 172.21.15.16:0/3030672028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c106150 0x7fc61c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.795+0000 7fc623800700 1 --2- 172.21.15.16:0/3030672028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61c1019e0 0x7fc61c105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.795+0000 7fc623800700 1 -- 172.21.15.16:0/3030672028 >> 172.21.15.16:0/3030672028 conn(0x7fc61c0fc8b0 msgr2=0x7fc61c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.795+0000 7fc623800700 1 -- 172.21.15.16:0/3030672028 shutdown_connections 2024-04-05T23:35:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.795+0000 7fc623800700 1 -- 172.21.15.16:0/3030672028 wait complete. 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.796+0000 7fc623800700 1 Processor -- start 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc623800700 1 -- start start 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc623800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61c101030 0x7fc61c199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc623800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c106150 0x7fc61c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc623800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c19de60 0x7fc61c19e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc623800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c19e8e0 con 0x7fc61c19de60 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc623800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c19ea50 con 0x7fc61c101030 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc623800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c19ec10 con 0x7fc61c106150 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc620e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61c101030 0x7fc61c199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc620e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61c101030 0x7fc61c199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47930/0 (socket says 172.21.15.16:47930) 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc620e00700 1 -- 172.21.15.16:0/2146317033 learned_addr learned my addr 172.21.15.16:0/2146317033 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc621800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c19de60 0x7fc61c19e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.797+0000 7fc61be00700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c106150 0x7fc61c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc620e00700 1 -- 172.21.15.16:0/2146317033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c106150 msgr2=0x7fc61c199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc620e00700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c106150 0x7fc61c199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc620e00700 1 -- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c19de60 msgr2=0x7fc61c19e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc620e00700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c19de60 0x7fc61c19e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc620e00700 1 -- 172.21.15.16:0/2146317033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6100095f0 con 0x7fc61c101030 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc621800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c19de60 0x7fc61c19e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc620e00700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61c101030 0x7fc61c199580 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc610000c00 tx=0x7fc610005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc619600700 1 -- 172.21.15.16:0/2146317033 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61001d070 con 0x7fc61c101030 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc623800700 1 -- 172.21.15.16:0/2146317033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61c1a6110 con 0x7fc61c101030 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc619600700 1 -- 172.21.15.16:0/2146317033 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc610022470 con 0x7fc61c101030 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc619600700 1 -- 172.21.15.16:0/2146317033 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc610017650 con 0x7fc61c101030 2024-04-05T23:35:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.798+0000 7fc623800700 1 -- 172.21.15.16:0/2146317033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c1a6590 con 0x7fc61c101030 2024-04-05T23:35:08.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.803+0000 7fc619600700 1 -- 172.21.15.16:0/2146317033 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fc6100225e0 con 0x7fc61c101030 2024-04-05T23:35:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc637383b80 1 Processor -- start 2024-04-05T23:35:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc637383b80 1 -- start start 2024-04-05T23:35:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc619600700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc6040a1290 0x7fc6040a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc637383b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f65f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc637383b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188f60030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc637383b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 0x557188f60a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc637383b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188ec6b80 con 0x557188f65b80 2024-04-05T23:35:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc637383b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188eb7930 con 0x557188f605c0 2024-04-05T23:35:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc637383b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188eb52f0 con 0x557188f664a0 2024-04-05T23:35:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.804+0000 7fc619600700 1 -- 172.21.15.16:0/2146317033 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fc6100cdf30 con 0x7fc61c101030 2024-04-05T23:35:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc620e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f65f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc620e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f65f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56672/0 (socket says 172.21.15.16:56672) 2024-04-05T23:35:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc620e00700 1 -- 172.21.15.16:0/1996440546 learned_addr learned my addr 172.21.15.16:0/1996440546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc621800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc6040a1290 0x7fc6040a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc61be00700 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188f60030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc621800700 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 0x557188f60a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc620e00700 1 -- 172.21.15.16:0/1996440546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 msgr2=0x557188f60030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc620e00700 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188f60030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc620e00700 1 -- 172.21.15.16:0/1996440546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 msgr2=0x557188f60a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc620e00700 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 0x557188f60a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc620e00700 1 -- 172.21.15.16:0/1996440546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557188d32350 con 0x557188f65b80 2024-04-05T23:35:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc621800700 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 0x557188f60a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:35:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.805+0000 7fc61be00700 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188f60030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc620e00700 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f65f60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc6100976c0 tx=0x7fc61009af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8f0f92d45528b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc621800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc6040a1290 0x7fc6040a3750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x557188f5c7f0 tx=0x7fc614009920 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc60e000700 1 -- 172.21.15.16:0/1996440546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61009b8c0 con 0x557188f65b80 2024-04-05T23:35:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc60e000700 1 -- 172.21.15.16:0/1996440546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc610087040 con 0x557188f65b80 2024-04-05T23:35:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc60e000700 1 -- 172.21.15.16:0/1996440546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61008b1d0 con 0x557188f65b80 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc637383b80 1 -- 172.21.15.16:0/1996440546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 msgr2=0x557188f65f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc637383b80 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f65f60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc6100976c0 tx=0x7fc61009af90 comp rx=0 tx=0).stop 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc637383b80 1 -- 172.21.15.16:0/1996440546 shutdown_connections 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc637383b80 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f65f60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc637383b80 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 0x557188f60a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc637383b80 1 --2- 172.21.15.16:0/1996440546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188f60030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.806+0000 7fc637383b80 1 -- 172.21.15.16:0/1996440546 >> 172.21.15.16:0/1996440546 conn(0x557188ed41c0 msgr2=0x557188f5abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.807+0000 7fc637383b80 1 -- 172.21.15.16:0/1996440546 shutdown_connections 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.807+0000 7fc637383b80 1 -- 172.21.15.16:0/1996440546 wait complete. 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.807+0000 7fc637383b80 1 -- start start 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.808+0000 7fc637383b80 1 Processor -- start 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.808+0000 7fc637383b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f605c0 0x557188f6c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.808+0000 7fc637383b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f69810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.808+0000 7fc637383b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188f69e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.808+0000 7fc637383b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188eb0be0 con 0x557188f605c0 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.808+0000 7fc637383b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188e4cd20 con 0x557188f65b80 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.808+0000 7fc637383b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188e4c9c0 con 0x557188f664a0 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.808+0000 7fc637383b80 10 client.0 _ll_register_callbacks cb 0x557188cb8340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc620e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f605c0 0x557188f6c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc620e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f605c0 0x557188f6c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56680/0 (socket says 172.21.15.16:56680) 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc620e00700 1 -- 172.21.15.16:0/3585177187 learned_addr learned my addr 172.21.15.16:0/3585177187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc61be00700 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f69810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc621800700 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188f69e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc620e00700 1 -- 172.21.15.16:0/3585177187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 msgr2=0x557188f69e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc620e00700 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188f69e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc620e00700 1 -- 172.21.15.16:0/3585177187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 msgr2=0x557188f69810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc620e00700 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f69810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc620e00700 1 -- 172.21.15.16:0/3585177187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6080098e0 con 0x557188f605c0 2024-04-05T23:35:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc61be00700 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f69810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc620e00700 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f605c0 0x557188f6c490 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc610097670 tx=0x7fc610099770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71812d676994d317 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc602a00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc602a00700 1 -- 172.21.15.16:0/3585177187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61009b8c0 con 0x557188f605c0 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc602a00700 1 -- 172.21.15.16:0/3585177187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fc61009b1c0 con 0x557188f605c0 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.809+0000 7fc637383b80 1 -- 172.21.15.16:0/3585177187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557188f531e0 con 0x557188f605c0 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc602a00700 1 -- 172.21.15.16:0/3585177187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc61009eff0 con 0x557188f605c0 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc637383b80 10 client.15831 fetch_fsmap learned FSMap version 36 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc637383b80 1 -- 172.21.15.16:0/3585177187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7fc61c1a6590 con 0x557188f605c0 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc602a00700 1 -- 172.21.15.16:0/3585177187 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557188f531e0 con 0x557188f605c0 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc602a00700 1 -- 172.21.15.16:0/3585177187 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fc610094f10 con 0x557188f605c0 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc637383b80 10 client.15831 fetch_fsmap finished waiting for FSMap version 36 2024-04-05T23:35:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc637383b80 10 client.15831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc637383b80 20 client.15831 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc637383b80 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557188f79720 0x557188f7bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc637383b80 4 client.15831 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.810+0000 7fc637383b80 1 -- 172.21.15.16:0/3585177187 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557188ea96a0 con 0x557188f79720 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.811+0000 7fc61be00700 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557188f79720 0x557188f7bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.811+0000 7fc61be00700 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557188f79720 0x557188f7bb40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36cf0ff44d0575b2 server_cookie=db320188f340fdee in_seq=0 out_seq=0 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.811+0000 7fc602a00700 10 client.15831 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc602a00700 1 -- 172.21.15.16:0/3585177187 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557188ea96a0 con 0x557188f79720 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc602a00700 10 client.15831 handle_command_reply: tid=0 2024-04-05T23:35:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc637383b80 1 client.15831 shutdown 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc637383b80 1 -- 172.21.15.16:0/3585177187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f605c0 msgr2=0x557188f6c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc637383b80 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f605c0 0x557188f6c490 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc610097670 tx=0x7fc610099770 comp rx=0 tx=0).stop 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc637383b80 1 -- 172.21.15.16:0/3585177187 shutdown_connections 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc637383b80 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557188f79720 0x557188f7bb40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc637383b80 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f605c0 0x557188f6c490 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc637383b80 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188f69e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc637383b80 1 --2- 172.21.15.16:0/3585177187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f69810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc602a00700 0 client.15831 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.812+0000 7fc637383b80 1 -- 172.21.15.16:0/3585177187 >> 172.21.15.16:0/3585177187 conn(0x557188ed41c0 msgr2=0x557188f5a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.813+0000 7fc637383b80 1 -- 172.21.15.16:0/3585177187 shutdown_connections 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.813+0000 7fc637383b80 1 -- 172.21.15.16:0/3585177187 wait complete. 2024-04-05T23:35:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.813+0000 7fc637383b80 20 client.15831 trim_cache size 0 max 16384 2024-04-05T23:35:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.828+0000 7fc637383b80 1 Processor -- start 2024-04-05T23:35:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.828+0000 7fc637383b80 1 -- start start 2024-04-05T23:35:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc637383b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f605c0 0x557188f60030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc637383b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f70cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc637383b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f664a0 0x557188f5aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc637383b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188d34640 con 0x557188f664a0 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc637383b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188f766b0 con 0x557188f65b80 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc637383b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188f57aa0 con 0x557188f605c0 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc621800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f664a0 0x557188f5aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc621800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f664a0 0x557188f5aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56690/0 (socket says 172.21.15.16:56690) 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc621800700 1 -- 172.21.15.16:0/2620317972 learned_addr learned my addr 172.21.15.16:0/2620317972 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc61be00700 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f70cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc620e00700 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f605c0 0x557188f60030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc621800700 1 -- 172.21.15.16:0/2620317972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f605c0 msgr2=0x557188f60030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.829+0000 7fc621800700 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f605c0 0x557188f60030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.830+0000 7fc621800700 1 -- 172.21.15.16:0/2620317972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 msgr2=0x557188f70cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:08.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.830+0000 7fc621800700 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f70cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.830+0000 7fc621800700 1 -- 172.21.15.16:0/2620317972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557188cdeb00 con 0x557188f664a0 2024-04-05T23:35:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.830+0000 7fc620e00700 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f605c0 0x557188f60030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:35:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.830+0000 7fc61be00700 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f70cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.830+0000 7fc621800700 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f664a0 0x557188f5aa00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc614009080 tx=0x7fc61400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae6c2dc9c6c2a9a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.830+0000 7fc60ea00700 1 -- 172.21.15.16:0/2620317972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61401a640 con 0x557188f664a0 2024-04-05T23:35:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.830+0000 7fc60ea00700 1 -- 172.21.15.16:0/2620317972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc61400f040 con 0x557188f664a0 2024-04-05T23:35:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.830+0000 7fc60ea00700 1 -- 172.21.15.16:0/2620317972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614024b00 con 0x557188f664a0 2024-04-05T23:35:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 -- 172.21.15.16:0/2620317972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f664a0 msgr2=0x557188f5aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f664a0 0x557188f5aa00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc614009080 tx=0x7fc61400b040 comp rx=0 tx=0).stop 2024-04-05T23:35:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 -- 172.21.15.16:0/2620317972 shutdown_connections 2024-04-05T23:35:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f664a0 0x557188f5aa00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f605c0 0x557188f60030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 --2- 172.21.15.16:0/2620317972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f65b80 0x557188f70cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 -- 172.21.15.16:0/2620317972 >> 172.21.15.16:0/2620317972 conn(0x557188ed41c0 msgr2=0x557188f648c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 -- 172.21.15.16:0/2620317972 shutdown_connections 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 -- 172.21.15.16:0/2620317972 wait complete. 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.831+0000 7fc637383b80 1 -- start start 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc637383b80 1 Processor -- start 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc637383b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 0x557188f6ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc637383b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f6d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc637383b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188fc6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc637383b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188fc6ac0 con 0x557188f65b80 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc637383b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188fc6c30 con 0x557188f605c0 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc637383b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557188fc6df0 con 0x557188f664a0 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc637383b80 10 client.0 _ll_register_callbacks cb 0x7fc608009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc61be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f6d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc620e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 0x557188f6ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc61be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f6d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56694/0 (socket says 172.21.15.16:56694) 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc61be00700 1 -- 172.21.15.16:0/2522944347 learned_addr learned my addr 172.21.15.16:0/2522944347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.832+0000 7fc621800700 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188fc6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc61be00700 1 -- 172.21.15.16:0/2522944347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 msgr2=0x557188fc6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc61be00700 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188fc6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc61be00700 1 -- 172.21.15.16:0/2522944347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 msgr2=0x557188f6ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc61be00700 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 0x557188f6ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc61be00700 1 -- 172.21.15.16:0/2522944347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc608012040 con 0x557188f65b80 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc621800700 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188fc6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc61be00700 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f6d3d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc61c19aad0 tx=0x7fc60801fcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a916d0d70d791dc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc602a00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc602a00700 1 -- 172.21.15.16:0/2522944347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608017040 con 0x557188f65b80 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc637383b80 1 -- 172.21.15.16:0/2522944347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc610087040 con 0x557188f65b80 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.833+0000 7fc602a00700 1 -- 172.21.15.16:0/2522944347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fc608028470 con 0x557188f65b80 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc602a00700 1 -- 172.21.15.16:0/2522944347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc608028630 con 0x557188f65b80 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc637383b80 10 client.15837 fetch_fsmap learned FSMap version 36 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc637383b80 1 -- 172.21.15.16:0/2522944347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7fc614008df0 con 0x557188f65b80 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc602a00700 1 -- 172.21.15.16:0/2522944347 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc610087040 con 0x557188f65b80 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc602a00700 1 -- 172.21.15.16:0/2522944347 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fc60802f070 con 0x557188f65b80 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc637383b80 10 client.15837 fetch_fsmap finished waiting for FSMap version 36 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc637383b80 10 client.15837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc637383b80 20 client.15837 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc637383b80 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557188fce790 0x557188fd0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc637383b80 4 client.15837 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:35:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc637383b80 1 -- 172.21.15.16:0/2522944347 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557188cffc90 con 0x557188fce790 2024-04-05T23:35:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.834+0000 7fc620e00700 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557188fce790 0x557188fd0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.835+0000 7fc620e00700 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557188fce790 0x557188fd0bb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eee14cb6577df5b server_cookie=5523f071806ef33d in_seq=0 out_seq=0 2024-04-05T23:35:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:08.835+0000 7fc602a00700 10 client.15837 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:08.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22114 4.86 MB/sec warmup 28 sec latency 3047.539 ms 2024-04-05T23:35:09.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:08 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:09.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:08 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:09.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:08 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:09.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22114 4.69 MB/sec warmup 29 sec latency 4047.631 ms 2024-04-05T23:35:10.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:09 smithi027 ceph-mon[78597]: pgmap v51: 97 pgs: 97 active+clean; 9.2 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 171 KiB/s wr, 56 op/s 2024-04-05T23:35:10.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:09 smithi016 ceph-mon[38248]: pgmap v51: 97 pgs: 97 active+clean; 9.2 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 171 KiB/s wr, 56 op/s 2024-04-05T23:35:10.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:09 smithi130 ceph-mon[79417]: pgmap v51: 97 pgs: 97 active+clean; 9.2 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 171 KiB/s wr, 56 op/s 2024-04-05T23:35:10.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22237 4.55 MB/sec warmup 30 sec latency 4953.031 ms 2024-04-05T23:35:11.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22237 4.40 MB/sec warmup 31 sec latency 1021.281 ms 2024-04-05T23:35:12.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:11 smithi027 ceph-mon[78597]: pgmap v52: 97 pgs: 97 active+clean; 18 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 1.4 MiB/s wr, 78 op/s 2024-04-05T23:35:12.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:11 smithi016 ceph-mon[38248]: pgmap v52: 97 pgs: 97 active+clean; 18 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 1.4 MiB/s wr, 78 op/s 2024-04-05T23:35:12.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:11 smithi130 ceph-mon[79417]: pgmap v52: 97 pgs: 97 active+clean; 18 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 1.4 MiB/s wr, 78 op/s 2024-04-05T23:35:12.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.663+0000 7fc602a00700 1 -- 172.21.15.16:0/2522944347 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557188cffc90 con 0x557188fce790 2024-04-05T23:35:12.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.663+0000 7fc602a00700 10 client.15837 handle_command_reply: tid=0 2024-04-05T23:35:12.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.663+0000 7fc637383b80 1 client.15837 shutdown 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 -- 172.21.15.16:0/2522944347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 msgr2=0x557188f6d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f6d3d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc61c19aad0 tx=0x7fc60801fcd0 comp rx=0 tx=0).stop 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 -- 172.21.15.16:0/2522944347 shutdown_connections 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557188fce790 0x557188fd0bb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557188f664a0 0x557188fc6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557188f605c0 0x557188f6ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 --2- 172.21.15.16:0/2522944347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557188f65b80 0x557188f6d3d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc602a00700 0 client.15837 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 -- 172.21.15.16:0/2522944347 >> 172.21.15.16:0/2522944347 conn(0x557188ed41c0 msgr2=0x557188f75b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 -- 172.21.15.16:0/2522944347 shutdown_connections 2024-04-05T23:35:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 1 -- 172.21.15.16:0/2522944347 wait complete. 2024-04-05T23:35:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.664+0000 7fc637383b80 20 client.15837 trim_cache size 0 max 16384 2024-04-05T23:35:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:35:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:35:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:35:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 -- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc6040a1290 msgr2=0x7fc6040a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc6040a1290 0x7fc6040a3750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x557188f5c7f0 tx=0x7fc614009920 comp rx=0 tx=0).stop 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 -- 172.21.15.16:0/2146317033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61c101030 msgr2=0x7fc61c199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61c101030 0x7fc61c199580 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc610000c00 tx=0x7fc610005740 comp rx=0 tx=0).stop 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 -- 172.21.15.16:0/2146317033 shutdown_connections 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc6040a1290 0x7fc6040a3750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c19de60 0x7fc61c19e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61c101030 0x7fc61c199580 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 --2- 172.21.15.16:0/2146317033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c106150 0x7fc61c199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 -- 172.21.15.16:0/2146317033 >> 172.21.15.16:0/2146317033 conn(0x7fc61c0fc8b0 msgr2=0x7fc61c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 -- 172.21.15.16:0/2146317033 shutdown_connections 2024-04-05T23:35:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:12.665+0000 7fc623800700 1 -- 172.21.15.16:0/2146317033 wait complete. 2024-04-05T23:35:12.982 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22237 4.26 MB/sec warmup 32 sec latency 2021.379 ms 2024-04-05T23:35:13.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22237 4.13 MB/sec warmup 33 sec latency 3021.489 ms 2024-04-05T23:35:14.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:13 smithi027 ceph-mon[78597]: pgmap v53: 97 pgs: 97 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 2.7 MiB/s wr, 97 op/s 2024-04-05T23:35:14.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:13 smithi016 ceph-mon[38248]: pgmap v53: 97 pgs: 97 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 2.7 MiB/s wr, 97 op/s 2024-04-05T23:35:14.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:13 smithi130 ceph-mon[79417]: pgmap v53: 97 pgs: 97 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 2.7 MiB/s wr, 97 op/s 2024-04-05T23:35:14.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22237 4.01 MB/sec warmup 34 sec latency 4021.628 ms 2024-04-05T23:35:15.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22238 3.90 MB/sec warmup 35 sec latency 4924.604 ms 2024-04-05T23:35:16.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:15 smithi027 ceph-mon[78597]: pgmap v54: 97 pgs: 97 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 2.7 MiB/s wr, 90 op/s 2024-04-05T23:35:16.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:15 smithi016 ceph-mon[38248]: pgmap v54: 97 pgs: 97 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 2.7 MiB/s wr, 90 op/s 2024-04-05T23:35:16.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:15 smithi130 ceph-mon[79417]: pgmap v54: 97 pgs: 97 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 2.7 MiB/s wr, 90 op/s 2024-04-05T23:35:16.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22238 3.79 MB/sec warmup 36 sec latency 1097.228 ms 2024-04-05T23:35:17.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22238 3.69 MB/sec warmup 37 sec latency 2097.384 ms 2024-04-05T23:35:18.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:17 smithi027 ceph-mon[78597]: pgmap v55: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 3.3 MiB/s wr, 119 op/s 2024-04-05T23:35:18.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:17 smithi016 ceph-mon[38248]: pgmap v55: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 3.3 MiB/s wr, 119 op/s 2024-04-05T23:35:18.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:17 smithi130 ceph-mon[79417]: pgmap v55: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 3.3 MiB/s wr, 119 op/s 2024-04-05T23:35:18.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22238 3.59 MB/sec warmup 38 sec latency 3097.456 ms 2024-04-05T23:35:19.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:18 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:19.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:18 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:19.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:18 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:35:19.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22238 3.50 MB/sec warmup 39 sec latency 4097.587 ms 2024-04-05T23:35:20.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:19 smithi027 ceph-mon[78597]: pgmap v56: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 3.2 MiB/s wr, 84 op/s 2024-04-05T23:35:20.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:19 smithi016 ceph-mon[38248]: pgmap v56: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 3.2 MiB/s wr, 84 op/s 2024-04-05T23:35:20.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:19 smithi130 ceph-mon[79417]: pgmap v56: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 3.2 MiB/s wr, 84 op/s 2024-04-05T23:35:20.983 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22304 3.41 MB/sec warmup 40 sec latency 4996.605 ms 2024-04-05T23:35:21.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22304 3.33 MB/sec warmup 41 sec latency 1057.105 ms 2024-04-05T23:35:22.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:21 smithi027 ceph-mon[78597]: pgmap v57: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 3.2 MiB/s wr, 87 op/s 2024-04-05T23:35:22.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:21 smithi016 ceph-mon[38248]: pgmap v57: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 3.2 MiB/s wr, 87 op/s 2024-04-05T23:35:22.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:21 smithi130 ceph-mon[79417]: pgmap v57: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 3.2 MiB/s wr, 87 op/s 2024-04-05T23:35:22.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22304 3.25 MB/sec warmup 42 sec latency 2057.236 ms 2024-04-05T23:35:23.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-05T23:35:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.978+0000 7fd0c6e00700 1 -- 172.21.15.16:0/1977471952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073280 msgr2=0x7fd0c0073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.978+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/1977471952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073280 0x7fd0c0073660 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009990 tx=0x7fd0b0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.978+0000 7fd0c6e00700 1 -- 172.21.15.16:0/1977471952 shutdown_connections 2024-04-05T23:35:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.978+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/1977471952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073280 0x7fd0c0073660 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.978+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/1977471952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c0111e40 0x7fd0c0114230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.978+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/1977471952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0c0073ba0 0x7fd0c0111900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.978+0000 7fd0c6e00700 1 -- 172.21.15.16:0/1977471952 >> 172.21.15.16:0/1977471952 conn(0x7fd0c00fc900 msgr2=0x7fd0c00fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.978+0000 7fd0c6e00700 1 -- 172.21.15.16:0/1977471952 shutdown_connections 2024-04-05T23:35:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.978+0000 7fd0c6e00700 1 -- 172.21.15.16:0/1977471952 wait complete. 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.979+0000 7fd0c6e00700 1 Processor -- start 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0c6e00700 1 -- start start 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0c6e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0c0073280 0x7fd0c01a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0c6e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073ba0 0x7fd0c01a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0c6e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c0111e40 0x7fd0c019d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0c6e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c019de70 con 0x7fd0c0073ba0 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0c6e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c019dfe0 con 0x7fd0c0111e40 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0c6e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c019e1a0 con 0x7fd0c0073280 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0bf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073ba0 0x7fd0c01a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0bf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073ba0 0x7fd0c01a3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36458/0 (socket says 172.21.15.16:36458) 2024-04-05T23:35:23.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0bf400700 1 -- 172.21.15.16:0/2433880721 learned_addr learned my addr 172.21.15.16:0/2433880721 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c0111e40 0x7fd0c019d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.980+0000 7fd0bfe00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0c0073280 0x7fd0c01a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0bf400700 1 -- 172.21.15.16:0/2433880721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0c0073280 msgr2=0x7fd0c01a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0bf400700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0c0073280 0x7fd0c01a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0bf400700 1 -- 172.21.15.16:0/2433880721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c0111e40 msgr2=0x7fd0c019d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0bf400700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c0111e40 0x7fd0c019d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0bf400700 1 -- 172.21.15.16:0/2433880721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00095f0 con 0x7fd0c0073ba0 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0bf400700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073ba0 0x7fd0c01a3bc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac00e9a0 tx=0x7fd0ac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0bcc00700 1 -- 172.21.15.16:0/2433880721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac009790 con 0x7fd0c0073ba0 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0bcc00700 1 -- 172.21.15.16:0/2433880721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0ac010430 con 0x7fd0c0073ba0 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0bcc00700 1 -- 172.21.15.16:0/2433880721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac018680 con 0x7fd0c0073ba0 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0c6e00700 1 -- 172.21.15.16:0/2433880721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c019e480 con 0x7fd0c0073ba0 2024-04-05T23:35:23.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.981+0000 7fd0c6e00700 1 -- 172.21.15.16:0/2433880721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c01aa670 con 0x7fd0c0073ba0 2024-04-05T23:35:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.984+0000 7fd0bcc00700 1 -- 172.21.15.16:0/2433880721 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fd0ac0105a0 con 0x7fd0c0073ba0 2024-04-05T23:35:23.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.986+0000 7fd0bcc00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd0a40a1290 0x7fd0a40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.986+0000 7fd0da8ebb80 1 Processor -- start 2024-04-05T23:35:23.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.986+0000 7fd0bcc00700 1 -- 172.21.15.16:0/2433880721 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd0ac014070 con 0x7fd0c0073ba0 2024-04-05T23:35:23.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.986+0000 7fd0bfe00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd0a40a1290 0x7fd0a40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.986+0000 7fd0da8ebb80 1 -- start start 2024-04-05T23:35:23.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.986+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfbd180 0x55dbdbfbd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfb7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbf1dbe0 con 0x55dbdbfb7ca0 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbf0e990 con 0x55dbdbfb72e0 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbf0c350 con 0x55dbdbfbd180 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0bfe00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd0a40a1290 0x7fd0a40a3750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0006010 tx=0x7fd0b000b260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0bfe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfbd180 0x55dbdbfbd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0c4e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0bfe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfbd180 0x55dbdbfbd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:47608/0 (socket says 172.21.15.16:47608) 2024-04-05T23:35:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0bfe00700 1 -- 172.21.15.16:0/3053648358 learned_addr learned my addr 172.21.15.16:0/3053648358 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0bf400700 1 --2- 172.21.15.16:0/3053648358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfb7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0c4e00700 1 -- 172.21.15.16:0/3053648358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfbd180 msgr2=0x55dbdbfbd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/3053648358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfbd180 0x55dbdbfbd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0c4e00700 1 -- 172.21.15.16:0/3053648358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb72e0 msgr2=0x55dbdbfb7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/3053648358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfb7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.987+0000 7fd0c4e00700 1 -- 172.21.15.16:0/3053648358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbdbd893b0 con 0x55dbdbfb7ca0 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.988+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/3053648358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc3f10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd0c006a4c0 tx=0x7fd0b8009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49457b9ebc592912 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.988+0000 7fd0b5600700 1 -- 172.21.15.16:0/3053648358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b800e040 con 0x55dbdbfb7ca0 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.988+0000 7fd0b5600700 1 -- 172.21.15.16:0/3053648358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0b800f040 con 0x55dbdbfb7ca0 2024-04-05T23:35:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.988+0000 7fd0b5600700 1 -- 172.21.15.16:0/3053648358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8004250 con 0x55dbdbfb7ca0 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/3053648358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 msgr2=0x55dbdbfc3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/3053648358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc3f10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd0c006a4c0 tx=0x7fd0b8009920 comp rx=0 tx=0).stop 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/3053648358 shutdown_connections 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/3053648358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc3f10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/3053648358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfbd180 0x55dbdbfbd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/3053648358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfb7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/3053648358 >> 172.21.15.16:0/3053648358 conn(0x55dbdbf2b220 msgr2=0x55dbdbfb2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/3053648358 shutdown_connections 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/3053648358 wait complete. 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.989+0000 7fd0da8ebb80 1 -- start start 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0da8ebb80 1 Processor -- start 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfc3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 0x55dbdbfc1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbf07c40 con 0x55dbdbfbd180 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbea3d80 con 0x55dbdbfb7ca0 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbea3a20 con 0x55dbdbfb72e0 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0da8ebb80 10 client.0 _ll_register_callbacks cb 0x55dbdbd0f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0bfe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfc3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0c4e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 0x55dbdbfc1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0bfe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfc3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:47618/0 (socket says 172.21.15.16:47618) 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0bfe00700 1 -- 172.21.15.16:0/2373539868 learned_addr learned my addr 172.21.15.16:0/2373539868 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.990+0000 7fd0bf400700 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0c4e00700 1 -- 172.21.15.16:0/2373539868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 msgr2=0x55dbdbfc3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfc3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0c4e00700 1 -- 172.21.15.16:0/2373539868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 msgr2=0x55dbdbfc1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0c4e00700 1 -- 172.21.15.16:0/2373539868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b8011230 con 0x55dbdbfbd180 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 0x55dbdbfc1740 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd0b800b010 tx=0x7fd0b8007730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5d61ea661b05b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0aa000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0aa000700 1 -- 172.21.15.16:0/2373539868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b800e040 con 0x55dbdbfbd180 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0aa000700 1 -- 172.21.15.16:0/2373539868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd0b8007db0 con 0x55dbdbfbd180 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.991+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2373539868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dbdbfaa490 con 0x55dbdbfbd180 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0aa000700 1 -- 172.21.15.16:0/2373539868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0b8005040 con 0x55dbdbfbd180 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0da8ebb80 10 client.15849 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2373539868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55dbdbd35b60 con 0x55dbdbfbd180 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0aa000700 1 -- 172.21.15.16:0/2373539868 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dbdbfaa490 con 0x55dbdbfbd180 2024-04-05T23:35:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0aa000700 1 -- 172.21.15.16:0/2373539868 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd0b802e070 con 0x55dbdbfbd180 2024-04-05T23:35:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0da8ebb80 10 client.15849 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:23.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22304 3.18 MB/sec warmup 43 sec latency 3057.309 ms 2024-04-05T23:35:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0da8ebb80 10 client.15849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0da8ebb80 20 client.15849 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dbdbfd14e0 0x55dbdbfd3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0da8ebb80 4 client.15849 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.992+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2373539868 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dbdbf00700 con 0x55dbdbfd14e0 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.993+0000 7fd0bfe00700 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dbdbfd14e0 0x55dbdbfd3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.993+0000 7fd0bfe00700 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dbdbfd14e0 0x55dbdbfd3900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94b65e984f21c5ab server_cookie=43ea0c96e619f343 in_seq=0 out_seq=0 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.993+0000 7fd0aa000700 10 client.15849 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0aa000700 1 -- 172.21.15.16:0/2373539868 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dbdbf00700 con 0x55dbdbfd14e0 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0aa000700 10 client.15849 handle_command_reply: tid=0 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 client.15849 shutdown 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2373539868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 msgr2=0x55dbdbfc1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 0x55dbdbfc1740 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd0b800b010 tx=0x7fd0b8007730 comp rx=0 tx=0).stop 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2373539868 shutdown_connections 2024-04-05T23:35:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dbdbfd14e0 0x55dbdbfd3900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 0x55dbdbfc1740 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfc3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0aa000700 0 client.15849 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2373539868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2373539868 >> 172.21.15.16:0/2373539868 conn(0x55dbdbf2b220 msgr2=0x55dbdbfac460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2373539868 shutdown_connections 2024-04-05T23:35:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2373539868 wait complete. 2024-04-05T23:35:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:23.994+0000 7fd0da8ebb80 20 client.15849 trim_cache size 0 max 16384 2024-04-05T23:35:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.013+0000 7fd0da8ebb80 1 Processor -- start 2024-04-05T23:35:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.013+0000 7fd0da8ebb80 1 -- start start 2024-04-05T23:35:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.013+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfcd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.013+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfb26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.013+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfbd180 0x55dbdbfac560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbd8b6a0 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbfd4020 con 0x55dbdbfbd180 2024-04-05T23:35:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbfbf700 con 0x55dbdbfb72e0 2024-04-05T23:35:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0bf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfb26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0c4e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfbd180 0x55dbdbfac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0bf400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfb26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36480/0 (socket says 172.21.15.16:36480) 2024-04-05T23:35:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0c4e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfbd180 0x55dbdbfac560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37658/0 (socket says 172.21.15.16:37658) 2024-04-05T23:35:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0bf400700 1 -- 172.21.15.16:0/2987661064 learned_addr learned my addr 172.21.15.16:0/2987661064 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0bfe00700 1 --2- 172.21.15.16:0/2987661064 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfcd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0c4e00700 1 -- 172.21.15.16:0/2987661064 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 msgr2=0x55dbdbfcd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/2987661064 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfcd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0c4e00700 1 -- 172.21.15.16:0/2987661064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 msgr2=0x55dbdbfb26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/2987661064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfb26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.014+0000 7fd0c4e00700 1 -- 172.21.15.16:0/2987661064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbdbecdea0 con 0x55dbdbfbd180 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.015+0000 7fd0c4e00700 1 --2- 172.21.15.16:0/2987661064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfbd180 0x55dbdbfac560 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8011520 tx=0x7fd0b8011b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c57ed8a0ff99ccc server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.015+0000 7fd0b6000700 1 -- 172.21.15.16:0/2987661064 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8007bb0 con 0x55dbdbfbd180 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.015+0000 7fd0b6000700 1 -- 172.21.15.16:0/2987661064 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0b8015d90 con 0x55dbdbfbd180 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.015+0000 7fd0b6000700 1 -- 172.21.15.16:0/2987661064 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8027690 con 0x55dbdbfbd180 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.015+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2987661064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfbd180 msgr2=0x55dbdbfac560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.015+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2987661064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfbd180 0x55dbdbfac560 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8011520 tx=0x7fd0b8011b10 comp rx=0 tx=0).stop 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.016+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2987661064 shutdown_connections 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.016+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2987661064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfbd180 0x55dbdbfac560 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.016+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2987661064 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfcd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.016+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/2987661064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfb26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.016+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2987661064 >> 172.21.15.16:0/2987661064 conn(0x55dbdbf2b220 msgr2=0x55dbdbfbfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.016+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2987661064 shutdown_connections 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.016+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/2987661064 wait complete. 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.016+0000 7fd0da8ebb80 1 -- start start 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.016+0000 7fd0da8ebb80 1 Processor -- start 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfc82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0da8ebb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 0x55dbdbfc6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbfc70f0 con 0x55dbdbfbd180 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbfc7260 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0da8ebb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbdbfc7420 con 0x55dbdbfb72e0 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0da8ebb80 10 client.0 _ll_register_callbacks cb 0x7fd0b0005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0bf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0c4e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 0x55dbdbfc6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0bf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37660/0 (socket says 172.21.15.16:37660) 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0bf400700 1 -- 172.21.15.16:0/1730897430 learned_addr learned my addr 172.21.15.16:0/1730897430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.017+0000 7fd0bfe00700 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfc82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0bf400700 1 -- 172.21.15.16:0/1730897430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 msgr2=0x55dbdbfc82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0bf400700 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfc82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0bf400700 1 -- 172.21.15.16:0/1730897430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 msgr2=0x55dbdbfc6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0bf400700 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 0x55dbdbfc6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0bf400700 1 -- 172.21.15.16:0/1730897430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dbdbd35b60 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0bf400700 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc6670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac096a30 tx=0x7fd0ac0cc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df5b9af770cae7a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0aa000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0aa000700 1 -- 172.21.15.16:0/1730897430 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0afea0 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0aa000700 1 -- 172.21.15.16:0/1730897430 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd0ac0a2440 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/1730897430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd0b800f040 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0aa000700 1 -- 172.21.15.16:0/1730897430 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0ac0996b0 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0da8ebb80 10 client.25889 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.018+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/1730897430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55dbdbd8dcd0 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.019+0000 7fd0aa000700 1 -- 172.21.15.16:0/1730897430 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0b800f040 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.019+0000 7fd0aa000700 1 -- 172.21.15.16:0/1730897430 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd0ac0a20d0 con 0x55dbdbfb7ca0 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.019+0000 7fd0da8ebb80 10 client.25889 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.019+0000 7fd0da8ebb80 10 client.25889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.019+0000 7fd0da8ebb80 20 client.25889 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.019+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dbdc026970 0x55dbdc028d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.019+0000 7fd0da8ebb80 4 client.25889 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.019+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/1730897430 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55dbdbd56cf0 con 0x55dbdc026970 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.019+0000 7fd0bfe00700 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dbdc026970 0x55dbdc028d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0bfe00700 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dbdc026970 0x55dbdc028d90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6d4a4ed09e0b26b server_cookie=762fa107892d2424 in_seq=0 out_seq=0 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0aa000700 10 client.25889 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0aa000700 1 -- 172.21.15.16:0/1730897430 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55dbdbd56cf0 con 0x55dbdc026970 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0aa000700 10 client.25889 handle_command_reply: tid=0 2024-04-05T23:35:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 client.25889 shutdown 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/1730897430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 msgr2=0x55dbdbfc6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc6670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac096a30 tx=0x7fd0ac0cc040 comp rx=0 tx=0).stop 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/1730897430 shutdown_connections 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dbdc026970 0x55dbdc028d90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbdbfbd180 0x55dbdbfc6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dbdbfb72e0 0x55dbdbfc82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 --2- 172.21.15.16:0/1730897430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dbdbfb7ca0 0x55dbdbfc6670 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0aa000700 0 client.25889 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/1730897430 >> 172.21.15.16:0/1730897430 conn(0x55dbdbf2b220 msgr2=0x55dbdbfcd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/1730897430 shutdown_connections 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 1 -- 172.21.15.16:0/1730897430 wait complete. 2024-04-05T23:35:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.020+0000 7fd0da8ebb80 20 client.25889 trim_cache size 0 max 16384 2024-04-05T23:35:24.018 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:35:24.018 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-04-05T23:35:24.018 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-04-05T23:35:24.018 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:35:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 -- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd0a40a1290 msgr2=0x7fd0a40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd0a40a1290 0x7fd0a40a3750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0006010 tx=0x7fd0b000b260 comp rx=0 tx=0).stop 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 -- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073ba0 msgr2=0x7fd0c01a3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073ba0 0x7fd0c01a3bc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac00e9a0 tx=0x7fd0ac00ecb0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 -- 172.21.15.16:0/2433880721 shutdown_connections 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd0a40a1290 0x7fd0a40a3750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c0111e40 0x7fd0c019d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0c0073280 0x7fd0c01a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 --2- 172.21.15.16:0/2433880721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0c0073ba0 0x7fd0c01a3bc0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 -- 172.21.15.16:0/2433880721 >> 172.21.15.16:0/2433880721 conn(0x7fd0c00fc900 msgr2=0x7fd0c00fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 -- 172.21.15.16:0/2433880721 shutdown_connections 2024-04-05T23:35:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.021+0000 7fd0c6e00700 1 -- 172.21.15.16:0/2433880721 wait complete. 2024-04-05T23:35:24.031 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-05T23:35:24.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:35:24.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:23 smithi016 ceph-mon[38248]: pgmap v58: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.9 MiB/s wr, 71 op/s 2024-04-05T23:35:24.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:23 smithi027 ceph-mon[78597]: pgmap v58: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.9 MiB/s wr, 71 op/s 2024-04-05T23:35:24.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.240+0000 7fec42600700 1 -- 172.21.15.16:0/3722165765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c1060b0 msgr2=0x7fec3c10dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.240+0000 7fec42600700 1 --2- 172.21.15.16:0/3722165765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c1060b0 0x7fec3c10dc30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec34009f90 tx=0x7fec3400b330 comp rx=0 tx=0).stop 2024-04-05T23:35:24.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.240+0000 7fec42600700 1 -- 172.21.15.16:0/3722165765 shutdown_connections 2024-04-05T23:35:24.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.240+0000 7fec42600700 1 --2- 172.21.15.16:0/3722165765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c1060b0 0x7fec3c10dc30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.240+0000 7fec42600700 1 --2- 172.21.15.16:0/3722165765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec3c100f90 0x7fec3c101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.240+0000 7fec42600700 1 --2- 172.21.15.16:0/3722165765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c101940 0x7fec3c105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.240+0000 7fec42600700 1 -- 172.21.15.16:0/3722165765 >> 172.21.15.16:0/3722165765 conn(0x7fec3c0fc830 msgr2=0x7fec3c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.240+0000 7fec42600700 1 -- 172.21.15.16:0/3722165765 shutdown_connections 2024-04-05T23:35:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.241+0000 7fec42600700 1 -- 172.21.15.16:0/3722165765 wait complete. 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.241+0000 7fec42600700 1 Processor -- start 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec42600700 1 -- start start 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec42600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c100f90 0x7fec3c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec42600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c1060b0 0x7fec3c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec42600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec3c074ea0 0x7fec3c073ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec42600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c1a9ff0 con 0x7fec3c1060b0 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec42600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c1aa160 con 0x7fec3c074ea0 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec42600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c1aa320 con 0x7fec3c100f90 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec3b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c100f90 0x7fec3c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec3b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c100f90 0x7fec3c0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:47656/0 (socket says 172.21.15.16:47656) 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec3b400700 1 -- 172.21.15.16:0/2037454867 learned_addr learned my addr 172.21.15.16:0/2037454867 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec3be00700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec3c074ea0 0x7fec3c073ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.242+0000 7fec3aa00700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c1060b0 0x7fec3c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec3be00700 1 -- 172.21.15.16:0/2037454867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c100f90 msgr2=0x7fec3c0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec3be00700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c100f90 0x7fec3c0731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec3be00700 1 -- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c1060b0 msgr2=0x7fec3c073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec3be00700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c1060b0 0x7fec3c073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec3be00700 1 -- 172.21.15.16:0/2037454867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c0095f0 con 0x7fec3c074ea0 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec3b400700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c100f90 0x7fec3c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec3be00700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec3c074ea0 0x7fec3c073ca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fec3400b6c0 tx=0x7fec34009440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec33e00700 1 -- 172.21.15.16:0/2037454867 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3400e040 con 0x7fec3c074ea0 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec33e00700 1 -- 172.21.15.16:0/2037454867 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec3400be50 con 0x7fec3c074ea0 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec34009c40 con 0x7fec3c074ea0 2024-04-05T23:35:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.243+0000 7fec33e00700 1 -- 172.21.15.16:0/2037454867 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34005040 con 0x7fec3c074ea0 2024-04-05T23:35:24.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.244+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3c1aa890 con 0x7fec3c074ea0 2024-04-05T23:35:24.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.245+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3c109380 con 0x7fec3c074ea0 2024-04-05T23:35:24.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.246+0000 7fec33e00700 1 -- 172.21.15.16:0/2037454867 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fec3401d690 con 0x7fec3c074ea0 2024-04-05T23:35:24.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.248+0000 7fec33e00700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fec200a1290 0x7fec200a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.248+0000 7fec33e00700 1 -- 172.21.15.16:0/2037454867 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fec340cde70 con 0x7fec3c074ea0 2024-04-05T23:35:24.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.248+0000 7fec3b400700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fec200a1290 0x7fec200a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.248+0000 7fec3b400700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fec200a1290 0x7fec200a3750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec2c005d10 tx=0x7fec2c009310 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.252+0000 7fec33e00700 1 -- 172.21.15.16:0/2037454867 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fec34097320 con 0x7fec3c074ea0 2024-04-05T23:35:24.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:23 smithi130 ceph-mon[79417]: pgmap v58: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.9 MiB/s wr, 71 op/s 2024-04-05T23:35:24.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.518+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec3c102460 con 0x7fec3c074ea0 2024-04-05T23:35:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.519+0000 7fec33e00700 1 -- 172.21.15.16:0/2037454867 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+10913 (secure 0 0 0) 0x7fec3c102460 con 0x7fec3c074ea0 2024-04-05T23:35:24.511 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:35:24.511 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:35:17.871248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":50,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_25534":{"gid":25534,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/1942303111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1942303111},{"type":"v1","addr":"172.21.15.130:6835","nonce":1942303111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:35:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.521+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fec200a1290 msgr2=0x7fec200a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.521+0000 7fec42600700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fec200a1290 0x7fec200a3750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec2c005d10 tx=0x7fec2c009310 comp rx=0 tx=0).stop 2024-04-05T23:35:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.521+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec3c074ea0 msgr2=0x7fec3c073ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.521+0000 7fec42600700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec3c074ea0 0x7fec3c073ca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fec3400b6c0 tx=0x7fec34009440 comp rx=0 tx=0).stop 2024-04-05T23:35:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.522+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 shutdown_connections 2024-04-05T23:35:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.522+0000 7fec42600700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fec200a1290 0x7fec200a3750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.522+0000 7fec42600700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec3c074ea0 0x7fec3c073ca0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.522+0000 7fec42600700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec3c100f90 0x7fec3c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.522+0000 7fec42600700 1 --2- 172.21.15.16:0/2037454867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec3c1060b0 0x7fec3c073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.522+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 >> 172.21.15.16:0/2037454867 conn(0x7fec3c0fc830 msgr2=0x7fec3c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.522+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 shutdown_connections 2024-04-05T23:35:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.522+0000 7fec42600700 1 -- 172.21.15.16:0/2037454867 wait complete. 2024-04-05T23:35:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 37 2024-04-05T23:35:24.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-05T23:35:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.746+0000 7fb763e00700 1 -- 172.21.15.16:0/609162525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 msgr2=0x7fb75c068600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.746+0000 7fb763e00700 1 --2- 172.21.15.16:0/609162525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 0x7fb75c068600 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb7500099c0 tx=0x7fb750009cd0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.746+0000 7fb763e00700 1 -- 172.21.15.16:0/609162525 shutdown_connections 2024-04-05T23:35:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.746+0000 7fb763e00700 1 --2- 172.21.15.16:0/609162525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 0x7fb75c068600 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.746+0000 7fb763e00700 1 --2- 172.21.15.16:0/609162525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75c1061d0 0x7fb75c10dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.746+0000 7fb763e00700 1 --2- 172.21.15.16:0/609162525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb75c068bd0 0x7fb75c105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.746+0000 7fb763e00700 1 -- 172.21.15.16:0/609162525 >> 172.21.15.16:0/609162525 conn(0x7fb75c076e80 msgr2=0x7fb75c077290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.747+0000 7fb763e00700 1 -- 172.21.15.16:0/609162525 shutdown_connections 2024-04-05T23:35:24.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.747+0000 7fb763e00700 1 -- 172.21.15.16:0/609162525 wait complete. 2024-04-05T23:35:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.748+0000 7fb763e00700 1 Processor -- start 2024-04-05T23:35:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.748+0000 7fb763e00700 1 -- start start 2024-04-05T23:35:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.748+0000 7fb763e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 0x7fb75c199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.748+0000 7fb763e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75c1061d0 0x7fb75c199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.748+0000 7fb763e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb75c19de30 0x7fb75c19e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.748+0000 7fb763e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c19e8b0 con 0x7fb75c068220 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.748+0000 7fb763e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c19ea20 con 0x7fb75c1061d0 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.748+0000 7fb763e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c19ebe0 con 0x7fb75c19de30 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.748+0000 7fb761400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 0x7fb75c199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb761400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 0x7fb75c199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36516/0 (socket says 172.21.15.16:36516) 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb761400700 1 -- 172.21.15.16:0/2563533648 learned_addr learned my addr 172.21.15.16:0/2563533648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb760a00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75c1061d0 0x7fb75c199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb761e00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb75c19de30 0x7fb75c19e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb761400700 1 -- 172.21.15.16:0/2563533648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb75c19de30 msgr2=0x7fb75c19e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb761400700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb75c19de30 0x7fb75c19e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb761400700 1 -- 172.21.15.16:0/2563533648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75c1061d0 msgr2=0x7fb75c199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb761400700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75c1061d0 0x7fb75c199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb761400700 1 -- 172.21.15.16:0/2563533648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7500095f0 con 0x7fb75c068220 2024-04-05T23:35:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.749+0000 7fb761400700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 0x7fb75c199550 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb75000b580 tx=0x7fb75000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.750+0000 7fb75a000700 1 -- 172.21.15.16:0/2563533648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75001c070 con 0x7fb75c068220 2024-04-05T23:35:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.750+0000 7fb75a000700 1 -- 172.21.15.16:0/2563533648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb75000fd40 con 0x7fb75c068220 2024-04-05T23:35:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.750+0000 7fb75a000700 1 -- 172.21.15.16:0/2563533648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750022b80 con 0x7fb75c068220 2024-04-05T23:35:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.750+0000 7fb763e00700 1 -- 172.21.15.16:0/2563533648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c1a60e0 con 0x7fb75c068220 2024-04-05T23:35:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.750+0000 7fb763e00700 1 -- 172.21.15.16:0/2563533648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c1a6500 con 0x7fb75c068220 2024-04-05T23:35:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.753+0000 7fb75a000700 1 -- 172.21.15.16:0/2563533648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fb750020070 con 0x7fb75c068220 2024-04-05T23:35:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.754+0000 7fb777945b80 1 Processor -- start 2024-04-05T23:35:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.754+0000 7fb75a000700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb7440a13e0 0x7fb7440a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.754+0000 7fb75a000700 1 -- 172.21.15.16:0/2563533648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fb7500cd930 con 0x7fb75c068220 2024-04-05T23:35:24.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.754+0000 7fb777945b80 1 -- start start 2024-04-05T23:35:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.754+0000 7fb760a00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb7440a13e0 0x7fb7440a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb777945b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc536560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb777945b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 0x555ccc530760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb777945b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc530ca0 0x555ccc53cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb777945b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc496be0 con 0x555ccc536180 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb777945b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc487990 con 0x555ccc5302e0 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb777945b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc485350 con 0x555ccc530ca0 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb761400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc536560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb761400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc536560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36532/0 (socket says 172.21.15.16:36532) 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb761400700 1 -- 172.21.15.16:0/1858238326 learned_addr learned my addr 172.21.15.16:0/1858238326 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb761e00700 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc530ca0 0x555ccc53cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb760a00700 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 0x555ccc530760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb761400700 1 -- 172.21.15.16:0/1858238326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc530ca0 msgr2=0x555ccc53cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb761400700 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc530ca0 0x555ccc53cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb761400700 1 -- 172.21.15.16:0/1858238326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 msgr2=0x555ccc530760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb761400700 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 0x555ccc530760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb761400700 1 -- 172.21.15.16:0/1858238326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ccc3023b0 con 0x555ccc536180 2024-04-05T23:35:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.755+0000 7fb760a00700 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 0x555ccc530760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.756+0000 7fb760a00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb7440a13e0 0x7fb7440a38a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb75c19aaa0 tx=0x7fb748011440 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.756+0000 7fb761400700 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc536560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb7500a2960 tx=0x7fb75009a5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7303137dd2900e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.756+0000 7fb74e600700 1 -- 172.21.15.16:0/1858238326 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75009aaa0 con 0x555ccc536180 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.756+0000 7fb74e600700 1 -- 172.21.15.16:0/1858238326 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb75009e8d0 con 0x555ccc536180 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.756+0000 7fb74e600700 1 -- 172.21.15.16:0/1858238326 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750099f90 con 0x555ccc536180 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 -- 172.21.15.16:0/1858238326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 msgr2=0x555ccc536560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc536560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb7500a2960 tx=0x7fb75009a5e0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 -- 172.21.15.16:0/1858238326 shutdown_connections 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc536560 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc530ca0 0x555ccc53cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 --2- 172.21.15.16:0/1858238326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 0x555ccc530760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 -- 172.21.15.16:0/1858238326 >> 172.21.15.16:0/1858238326 conn(0x555ccc4a4220 msgr2=0x555ccc52b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 -- 172.21.15.16:0/1858238326 shutdown_connections 2024-04-05T23:35:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 -- 172.21.15.16:0/1858238326 wait complete. 2024-04-05T23:35:24.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.757+0000 7fb777945b80 1 -- start start 2024-04-05T23:35:24.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.758+0000 7fb777945b80 1 Processor -- start 2024-04-05T23:35:24.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.758+0000 7fb777945b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc5302e0 0x555ccc53fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.758+0000 7fb777945b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc5401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.758+0000 7fb777945b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc539fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.758+0000 7fb777945b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc480c40 con 0x555ccc536180 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.758+0000 7fb777945b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc41cd80 con 0x555ccc530ca0 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.758+0000 7fb777945b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc41ca20 con 0x555ccc5302e0 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.758+0000 7fb777945b80 10 client.0 _ll_register_callbacks cb 0x555ccc2883a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.758+0000 7fb760a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc5401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb761e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc539fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb760a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc5401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37710/0 (socket says 172.21.15.16:37710) 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb761400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc5302e0 0x555ccc53fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb760a00700 1 -- 172.21.15.16:0/4161163133 learned_addr learned my addr 172.21.15.16:0/4161163133 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb760a00700 1 -- 172.21.15.16:0/4161163133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc5302e0 msgr2=0x555ccc53fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb760a00700 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc5302e0 0x555ccc53fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb760a00700 1 -- 172.21.15.16:0/4161163133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 msgr2=0x555ccc539fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb760a00700 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc539fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb760a00700 1 -- 172.21.15.16:0/4161163133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c1a6500 con 0x555ccc530ca0 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb761400700 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc5302e0 0x555ccc53fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb760a00700 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc5401b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb74800f590 tx=0x7fb7480047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7a4437d840773ed server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb742a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.759+0000 7fb742a00700 1 -- 172.21.15.16:0/4161163133 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb748011a10 con 0x555ccc530ca0 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb777945b80 1 -- 172.21.15.16:0/4161163133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ccc523490 con 0x555ccc530ca0 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb742a00700 1 -- 172.21.15.16:0/4161163133 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fb748011b70 con 0x555ccc530ca0 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb742a00700 1 -- 172.21.15.16:0/4161163133 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb748011d30 con 0x555ccc530ca0 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb777945b80 10 client.25907 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb777945b80 1 -- 172.21.15.16:0/4161163133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x555ccc2aeb60 con 0x555ccc530ca0 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb742a00700 1 -- 172.21.15.16:0/4161163133 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb748033080 con 0x555ccc530ca0 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb742a00700 1 -- 172.21.15.16:0/4161163133 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb7480275b0 con 0x555ccc530ca0 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb777945b80 10 client.25907 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb777945b80 10 client.25907 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb777945b80 20 client.25907 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb777945b80 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x555ccc54a2a0 0x555ccc54c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb777945b80 4 client.25907 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.760+0000 7fb777945b80 1 -- 172.21.15.16:0/4161163133 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ccc479700 con 0x555ccc54a2a0 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.761+0000 7fb761400700 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x555ccc54a2a0 0x555ccc54c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.761+0000 7fb742a00700 10 client.25907 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.761+0000 7fb761400700 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x555ccc54a2a0 0x555ccc54c6c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e669e031c857e14a server_cookie=99f387851ec0e68d in_seq=0 out_seq=0 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb742a00700 1 -- 172.21.15.16:0/4161163133 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555ccc479700 con 0x555ccc54a2a0 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb742a00700 10 client.25907 handle_command_reply: tid=0 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 client.25907 shutdown 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 -- 172.21.15.16:0/4161163133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 msgr2=0x555ccc5401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc5401b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb74800f590 tx=0x7fb7480047c0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 -- 172.21.15.16:0/4161163133 shutdown_connections 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x555ccc54a2a0 0x555ccc54c6c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc536180 0x555ccc539fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb742a00700 0 client.25907 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:35:24.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc5302e0 0x555ccc53fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 --2- 172.21.15.16:0/4161163133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc5401b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 -- 172.21.15.16:0/4161163133 >> 172.21.15.16:0/4161163133 conn(0x555ccc4a4220 msgr2=0x555ccc525460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 -- 172.21.15.16:0/4161163133 shutdown_connections 2024-04-05T23:35:24.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.762+0000 7fb777945b80 1 -- 172.21.15.16:0/4161163133 wait complete. 2024-04-05T23:35:24.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.763+0000 7fb777945b80 20 client.25907 trim_cache size 0 max 16384 2024-04-05T23:35:24.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.776+0000 7fb777945b80 1 Processor -- start 2024-04-05T23:35:24.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.776+0000 7fb777945b80 1 -- start start 2024-04-05T23:35:24.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb777945b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc5302e0 0x555ccc538f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb777945b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc535950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb777945b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 0x555ccc525560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb777945b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc3046a0 con 0x555ccc5302e0 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb777945b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc53b590 con 0x555ccc530ca0 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb777945b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc536c20 con 0x555ccc536180 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb761400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc5302e0 0x555ccc538f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb761400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc5302e0 0x555ccc538f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36548/0 (socket says 172.21.15.16:36548) 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb761400700 1 -- 172.21.15.16:0/1180563963 learned_addr learned my addr 172.21.15.16:0/1180563963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb761e00700 1 --2- 172.21.15.16:0/1180563963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 0x555ccc525560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb760a00700 1 --2- 172.21.15.16:0/1180563963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc535950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb761400700 1 -- 172.21.15.16:0/1180563963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 msgr2=0x555ccc525560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb761400700 1 --2- 172.21.15.16:0/1180563963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 0x555ccc525560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb761400700 1 -- 172.21.15.16:0/1180563963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 msgr2=0x555ccc535950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb761400700 1 --2- 172.21.15.16:0/1180563963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc535950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.777+0000 7fb761400700 1 -- 172.21.15.16:0/1180563963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ccc446ea0 con 0x555ccc5302e0 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.778+0000 7fb761400700 1 --2- 172.21.15.16:0/1180563963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc5302e0 0x555ccc538f30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb750009fd0 tx=0x7fb750009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36380c825ad4d31 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.778+0000 7fb74f000700 1 -- 172.21.15.16:0/1180563963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7500a2f10 con 0x555ccc5302e0 2024-04-05T23:35:24.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.778+0000 7fb74f000700 1 -- 172.21.15.16:0/1180563963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb75009aaa0 con 0x555ccc5302e0 2024-04-05T23:35:24.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.778+0000 7fb74f000700 1 -- 172.21.15.16:0/1180563963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7500a2a00 con 0x555ccc5302e0 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.778+0000 7fb777945b80 1 -- 172.21.15.16:0/1180563963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc5302e0 msgr2=0x555ccc538f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.778+0000 7fb777945b80 1 --2- 172.21.15.16:0/1180563963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc5302e0 0x555ccc538f30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb750009fd0 tx=0x7fb750009310 comp rx=0 tx=0).stop 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.779+0000 7fb777945b80 1 -- 172.21.15.16:0/1180563963 shutdown_connections 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.779+0000 7fb777945b80 1 --2- 172.21.15.16:0/1180563963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc5302e0 0x555ccc538f30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.779+0000 7fb777945b80 1 --2- 172.21.15.16:0/1180563963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 0x555ccc525560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.779+0000 7fb777945b80 1 --2- 172.21.15.16:0/1180563963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc530ca0 0x555ccc535950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.779+0000 7fb777945b80 1 -- 172.21.15.16:0/1180563963 >> 172.21.15.16:0/1180563963 conn(0x555ccc4a4220 msgr2=0x555ccc535240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.779+0000 7fb777945b80 1 -- 172.21.15.16:0/1180563963 shutdown_connections 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.779+0000 7fb777945b80 1 -- 172.21.15.16:0/1180563963 wait complete. 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.779+0000 7fb777945b80 1 -- start start 2024-04-05T23:35:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.779+0000 7fb777945b80 1 Processor -- start 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb777945b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 0x555ccc546850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb777945b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc530ca0 0x555ccc543ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb777945b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 0x555ccc544420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb777945b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc544b00 con 0x555ccc530ca0 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb777945b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc59d0f0 con 0x555ccc5302e0 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb777945b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ccc59d230 con 0x555ccc536180 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb777945b80 10 client.0 _ll_register_callbacks cb 0x7fb748013e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb760a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc530ca0 0x555ccc543ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb760a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc530ca0 0x555ccc543ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36560/0 (socket says 172.21.15.16:36560) 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb760a00700 1 -- 172.21.15.16:0/2412160243 learned_addr learned my addr 172.21.15.16:0/2412160243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb761400700 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 0x555ccc546850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb761e00700 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 0x555ccc544420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb760a00700 1 -- 172.21.15.16:0/2412160243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 msgr2=0x555ccc544420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb760a00700 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 0x555ccc544420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb760a00700 1 -- 172.21.15.16:0/2412160243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 msgr2=0x555ccc546850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb760a00700 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 0x555ccc546850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.780+0000 7fb760a00700 1 -- 172.21.15.16:0/2412160243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ccc3023b0 con 0x555ccc530ca0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.781+0000 7fb760a00700 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc530ca0 0x555ccc543ce0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb7480279e0 tx=0x7fb748027650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288872d08a328ea5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.781+0000 7fb742a00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.781+0000 7fb742a00700 1 -- 172.21.15.16:0/2412160243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb748027d90 con 0x555ccc530ca0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.781+0000 7fb742a00700 1 -- 172.21.15.16:0/2412160243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fb748011840 con 0x555ccc530ca0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.781+0000 7fb777945b80 1 -- 172.21.15.16:0/2412160243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb75009e8d0 con 0x555ccc530ca0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.781+0000 7fb742a00700 1 -- 172.21.15.16:0/2412160243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb748011a00 con 0x555ccc530ca0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.781+0000 7fb777945b80 10 client.15879 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.781+0000 7fb777945b80 1 -- 172.21.15.16:0/2412160243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x555ccc306cd0 con 0x555ccc530ca0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.782+0000 7fb742a00700 1 -- 172.21.15.16:0/2412160243 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb75009e8d0 con 0x555ccc530ca0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.782+0000 7fb742a00700 1 -- 172.21.15.16:0/2412160243 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb74801d790 con 0x555ccc530ca0 2024-04-05T23:35:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.782+0000 7fb777945b80 10 client.15879 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.782+0000 7fb777945b80 10 client.15879 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.782+0000 7fb777945b80 20 client.15879 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.782+0000 7fb777945b80 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x555ccc59e8f0 0x555ccc5a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.782+0000 7fb777945b80 4 client.15879 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.782+0000 7fb777945b80 1 -- 172.21.15.16:0/2412160243 --> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555ccc2cfcf0 con 0x555ccc59e8f0 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.782+0000 7fb761400700 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x555ccc59e8f0 0x555ccc5a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb761400700 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x555ccc59e8f0 0x555ccc5a0ce0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f77241ad7ea6949b server_cookie=15c07ed7ca8d8a67 in_seq=0 out_seq=0 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb742a00700 10 client.15879 ms_handle_connect on v2:172.21.15.130:6832/477825490 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb742a00700 1 -- 172.21.15.16:0/2412160243 <== mds.4 v2:172.21.15.130:6832/477825490 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555ccc2cfcf0 con 0x555ccc59e8f0 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb742a00700 10 client.15879 handle_command_reply: tid=0 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 client.15879 shutdown 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 -- 172.21.15.16:0/2412160243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc530ca0 msgr2=0x555ccc543ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc530ca0 0x555ccc543ce0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb7480279e0 tx=0x7fb748027650 comp rx=0 tx=0).stop 2024-04-05T23:35:24.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 -- 172.21.15.16:0/2412160243 shutdown_connections 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ccc536180 0x555ccc544420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ccc5302e0 0x555ccc546850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.130:6832/477825490,v1:172.21.15.130:6833/477825490] conn(0x555ccc59e8f0 0x555ccc5a0ce0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 --2- 172.21.15.16:0/2412160243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ccc530ca0 0x555ccc543ce0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 -- 172.21.15.16:0/2412160243 >> 172.21.15.16:0/2412160243 conn(0x555ccc4a4220 msgr2=0x555ccc53f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb742a00700 0 client.15879 ms_handle_reset on v2:172.21.15.130:6832/477825490 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 -- 172.21.15.16:0/2412160243 shutdown_connections 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 1 -- 172.21.15.16:0/2412160243 wait complete. 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.783+0000 7fb777945b80 20 client.15879 trim_cache size 0 max 16384 2024-04-05T23:35:24.780 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:35:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 -- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb7440a13e0 msgr2=0x7fb7440a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb7440a13e0 0x7fb7440a38a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb75c19aaa0 tx=0x7fb748011440 comp rx=0 tx=0).stop 2024-04-05T23:35:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 -- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 msgr2=0x7fb75c199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 0x7fb75c199550 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb75000b580 tx=0x7fb75000bfd0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 -- 172.21.15.16:0/2563533648 shutdown_connections 2024-04-05T23:35:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb7440a13e0 0x7fb7440a38a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb75c068220 0x7fb75c199550 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb75c19de30 0x7fb75c19e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 --2- 172.21.15.16:0/2563533648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb75c1061d0 0x7fb75c199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 -- 172.21.15.16:0/2563533648 >> 172.21.15.16:0/2563533648 conn(0x7fb75c076e80 msgr2=0x7fb75c0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 -- 172.21.15.16:0/2563533648 shutdown_connections 2024-04-05T23:35:24.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:24.784+0000 7fb763e00700 1 -- 172.21.15.16:0/2563533648 wait complete. 2024-04-05T23:35:24.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-05T23:35:24.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22304 3.10 MB/sec warmup 44 sec latency 4057.364 ms 2024-04-05T23:35:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.005+0000 7fd90da00700 1 -- 172.21.15.16:0/2289764901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908101a50 msgr2=0x7fd908105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.005+0000 7fd90da00700 1 --2- 172.21.15.16:0/2289764901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908101a50 0x7fd908105a90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009990 tx=0x7fd8f4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.005+0000 7fd90da00700 1 -- 172.21.15.16:0/2289764901 shutdown_connections 2024-04-05T23:35:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.005+0000 7fd90da00700 1 --2- 172.21.15.16:0/2289764901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9081061c0 0x7fd90810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.005+0000 7fd90da00700 1 --2- 172.21.15.16:0/2289764901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9081010a0 0x7fd908101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.005+0000 7fd90da00700 1 --2- 172.21.15.16:0/2289764901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd908101a50 0x7fd908105a90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.005+0000 7fd90da00700 1 -- 172.21.15.16:0/2289764901 >> 172.21.15.16:0/2289764901 conn(0x7fd9080fc900 msgr2=0x7fd9080fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.005+0000 7fd90da00700 1 -- 172.21.15.16:0/2289764901 shutdown_connections 2024-04-05T23:35:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.005+0000 7fd90da00700 1 -- 172.21.15.16:0/2289764901 wait complete. 2024-04-05T23:35:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.006+0000 7fd90da00700 1 Processor -- start 2024-04-05T23:35:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd90da00700 1 -- start start 2024-04-05T23:35:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd90da00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9081010a0 0x7fd908199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd90da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9081061c0 0x7fd908199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd90da00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90819de00 0x7fd90819e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd90da00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90819e880 con 0x7fd9081010a0 2024-04-05T23:35:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd90da00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90819e9f0 con 0x7fd9081061c0 2024-04-05T23:35:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd90da00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90819ebb0 con 0x7fd90819de00 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd906a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9081010a0 0x7fd908199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd906000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9081061c0 0x7fd908199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd906a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9081010a0 0x7fd908199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36588/0 (socket says 172.21.15.16:36588) 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd906000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9081061c0 0x7fd908199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37752/0 (socket says 172.21.15.16:37752) 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd906a00700 1 -- 172.21.15.16:0/3357772534 learned_addr learned my addr 172.21.15.16:0/3357772534 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.007+0000 7fd907400700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90819de00 0x7fd90819e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd906a00700 1 -- 172.21.15.16:0/3357772534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90819de00 msgr2=0x7fd90819e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd906a00700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90819de00 0x7fd90819e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd906a00700 1 -- 172.21.15.16:0/3357772534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9081061c0 msgr2=0x7fd908199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd906a00700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9081061c0 0x7fd908199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd906a00700 1 -- 172.21.15.16:0/3357772534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f40095f0 con 0x7fd9081010a0 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd906000700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9081061c0 0x7fd908199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd906a00700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9081010a0 0x7fd908199520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800e9a0 tx=0x7fd8f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd8ff400700 1 -- 172.21.15.16:0/3357772534 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8009790 con 0x7fd9081010a0 2024-04-05T23:35:25.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd90da00700 1 -- 172.21.15.16:0/3357772534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9081a6110 con 0x7fd9081010a0 2024-04-05T23:35:25.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd8ff400700 1 -- 172.21.15.16:0/3357772534 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8f8010430 con 0x7fd9081010a0 2024-04-05T23:35:25.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.008+0000 7fd8ff400700 1 -- 172.21.15.16:0/3357772534 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8018620 con 0x7fd9081010a0 2024-04-05T23:35:25.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.009+0000 7fd90da00700 1 -- 172.21.15.16:0/3357772534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9081a65f0 con 0x7fd9081010a0 2024-04-05T23:35:25.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.013+0000 7fd8ff400700 1 -- 172.21.15.16:0/3357772534 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fd8f80105a0 con 0x7fd9081010a0 2024-04-05T23:35:25.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.014+0000 7fd9214adb80 1 Processor -- start 2024-04-05T23:35:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.014+0000 7fd9214adb80 1 -- start start 2024-04-05T23:35:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.014+0000 7fd8ff400700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8ec0a1290 0x7fd8ec0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.014+0000 7fd8ff400700 1 -- 172.21.15.16:0/3357772534 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd8f8014070 con 0x7fd9081010a0 2024-04-05T23:35:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 0x5583e3d69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d632e0 0x5583e3d63760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 0x5583e3d6ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd9214adb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3cc9be0 con 0x5583e3d632e0 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd9214adb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3cba990 con 0x5583e3d69180 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd9214adb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3cb8350 con 0x5583e3d63ca0 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd907400700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8ec0a1290 0x7fd8ec0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d632e0 0x5583e3d63760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d632e0 0x5583e3d63760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36604/0 (socket says 172.21.15.16:36604) 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906000700 1 -- 172.21.15.16:0/233316156 learned_addr learned my addr 172.21.15.16:0/233316156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906a00700 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 0x5583e3d69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd907400700 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 0x5583e3d6ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906000700 1 -- 172.21.15.16:0/233316156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 msgr2=0x5583e3d6ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906000700 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 0x5583e3d6ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906000700 1 -- 172.21.15.16:0/233316156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 msgr2=0x5583e3d69560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906000700 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 0x5583e3d69560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906000700 1 -- 172.21.15.16:0/233316156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583e3b353b0 con 0x5583e3d632e0 2024-04-05T23:35:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd906a00700 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 0x5583e3d69560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.015+0000 7fd907400700 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 0x5583e3d6ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:35:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.016+0000 7fd907400700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8ec0a1290 0x7fd8ec0a3750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd900009ec0 tx=0x7fd900009920 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.016+0000 7fd906000700 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d632e0 0x5583e3d63760 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009990 tx=0x7fd8f4005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1936380a0893e2a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.016+0000 7fd8f3e00700 1 -- 172.21.15.16:0/233316156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f401d070 con 0x5583e3d632e0 2024-04-05T23:35:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.016+0000 7fd8f3e00700 1 -- 172.21.15.16:0/233316156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8f401e040 con 0x5583e3d632e0 2024-04-05T23:35:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.016+0000 7fd8f3e00700 1 -- 172.21.15.16:0/233316156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4004bf0 con 0x5583e3d632e0 2024-04-05T23:35:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 -- 172.21.15.16:0/233316156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d632e0 msgr2=0x5583e3d63760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d632e0 0x5583e3d63760 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009990 tx=0x7fd8f4005f00 comp rx=0 tx=0).stop 2024-04-05T23:35:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 -- 172.21.15.16:0/233316156 shutdown_connections 2024-04-05T23:35:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 0x5583e3d6ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 0x5583e3d69560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 --2- 172.21.15.16:0/233316156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d632e0 0x5583e3d63760 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 -- 172.21.15.16:0/233316156 >> 172.21.15.16:0/233316156 conn(0x5583e3cd7220 msgr2=0x5583e3d5e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 -- 172.21.15.16:0/233316156 shutdown_connections 2024-04-05T23:35:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 -- 172.21.15.16:0/233316156 wait complete. 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.017+0000 7fd9214adb80 1 -- start start 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd9214adb80 1 Processor -- start 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d632e0 0x5583e3d6feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 0x5583e3d6d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd9214adb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3cb3c40 con 0x5583e3d69180 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd9214adb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3c4fd80 con 0x5583e3d632e0 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd9214adb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3c4fa20 con 0x5583e3d63ca0 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd9214adb80 10 client.0 _ll_register_callbacks cb 0x5583e3abb3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd907400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd907400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36616/0 (socket says 172.21.15.16:36616) 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd907400700 1 -- 172.21.15.16:0/3161270558 learned_addr learned my addr 172.21.15.16:0/3161270558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd906a00700 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d632e0 0x5583e3d6feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.018+0000 7fd906000700 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 0x5583e3d6d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd907400700 1 -- 172.21.15.16:0/3161270558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 msgr2=0x5583e3d6d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd907400700 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 0x5583e3d6d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd907400700 1 -- 172.21.15.16:0/3161270558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d632e0 msgr2=0x5583e3d6feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd907400700 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d632e0 0x5583e3d6feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd907400700 1 -- 172.21.15.16:0/3161270558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9081a65f0 con 0x5583e3d69180 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd907400700 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6da00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd900019ed0 tx=0x7fd900004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d7ed50ecdbca05 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd8f0c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3161270558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000e040 con 0x5583e3d69180 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd9214adb80 1 -- 172.21.15.16:0/3161270558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583e3d56490 con 0x5583e3d69180 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.019+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3161270558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd9000233f0 con 0x5583e3d69180 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3161270558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd900017e60 con 0x5583e3d69180 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd9214adb80 10 client.15891 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd9214adb80 1 -- 172.21.15.16:0/3161270558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7fd90810e5b0 con 0x5583e3d69180 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3161270558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5583e3d56490 con 0x5583e3d69180 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3161270558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd900023ca0 con 0x5583e3d69180 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd9214adb80 10 client.15891 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd9214adb80 10 client.15891 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.d 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd9214adb80 20 client.15891 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x5583e3d7d2f0 0x5583e3d7f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd9214adb80 4 client.15891 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd9214adb80 1 -- 172.21.15.16:0/3161270558 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583e3cac700 con 0x5583e3d7d2f0 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.020+0000 7fd906a00700 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x5583e3d7d2f0 0x5583e3d7f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.021+0000 7fd906a00700 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x5583e3d7d2f0 0x5583e3d7f710 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=50e7d68205c9e12a server_cookie=af0d7c5d008c4aa2 in_seq=0 out_seq=0 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.021+0000 7fd8f0c00700 10 client.15891 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.021+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3161270558 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5583e3cac700 con 0x5583e3d7d2f0 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.021+0000 7fd8f0c00700 10 client.15891 handle_command_reply: tid=0 2024-04-05T23:35:25.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 client.15891 shutdown 2024-04-05T23:35:25.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 -- 172.21.15.16:0/3161270558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 msgr2=0x5583e3d6da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6da00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd900019ed0 tx=0x7fd900004b40 comp rx=0 tx=0).stop 2024-04-05T23:35:25.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 -- 172.21.15.16:0/3161270558 shutdown_connections 2024-04-05T23:35:25.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x5583e3d7d2f0 0x5583e3d7f710 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6da00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d632e0 0x5583e3d6feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3161270558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d63ca0 0x5583e3d6d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd8f0c00700 0 client.15891 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:35:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 -- 172.21.15.16:0/3161270558 >> 172.21.15.16:0/3161270558 conn(0x5583e3cd7220 msgr2=0x5583e3d58460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 -- 172.21.15.16:0/3161270558 shutdown_connections 2024-04-05T23:35:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 1 -- 172.21.15.16:0/3161270558 wait complete. 2024-04-05T23:35:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.022+0000 7fd9214adb80 20 client.15891 trim_cache size 0 max 16384 2024-04-05T23:35:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.036+0000 7fd9214adb80 1 Processor -- start 2024-04-05T23:35:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.036+0000 7fd9214adb80 1 -- start start 2024-04-05T23:35:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.036+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 0x5583e3d58560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.036+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.036+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 0x5583e3d6feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.036+0000 7fd9214adb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3b376a0 con 0x5583e3d63ca0 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.036+0000 7fd9214adb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3d58aa0 con 0x5583e3d69180 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.036+0000 7fd9214adb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3d6b810 con 0x5583e3d632e0 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36630/0 (socket says 172.21.15.16:36630) 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906000700 1 -- 172.21.15.16:0/3126733212 learned_addr learned my addr 172.21.15.16:0/3126733212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906a00700 1 --2- 172.21.15.16:0/3126733212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 0x5583e3d58560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd907400700 1 --2- 172.21.15.16:0/3126733212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 0x5583e3d6feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906000700 1 -- 172.21.15.16:0/3126733212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 msgr2=0x5583e3d58560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906000700 1 --2- 172.21.15.16:0/3126733212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 0x5583e3d58560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906000700 1 -- 172.21.15.16:0/3126733212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 msgr2=0x5583e3d6feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906000700 1 --2- 172.21.15.16:0/3126733212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 0x5583e3d6feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906000700 1 -- 172.21.15.16:0/3126733212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583e3ae1b60 con 0x5583e3d63ca0 2024-04-05T23:35:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd906000700 1 --2- 172.21.15.16:0/3126733212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f970 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4004760 tx=0x7fd8f40046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae463e04d20d3c66 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.037+0000 7fd8fcc00700 1 -- 172.21.15.16:0/3126733212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f401d040 con 0x5583e3d63ca0 2024-04-05T23:35:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd8fcc00700 1 -- 172.21.15.16:0/3126733212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8f4004500 con 0x5583e3d63ca0 2024-04-05T23:35:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd8fcc00700 1 -- 172.21.15.16:0/3126733212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400f460 con 0x5583e3d63ca0 2024-04-05T23:35:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd9214adb80 1 -- 172.21.15.16:0/3126733212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d63ca0 msgr2=0x5583e3d6f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3126733212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f970 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4004760 tx=0x7fd8f40046f0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd9214adb80 1 -- 172.21.15.16:0/3126733212 shutdown_connections 2024-04-05T23:35:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3126733212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d69180 0x5583e3d6feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3126733212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 0x5583e3d58560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3126733212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f970 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd9214adb80 1 -- 172.21.15.16:0/3126733212 >> 172.21.15.16:0/3126733212 conn(0x5583e3cd7220 msgr2=0x5583e3d67360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd9214adb80 1 -- 172.21.15.16:0/3126733212 shutdown_connections 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.038+0000 7fd9214adb80 1 -- 172.21.15.16:0/3126733212 wait complete. 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.039+0000 7fd9214adb80 1 -- start start 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.039+0000 7fd9214adb80 1 Processor -- start 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.039+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 0x5583e3d74510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.039+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.039+0000 7fd9214adb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.039+0000 7fd9214adb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3d70270 con 0x5583e3d69180 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.039+0000 7fd9214adb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3d64b30 con 0x5583e3d63ca0 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.039+0000 7fd9214adb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583e3d64cf0 con 0x5583e3d632e0 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.039+0000 7fd9214adb80 10 client.0 _ll_register_callbacks cb 0x7fd9000198a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd907400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd907400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36636/0 (socket says 172.21.15.16:36636) 2024-04-05T23:35:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd907400700 1 -- 172.21.15.16:0/3253690953 learned_addr learned my addr 172.21.15.16:0/3253690953 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd906a00700 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 0x5583e3d74510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd906000700 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd907400700 1 -- 172.21.15.16:0/3253690953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 msgr2=0x5583e3d74510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd907400700 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 0x5583e3d74510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd907400700 1 -- 172.21.15.16:0/3253690953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d63ca0 msgr2=0x5583e3d6f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd907400700 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd907400700 1 -- 172.21.15.16:0/3253690953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583e3b353b0 con 0x5583e3d69180 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd906a00700 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 0x5583e3d74510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd907400700 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6fb30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd9000060b0 tx=0x7fd900017f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f64a675945e380 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.040+0000 7fd8f0c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3253690953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900023c30 con 0x5583e3d69180 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3253690953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fd900023d90 con 0x5583e3d69180 2024-04-05T23:35:25.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd9214adb80 1 -- 172.21.15.16:0/3253690953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8f401e040 con 0x5583e3d69180 2024-04-05T23:35:25.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3253690953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9000363f0 con 0x5583e3d69180 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd9214adb80 10 client.15897 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd9214adb80 1 -- 172.21.15.16:0/3253690953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x5583e3c79ea0 con 0x5583e3d69180 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3253690953 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8f401e040 con 0x5583e3d69180 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3253690953 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd90002c3f0 con 0x5583e3d69180 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd9214adb80 10 client.15897 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd9214adb80 10 client.15897 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.d 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd9214adb80 20 client.15897 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x5583e3dd1b20 0x5583e3dd3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd9214adb80 4 client.15897 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.041+0000 7fd9214adb80 1 -- 172.21.15.16:0/3253690953 --> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5583e3b02cf0 con 0x5583e3dd1b20 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.042+0000 7fd906a00700 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x5583e3dd1b20 0x5583e3dd3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.042+0000 7fd906a00700 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x5583e3dd1b20 0x5583e3dd3f40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ab50daa865462b20 server_cookie=7896a87373c18fd3 in_seq=0 out_seq=0 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.042+0000 7fd8f0c00700 10 client.15897 ms_handle_connect on v2:172.21.15.16:6836/2898043660 2024-04-05T23:35:25.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.042+0000 7fd8f0c00700 1 -- 172.21.15.16:0/3253690953 <== mds.3 v2:172.21.15.16:6836/2898043660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5583e3b02cf0 con 0x5583e3dd1b20 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.042+0000 7fd8f0c00700 10 client.15897 handle_command_reply: tid=0 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.042+0000 7fd9214adb80 1 client.15897 shutdown 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 -- 172.21.15.16:0/3253690953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 msgr2=0x5583e3d6fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6fb30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd9000060b0 tx=0x7fd900017f90 comp rx=0 tx=0).stop 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 -- 172.21.15.16:0/3253690953 shutdown_connections 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.16:6836/2898043660,v1:172.21.15.16:6837/2898043660] conn(0x5583e3dd1b20 0x5583e3dd3f40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5583e3d69180 0x5583e3d6fb30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583e3d632e0 0x5583e3d74510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 --2- 172.21.15.16:0/3253690953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5583e3d63ca0 0x5583e3d6f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd8f0c00700 0 client.15897 ms_handle_reset on v2:172.21.15.16:6836/2898043660 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 -- 172.21.15.16:0/3253690953 >> 172.21.15.16:0/3253690953 conn(0x5583e3cd7220 msgr2=0x5583e3d5e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 -- 172.21.15.16:0/3253690953 shutdown_connections 2024-04-05T23:35:25.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 1 -- 172.21.15.16:0/3253690953 wait complete. 2024-04-05T23:35:25.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd9214adb80 20 client.15897 trim_cache size 0 max 16384 2024-04-05T23:35:25.039 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:35:25.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd90da00700 1 -- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8ec0a1290 msgr2=0x7fd8ec0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd90da00700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8ec0a1290 0x7fd8ec0a3750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd900009ec0 tx=0x7fd900009920 comp rx=0 tx=0).stop 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd90da00700 1 -- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9081010a0 msgr2=0x7fd908199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.043+0000 7fd90da00700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9081010a0 0x7fd908199520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800e9a0 tx=0x7fd8f800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.044+0000 7fd90da00700 1 -- 172.21.15.16:0/3357772534 shutdown_connections 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.044+0000 7fd90da00700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8ec0a1290 0x7fd8ec0a3750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.044+0000 7fd90da00700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9081010a0 0x7fd908199520 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.044+0000 7fd90da00700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90819de00 0x7fd90819e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.044+0000 7fd90da00700 1 --2- 172.21.15.16:0/3357772534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9081061c0 0x7fd908199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.044+0000 7fd90da00700 1 -- 172.21.15.16:0/3357772534 >> 172.21.15.16:0/3357772534 conn(0x7fd9080fc900 msgr2=0x7fd9080fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.044+0000 7fd90da00700 1 -- 172.21.15.16:0/3357772534 shutdown_connections 2024-04-05T23:35:25.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.044+0000 7fd90da00700 1 -- 172.21.15.16:0/3357772534 wait complete. 2024-04-05T23:35:25.051 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:24 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2037454867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:35:25.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-05T23:35:25.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:24 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2037454867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:35:25.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.282+0000 7f7b01800700 1 -- 172.21.15.16:0/3279655179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc1010a0 msgr2=0x7f7afc101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.282+0000 7f7b01800700 1 --2- 172.21.15.16:0/3279655179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc1010a0 0x7f7afc101480 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009990 tx=0x7f7aec009ca0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.282+0000 7f7b01800700 1 -- 172.21.15.16:0/3279655179 shutdown_connections 2024-04-05T23:35:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.282+0000 7f7b01800700 1 --2- 172.21.15.16:0/3279655179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc1010a0 0x7f7afc101480 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.282+0000 7f7b01800700 1 --2- 172.21.15.16:0/3279655179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7afc1061c0 0x7f7afc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.282+0000 7f7b01800700 1 --2- 172.21.15.16:0/3279655179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7afc101a50 0x7f7afc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.282+0000 7f7b01800700 1 -- 172.21.15.16:0/3279655179 >> 172.21.15.16:0/3279655179 conn(0x7f7afc0fc900 msgr2=0x7f7afc0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.282+0000 7f7b01800700 1 -- 172.21.15.16:0/3279655179 shutdown_connections 2024-04-05T23:35:25.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.282+0000 7f7b01800700 1 -- 172.21.15.16:0/3279655179 wait complete. 2024-04-05T23:35:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.283+0000 7f7b01800700 1 Processor -- start 2024-04-05T23:35:25.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7b01800700 1 -- start start 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7b01800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7afc1010a0 0x7f7afc197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7b01800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7afc1061c0 0x7f7afc1978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7b01800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc19bc50 0x7f7afc19c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7b01800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc19c6d0 con 0x7f7afc19bc50 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7b01800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc19c840 con 0x7f7afc1010a0 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7b01800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc19ca00 con 0x7f7afc1061c0 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7afb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc19bc50 0x7f7afc19c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7afaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7afc1010a0 0x7f7afc197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7afb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc19bc50 0x7f7afc19c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36658/0 (socket says 172.21.15.16:36658) 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7afaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7afc1010a0 0x7f7afc197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37800/0 (socket says 172.21.15.16:37800) 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7afb400700 1 -- 172.21.15.16:0/835016346 learned_addr learned my addr 172.21.15.16:0/835016346 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.284+0000 7f7af3e00700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7afc1061c0 0x7f7afc1978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7afb400700 1 -- 172.21.15.16:0/835016346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7afc1061c0 msgr2=0x7f7afc1978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7afb400700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7afc1061c0 0x7f7afc1978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7afb400700 1 -- 172.21.15.16:0/835016346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7afc1010a0 msgr2=0x7f7afc197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7afb400700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7afc1010a0 0x7f7afc197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7afb400700 1 -- 172.21.15.16:0/835016346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec0095f0 con 0x7f7afc19bc50 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7afaa00700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7afc1010a0 0x7f7afc197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7af3e00700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7afc1061c0 0x7f7afc1978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7afb400700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc19bc50 0x7f7afc19c0d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7af400ef30 tx=0x7f7af400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7af3400700 1 -- 172.21.15.16:0/835016346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4018070 con 0x7f7afc19bc50 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7af3400700 1 -- 172.21.15.16:0/835016346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7af400f040 con 0x7f7afc19bc50 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7b01800700 1 -- 172.21.15.16:0/835016346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc1a3f00 con 0x7f7afc19bc50 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.285+0000 7f7af3400700 1 -- 172.21.15.16:0/835016346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af400cd00 con 0x7f7afc19bc50 2024-04-05T23:35:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.286+0000 7f7b01800700 1 -- 172.21.15.16:0/835016346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc1a43e0 con 0x7f7afc19bc50 2024-04-05T23:35:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.289+0000 7f7af3400700 1 -- 172.21.15.16:0/835016346 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f7af4003a00 con 0x7f7afc19bc50 2024-04-05T23:35:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.290+0000 7f7b15419b80 1 Processor -- start 2024-04-05T23:35:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.290+0000 7f7af3400700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ae00a1290 0x7f7ae00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.290+0000 7f7b15419b80 1 -- start start 2024-04-05T23:35:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.290+0000 7f7af3400700 1 -- 172.21.15.16:0/835016346 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f7af40cc7d0 con 0x7f7afc19bc50 2024-04-05T23:35:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.290+0000 7f7af3e00700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ae00a1290 0x7f7ae00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.290+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b7250 0x5635af7b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.290+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.290+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7c5f40 0x5635af7c8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7b15419b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af719be0 con 0x5635af7b7250 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7b15419b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af70a990 con 0x5635af7c5f40 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7b15419b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af708350 con 0x5635af7b3820 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7afaa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b7250 0x5635af7b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7afaa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b7250 0x5635af7b32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36672/0 (socket says 172.21.15.16:36672) 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7afaa00700 1 -- 172.21.15.16:0/2147337270 learned_addr learned my addr 172.21.15.16:0/2147337270 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7af3e00700 1 --2- 172.21.15.16:0/2147337270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7af3e00700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ae00a1290 0x7f7ae00a3750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7afc1988c0 tx=0x7f7ae40073d0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7afb400700 1 --2- 172.21.15.16:0/2147337270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7c5f40 0x5635af7c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7afaa00700 1 -- 172.21.15.16:0/2147337270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 msgr2=0x5635af7b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7afaa00700 1 --2- 172.21.15.16:0/2147337270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7afaa00700 1 -- 172.21.15.16:0/2147337270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7c5f40 msgr2=0x5635af7c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7afaa00700 1 --2- 172.21.15.16:0/2147337270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7c5f40 0x5635af7c8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.291+0000 7f7afaa00700 1 -- 172.21.15.16:0/2147337270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635af5853b0 con 0x5635af7b7250 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.292+0000 7f7afaa00700 1 --2- 172.21.15.16:0/2147337270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b7250 0x5635af7b32e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00b5e0 tx=0x7f7aec00b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8436917ca43b16a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.292+0000 7f7aebe00700 1 -- 172.21.15.16:0/2147337270 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec01d070 con 0x5635af7b7250 2024-04-05T23:35:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.292+0000 7f7aebe00700 1 -- 172.21.15.16:0/2147337270 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7aec01e040 con 0x5635af7b7250 2024-04-05T23:35:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.292+0000 7f7aebe00700 1 -- 172.21.15.16:0/2147337270 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec005400 con 0x5635af7b7250 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.292+0000 7f7b15419b80 1 -- 172.21.15.16:0/2147337270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b7250 msgr2=0x5635af7b32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.292+0000 7f7b15419b80 1 --2- 172.21.15.16:0/2147337270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b7250 0x5635af7b32e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00b5e0 tx=0x7f7aec00b610 comp rx=0 tx=0).stop 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.293+0000 7f7b15419b80 1 -- 172.21.15.16:0/2147337270 shutdown_connections 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.293+0000 7f7b15419b80 1 --2- 172.21.15.16:0/2147337270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b7250 0x5635af7b32e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.293+0000 7f7b15419b80 1 --2- 172.21.15.16:0/2147337270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7c5f40 0x5635af7c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.293+0000 7f7b15419b80 1 --2- 172.21.15.16:0/2147337270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.293+0000 7f7b15419b80 1 -- 172.21.15.16:0/2147337270 >> 172.21.15.16:0/2147337270 conn(0x5635af727220 msgr2=0x5635af7ae570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.293+0000 7f7b15419b80 1 -- 172.21.15.16:0/2147337270 shutdown_connections 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.293+0000 7f7b15419b80 1 -- 172.21.15.16:0/2147337270 wait complete. 2024-04-05T23:35:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.293+0000 7f7b15419b80 1 -- start start 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.294+0000 7f7b15419b80 1 Processor -- start 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.294+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b3820 0x5635af7c5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.294+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7c2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.294+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7c5f40 0x5635af7c3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.294+0000 7f7b15419b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af703c40 con 0x5635af7b3820 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.294+0000 7f7b15419b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af69fd80 con 0x5635af7b7250 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.294+0000 7f7b15419b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af69fa20 con 0x5635af7c5f40 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7b15419b80 10 client.0 _ll_register_callbacks cb 0x5635af50b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afaa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b3820 0x5635af7c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afaa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b3820 0x5635af7c5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36674/0 (socket says 172.21.15.16:36674) 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7af3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7c2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afaa00700 1 -- 172.21.15.16:0/485591426 learned_addr learned my addr 172.21.15.16:0/485591426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afb400700 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7c5f40 0x5635af7c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afaa00700 1 -- 172.21.15.16:0/485591426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7c5f40 msgr2=0x5635af7c3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afaa00700 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7c5f40 0x5635af7c3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afaa00700 1 -- 172.21.15.16:0/485591426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 msgr2=0x5635af7c2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afaa00700 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7c2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afaa00700 1 -- 172.21.15.16:0/485591426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc10e5b0 con 0x5635af7b3820 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7afaa00700 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b3820 0x5635af7c5980 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7aec000c00 tx=0x7f7aec004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bb05e3f015f86 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.295+0000 7f7ae8c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.296+0000 7f7ae8c00700 1 -- 172.21.15.16:0/485591426 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec01d030 con 0x5635af7b3820 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.296+0000 7f7ae8c00700 1 -- 172.21.15.16:0/485591426 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f7aec005410 con 0x5635af7b3820 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.296+0000 7f7b15419b80 1 -- 172.21.15.16:0/485591426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5635af7a6490 con 0x5635af7b3820 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.296+0000 7f7ae8c00700 1 -- 172.21.15.16:0/485591426 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7aec0058e0 con 0x5635af7b3820 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.296+0000 7f7b15419b80 10 client.15909 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.296+0000 7f7b15419b80 1 -- 172.21.15.16:0/485591426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x5635af531b60 con 0x5635af7b3820 2024-04-05T23:35:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.296+0000 7f7ae8c00700 1 -- 172.21.15.16:0/485591426 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5635af7a6490 con 0x5635af7b3820 2024-04-05T23:35:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.296+0000 7f7ae8c00700 1 -- 172.21.15.16:0/485591426 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f7aec02c440 con 0x5635af7b3820 2024-04-05T23:35:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.297+0000 7f7b15419b80 10 client.15909 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.297+0000 7f7b15419b80 10 client.15909 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-05T23:35:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.297+0000 7f7b15419b80 20 client.15909 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.297+0000 7f7b15419b80 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5635af7d57c0 0x5635af7d7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.297+0000 7f7b15419b80 4 client.15909 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.297+0000 7f7b15419b80 1 -- 172.21.15.16:0/485591426 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635af6fc700 con 0x5635af7d57c0 2024-04-05T23:35:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.297+0000 7f7af3e00700 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5635af7d57c0 0x5635af7d7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.297+0000 7f7af3e00700 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5635af7d57c0 0x5635af7d7bb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ecb113ad0848d4b server_cookie=c1707aea796c9c8c in_seq=0 out_seq=0 2024-04-05T23:35:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.297+0000 7f7ae8c00700 10 client.15909 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:35:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.298+0000 7f7ae8c00700 1 -- 172.21.15.16:0/485591426 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5635af6fc700 con 0x5635af7d57c0 2024-04-05T23:35:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.298+0000 7f7ae8c00700 10 client.15909 handle_command_reply: tid=0 2024-04-05T23:35:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.298+0000 7f7b15419b80 1 client.15909 shutdown 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.298+0000 7f7b15419b80 1 -- 172.21.15.16:0/485591426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b3820 msgr2=0x5635af7c5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.298+0000 7f7b15419b80 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b3820 0x5635af7c5980 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7aec000c00 tx=0x7f7aec004a90 comp rx=0 tx=0).stop 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7b15419b80 1 -- 172.21.15.16:0/485591426 shutdown_connections 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7b15419b80 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5635af7d57c0 0x5635af7d7bb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7b15419b80 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7b3820 0x5635af7c5980 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7ae8c00700 0 client.15909 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7b15419b80 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7c5f40 0x5635af7c3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7b15419b80 1 --2- 172.21.15.16:0/485591426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7c2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7b15419b80 1 -- 172.21.15.16:0/485591426 >> 172.21.15.16:0/485591426 conn(0x5635af727220 msgr2=0x5635af7a8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7b15419b80 1 -- 172.21.15.16:0/485591426 shutdown_connections 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7b15419b80 1 -- 172.21.15.16:0/485591426 wait complete. 2024-04-05T23:35:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.299+0000 7f7b15419b80 20 client.15909 trim_cache size 0 max 16384 2024-04-05T23:35:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.312+0000 7f7b15419b80 1 Processor -- start 2024-04-05T23:35:25.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.312+0000 7f7b15419b80 1 -- start start 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7c2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 0x5635af7a8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7b15419b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af5876a0 con 0x5635af7c5f40 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7b15419b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af7b45c0 con 0x5635af7b7250 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7b15419b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af7cd1e0 con 0x5635af7b3820 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7afb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 0x5635af7a8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7afaa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7c2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7af3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7afaa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7c2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:47800/0 (socket says 172.21.15.16:47800) 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7afaa00700 1 -- 172.21.15.16:0/2356708117 learned_addr learned my addr 172.21.15.16:0/2356708117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7afaa00700 1 -- 172.21.15.16:0/2356708117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 msgr2=0x5635af7ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7afaa00700 1 --2- 172.21.15.16:0/2356708117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7afaa00700 1 -- 172.21.15.16:0/2356708117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 msgr2=0x5635af7a8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7afaa00700 1 --2- 172.21.15.16:0/2356708117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 0x5635af7a8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7afaa00700 1 -- 172.21.15.16:0/2356708117 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635af6c9ea0 con 0x5635af7b3820 2024-04-05T23:35:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.313+0000 7f7af3e00700 1 --2- 172.21.15.16:0/2356708117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.314+0000 7f7afaa00700 1 --2- 172.21.15.16:0/2356708117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7c2c80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0098e0 tx=0x7f7aec00b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aee4c5af9f4cb3d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.314+0000 7f7af0c00700 1 -- 172.21.15.16:0/2356708117 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec01d040 con 0x5635af7b3820 2024-04-05T23:35:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.314+0000 7f7af0c00700 1 -- 172.21.15.16:0/2356708117 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7aec004bb0 con 0x5635af7b3820 2024-04-05T23:35:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.314+0000 7f7af0c00700 1 -- 172.21.15.16:0/2356708117 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00f6c0 con 0x5635af7b3820 2024-04-05T23:35:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.314+0000 7f7b15419b80 1 -- 172.21.15.16:0/2356708117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 msgr2=0x5635af7c2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.314+0000 7f7b15419b80 1 --2- 172.21.15.16:0/2356708117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7c2c80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0098e0 tx=0x7f7aec00b630 comp rx=0 tx=0).stop 2024-04-05T23:35:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.315+0000 7f7b15419b80 1 -- 172.21.15.16:0/2356708117 shutdown_connections 2024-04-05T23:35:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.315+0000 7f7b15419b80 1 --2- 172.21.15.16:0/2356708117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 0x5635af7a8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.315+0000 7f7b15419b80 1 --2- 172.21.15.16:0/2356708117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7c2c80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.315+0000 7f7b15419b80 1 --2- 172.21.15.16:0/2356708117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.315+0000 7f7b15419b80 1 -- 172.21.15.16:0/2356708117 >> 172.21.15.16:0/2356708117 conn(0x5635af727220 msgr2=0x5635af7c2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.315+0000 7f7b15419b80 1 -- 172.21.15.16:0/2356708117 shutdown_connections 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.315+0000 7f7b15419b80 1 -- 172.21.15.16:0/2356708117 wait complete. 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.315+0000 7f7b15419b80 1 -- start start 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.315+0000 7f7b15419b80 1 Processor -- start 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7b15419b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 0x5635af7cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7b15419b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af828660 con 0x5635af7c5f40 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7b15419b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af8287d0 con 0x5635af7b7250 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7b15419b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635af828990 con 0x5635af7b3820 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7b15419b80 10 client.0 _ll_register_callbacks cb 0x7f7af4009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7af3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7afb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 0x5635af7cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7af3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37848/0 (socket says 172.21.15.16:37848) 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7af3e00700 1 -- 172.21.15.16:0/3337671536 learned_addr learned my addr 172.21.15.16:0/3337671536 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7afaa00700 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.316+0000 7f7af3e00700 1 -- 172.21.15.16:0/3337671536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 msgr2=0x5635af7cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7af3e00700 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7af3e00700 1 -- 172.21.15.16:0/3337671536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 msgr2=0x5635af7cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7af3e00700 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 0x5635af7cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7af3e00700 1 -- 172.21.15.16:0/3337671536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae40117a0 con 0x5635af7b7250 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7afb400700 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 0x5635af7cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7afaa00700 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7af3e00700 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7cb190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4014000 tx=0x7f7ae4013070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d77a9cd407ca460 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7ae8c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7ae8c00700 1 -- 172.21.15.16:0/3337671536 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4024040 con 0x5635af7b7250 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7b15419b80 1 -- 172.21.15.16:0/3337671536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7aec01e040 con 0x5635af7b7250 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7ae8c00700 1 -- 172.21.15.16:0/3337671536 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f7ae4017070 con 0x5635af7b7250 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7ae8c00700 1 -- 172.21.15.16:0/3337671536 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ae4028730 con 0x5635af7b7250 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7b15419b80 10 client.25949 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.317+0000 7f7b15419b80 1 -- 172.21.15.16:0/3337671536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x5635af531b60 con 0x5635af7b7250 2024-04-05T23:35:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.318+0000 7f7ae8c00700 1 -- 172.21.15.16:0/3337671536 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ae403d080 con 0x5635af7b7250 2024-04-05T23:35:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.318+0000 7f7ae8c00700 1 -- 172.21.15.16:0/3337671536 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f7ae4031b20 con 0x5635af7b7250 2024-04-05T23:35:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.318+0000 7f7b15419b80 10 client.25949 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.318+0000 7f7b15419b80 10 client.25949 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-05T23:35:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.318+0000 7f7b15419b80 20 client.25949 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.318+0000 7f7b15419b80 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5635af82a1d0 0x5635af82c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.318+0000 7f7b15419b80 4 client.25949 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:35:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.318+0000 7f7b15419b80 1 -- 172.21.15.16:0/3337671536 --> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5635af552cf0 con 0x5635af82a1d0 2024-04-05T23:35:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.318+0000 7f7afaa00700 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5635af82a1d0 0x5635af82c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7afaa00700 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5635af82a1d0 0x5635af82c5f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6f3b9f5c29c9deef server_cookie=9df3b8cd60070177 in_seq=0 out_seq=0 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7ae8c00700 10 client.25949 ms_handle_connect on v2:172.21.15.130:6836/3873050281 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7ae8c00700 1 -- 172.21.15.16:0/3337671536 <== mds.2 v2:172.21.15.130:6836/3873050281 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+1376 (crc 0 0 0) 0x5635af552cf0 con 0x5635af82a1d0 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7ae8c00700 10 client.25949 handle_command_reply: tid=0 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7b15419b80 1 client.25949 shutdown 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7b15419b80 1 -- 172.21.15.16:0/3337671536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 msgr2=0x5635af7cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7b15419b80 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7cb190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4014000 tx=0x7f7ae4013070 comp rx=0 tx=0).stop 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7b15419b80 1 -- 172.21.15.16:0/3337671536 shutdown_connections 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7b15419b80 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.130:6836/3873050281,v1:172.21.15.130:6837/3873050281] conn(0x5635af82a1d0 0x5635af82c5f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7b15419b80 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635af7c5f40 0x5635af7cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7b15419b80 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635af7b3820 0x5635af7cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7ae8c00700 0 client.25949 ms_handle_reset on v2:172.21.15.130:6836/3873050281 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7b15419b80 1 --2- 172.21.15.16:0/3337671536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635af7b7250 0x5635af7cb190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.319+0000 7f7b15419b80 1 -- 172.21.15.16:0/3337671536 >> 172.21.15.16:0/3337671536 conn(0x5635af727220 msgr2=0x5635af7bff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b15419b80 1 -- 172.21.15.16:0/3337671536 shutdown_connections 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b15419b80 1 -- 172.21.15.16:0/3337671536 wait complete. 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b15419b80 20 client.25949 trim_cache size 0 max 16384 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 283048517, 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511628322, 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP" 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:35:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 2136576956, 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511628323, 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS" 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 2408267209, 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511628324, 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP" 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-04-05T23:35:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 2573444147, 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511628327, 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC" 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 3084814481, 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511628328, 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC" 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "damage_type": "backtrace", 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 3667539661, 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 1099511628326, 2024-04-05T23:35:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS" 2024-04-05T23:35:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:35:25.322 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-04-05T23:35:25.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 -- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ae00a1290 msgr2=0x7f7ae00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ae00a1290 0x7f7ae00a3750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7afc1988c0 tx=0x7f7ae40073d0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 -- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc19bc50 msgr2=0x7f7afc19c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc19bc50 0x7f7afc19c0d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7af400ef30 tx=0x7f7af400c3a0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 -- 172.21.15.16:0/835016346 shutdown_connections 2024-04-05T23:35:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ae00a1290 0x7f7ae00a3750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7afc19bc50 0x7f7afc19c0d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7afc1010a0 0x7f7afc197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 --2- 172.21.15.16:0/835016346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7afc1061c0 0x7f7afc1978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 -- 172.21.15.16:0/835016346 >> 172.21.15.16:0/835016346 conn(0x7f7afc0fc900 msgr2=0x7f7afc0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.320+0000 7f7b01800700 1 -- 172.21.15.16:0/835016346 shutdown_connections 2024-04-05T23:35:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.321+0000 7f7b01800700 1 -- 172.21.15.16:0/835016346 wait complete. 2024-04-05T23:35:25.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-05T23:35:25.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:24 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2037454867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:35:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.539+0000 7fde4ec00700 1 -- 172.21.15.16:0/175250761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde48101a50 msgr2=0x7fde48105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.539+0000 7fde4ec00700 1 --2- 172.21.15.16:0/175250761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde48101a50 0x7fde48105a90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fde300098f0 tx=0x7fde30009c00 comp rx=0 tx=0).stop 2024-04-05T23:35:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.539+0000 7fde4ec00700 1 -- 172.21.15.16:0/175250761 shutdown_connections 2024-04-05T23:35:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.539+0000 7fde4ec00700 1 --2- 172.21.15.16:0/175250761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde481061c0 0x7fde4810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.539+0000 7fde4ec00700 1 --2- 172.21.15.16:0/175250761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde481010a0 0x7fde48101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.539+0000 7fde4ec00700 1 --2- 172.21.15.16:0/175250761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde48101a50 0x7fde48105a90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.539+0000 7fde4ec00700 1 -- 172.21.15.16:0/175250761 >> 172.21.15.16:0/175250761 conn(0x7fde480fc900 msgr2=0x7fde480fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.539+0000 7fde4ec00700 1 -- 172.21.15.16:0/175250761 shutdown_connections 2024-04-05T23:35:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.539+0000 7fde4ec00700 1 -- 172.21.15.16:0/175250761 wait complete. 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.540+0000 7fde4ec00700 1 Processor -- start 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.540+0000 7fde4ec00700 1 -- start start 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.540+0000 7fde4ec00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde481010a0 0x7fde480731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4ec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde481061c0 0x7fde48073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4ec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde48073c60 0x7fde481a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4ec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde481a5b90 con 0x7fde48073c60 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4ec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde481a5d00 con 0x7fde481061c0 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4ec00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde481a5ec0 con 0x7fde481010a0 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde48073c60 0x7fde481a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde48073c60 0x7fde481a5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36712/0 (socket says 172.21.15.16:36712) 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4cc00700 1 -- 172.21.15.16:0/2093204805 learned_addr learned my addr 172.21.15.16:0/2093204805 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde47e00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde481010a0 0x7fde480731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde3fe00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde481061c0 0x7fde48073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4cc00700 1 -- 172.21.15.16:0/2093204805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde481010a0 msgr2=0x7fde480731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4cc00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde481010a0 0x7fde480731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4cc00700 1 -- 172.21.15.16:0/2093204805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde481061c0 msgr2=0x7fde48073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4cc00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde481061c0 0x7fde48073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.541+0000 7fde4cc00700 1 -- 172.21.15.16:0/2093204805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40009d00 con 0x7fde48073c60 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.542+0000 7fde4cc00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde48073c60 0x7fde481a5590 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fde4000e3c0 tx=0x7fde4000e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.542+0000 7fde45600700 1 -- 172.21.15.16:0/2093204805 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde400090d0 con 0x7fde48073c60 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.542+0000 7fde4ec00700 1 -- 172.21.15.16:0/2093204805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde30009520 con 0x7fde48073c60 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.542+0000 7fde45600700 1 -- 172.21.15.16:0/2093204805 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde4000f040 con 0x7fde48073c60 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.542+0000 7fde45600700 1 -- 172.21.15.16:0/2093204805 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40007c20 con 0x7fde48073c60 2024-04-05T23:35:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.542+0000 7fde4ec00700 1 -- 172.21.15.16:0/2093204805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde481a64f0 con 0x7fde48073c60 2024-04-05T23:35:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.545+0000 7fde45600700 1 -- 172.21.15.16:0/2093204805 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fde40003a00 con 0x7fde48073c60 2024-04-05T23:35:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.546+0000 7fde45600700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fde2c0a1290 0x7fde2c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.546+0000 7fde627a8b80 1 Processor -- start 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde627a8b80 1 -- start start 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde45600700 1 -- 172.21.15.16:0/2093204805 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fde400ccbe0 con 0x7fde48073c60 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde47e00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fde2c0a1290 0x7fde2c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 0x55f65b7ea560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e42e0 0x55f65b7e4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde627a8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b74abe0 con 0x55f65b7e4ca0 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde627a8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b73b990 con 0x55f65b7e42e0 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde627a8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b739350 con 0x55f65b7ea180 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde4cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde4cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36716/0 (socket says 172.21.15.16:36716) 2024-04-05T23:35:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde4cc00700 1 -- 172.21.15.16:0/3473679373 learned_addr learned my addr 172.21.15.16:0/3473679373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde47e00700 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 0x55f65b7ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.547+0000 7fde3fe00700 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e42e0 0x55f65b7e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde4cc00700 1 -- 172.21.15.16:0/3473679373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 msgr2=0x55f65b7ea560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde4cc00700 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 0x55f65b7ea560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde4cc00700 1 -- 172.21.15.16:0/3473679373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e42e0 msgr2=0x55f65b7e4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde4cc00700 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e42e0 0x55f65b7e4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde4cc00700 1 -- 172.21.15.16:0/3473679373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f65b5b63b0 con 0x55f65b7e4ca0 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde47e00700 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 0x55f65b7ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde47e00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fde2c0a1290 0x7fde2c0a3750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fde38009c70 tx=0x7fde38011720 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde4cc00700 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f0f10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fde400cce40 tx=0x7fde40099be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b68784e41fa620 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde3d200700 1 -- 172.21.15.16:0/3473679373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4009a490 con 0x55f65b7e4ca0 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde3d200700 1 -- 172.21.15.16:0/3473679373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde40099d90 con 0x55f65b7e4ca0 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.548+0000 7fde3d200700 1 -- 172.21.15.16:0/3473679373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde400092c0 con 0x55f65b7e4ca0 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.549+0000 7fde627a8b80 1 -- 172.21.15.16:0/3473679373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e4ca0 msgr2=0x55f65b7f0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.549+0000 7fde627a8b80 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f0f10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fde400cce40 tx=0x7fde40099be0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.549+0000 7fde627a8b80 1 -- 172.21.15.16:0/3473679373 shutdown_connections 2024-04-05T23:35:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.549+0000 7fde627a8b80 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f0f10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.549+0000 7fde627a8b80 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 0x55f65b7ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.549+0000 7fde627a8b80 1 --2- 172.21.15.16:0/3473679373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e42e0 0x55f65b7e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.549+0000 7fde627a8b80 1 -- 172.21.15.16:0/3473679373 >> 172.21.15.16:0/3473679373 conn(0x55f65b758220 msgr2=0x55f65b7df570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.549+0000 7fde627a8b80 1 -- 172.21.15.16:0/3473679373 shutdown_connections 2024-04-05T23:35:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.549+0000 7fde627a8b80 1 -- 172.21.15.16:0/3473679373 wait complete. 2024-04-05T23:35:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.550+0000 7fde627a8b80 1 -- start start 2024-04-05T23:35:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.550+0000 7fde627a8b80 1 Processor -- start 2024-04-05T23:35:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e4ca0 0x55f65b7ee110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7ea180 0x55f65b7ee790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde627a8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b734c40 con 0x55f65b7e42e0 2024-04-05T23:35:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde627a8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b6d0d80 con 0x55f65b7ea180 2024-04-05T23:35:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde627a8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b6d0a20 con 0x55f65b7e4ca0 2024-04-05T23:35:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde627a8b80 10 client.0 _ll_register_callbacks cb 0x55f65b53c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde47e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde47e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36724/0 (socket says 172.21.15.16:36724) 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde47e00700 1 -- 172.21.15.16:0/1081958474 learned_addr learned my addr 172.21.15.16:0/1081958474 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde3fe00700 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e4ca0 0x55f65b7ee110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde4cc00700 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7ea180 0x55f65b7ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde47e00700 1 -- 172.21.15.16:0/1081958474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e4ca0 msgr2=0x55f65b7ee110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde47e00700 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e4ca0 0x55f65b7ee110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde47e00700 1 -- 172.21.15.16:0/1081958474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7ea180 msgr2=0x55f65b7ee790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde47e00700 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7ea180 0x55f65b7ee790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde47e00700 1 -- 172.21.15.16:0/1081958474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde481a64f0 con 0x55f65b7e42e0 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.551+0000 7fde4cc00700 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7ea180 0x55f65b7ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.552+0000 7fde47e00700 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f0e20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fde3801ea00 tx=0x7fde380047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d378eed4832f3055 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.552+0000 7fde35600700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.552+0000 7fde35600700 1 -- 172.21.15.16:0/1081958474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3801ee80 con 0x55f65b7e42e0 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.552+0000 7fde627a8b80 1 -- 172.21.15.16:0/1081958474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f65b7d7490 con 0x55f65b7e42e0 2024-04-05T23:35:25.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.552+0000 7fde35600700 1 -- 172.21.15.16:0/1081958474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fde3800f850 con 0x55f65b7e42e0 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.552+0000 7fde35600700 1 -- 172.21.15.16:0/1081958474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde3800fa10 con 0x55f65b7e42e0 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.552+0000 7fde627a8b80 10 client.15927 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.552+0000 7fde627a8b80 1 -- 172.21.15.16:0/1081958474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55f65b562b60 con 0x55f65b7e42e0 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.552+0000 7fde35600700 1 -- 172.21.15.16:0/1081958474 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f65b7d7490 con 0x55f65b7e42e0 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.553+0000 7fde35600700 1 -- 172.21.15.16:0/1081958474 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fde38032ce0 con 0x55f65b7e42e0 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.553+0000 7fde627a8b80 10 client.15927 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.553+0000 7fde627a8b80 10 client.15927 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.553+0000 7fde627a8b80 20 client.15927 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.553+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55f65b7fe450 0x55f65b800870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.553+0000 7fde627a8b80 4 client.15927 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.553+0000 7fde627a8b80 1 -- 172.21.15.16:0/1081958474 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f65b72d700 con 0x55f65b7fe450 2024-04-05T23:35:25.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.553+0000 7fde3fe00700 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55f65b7fe450 0x55f65b800870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.554+0000 7fde3fe00700 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55f65b7fe450 0x55f65b800870 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4c2de43f01118bf3 server_cookie=1cbebc45e97b626 in_seq=0 out_seq=0 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.554+0000 7fde35600700 10 client.15927 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.554+0000 7fde35600700 1 -- 172.21.15.16:0/1081958474 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f65b72d700 con 0x55f65b7fe450 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.554+0000 7fde35600700 10 client.15927 handle_command_reply: tid=0 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 client.15927 shutdown 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 -- 172.21.15.16:0/1081958474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 msgr2=0x55f65b7f0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f0e20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fde3801ea00 tx=0x7fde380047c0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 -- 172.21.15.16:0/1081958474 shutdown_connections 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f0e20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55f65b7fe450 0x55f65b800870 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7ea180 0x55f65b7ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1081958474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e4ca0 0x55f65b7ee110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 -- 172.21.15.16:0/1081958474 >> 172.21.15.16:0/1081958474 conn(0x55f65b758220 msgr2=0x55f65b7d9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde35600700 0 client.15927 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:35:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 -- 172.21.15.16:0/1081958474 shutdown_connections 2024-04-05T23:35:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 1 -- 172.21.15.16:0/1081958474 wait complete. 2024-04-05T23:35:25.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.555+0000 7fde627a8b80 20 client.15927 trim_cache size 0 max 16384 2024-04-05T23:35:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde627a8b80 1 Processor -- start 2024-04-05T23:35:25.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde627a8b80 1 -- start start 2024-04-05T23:35:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e42e0 0x55f65b7d9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 0x55f65b7e91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7ea180 0x55f65b7e9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde627a8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b5b86a0 con 0x55f65b7ea180 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde627a8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b7ef070 con 0x55f65b7e4ca0 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde627a8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b7e8dd0 con 0x55f65b7e42e0 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde4cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7ea180 0x55f65b7e9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde47e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e42e0 0x55f65b7d9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde3fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 0x55f65b7e91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde47e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e42e0 0x55f65b7d9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:47852/0 (socket says 172.21.15.16:47852) 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.569+0000 7fde47e00700 1 -- 172.21.15.16:0/1846822948 learned_addr learned my addr 172.21.15.16:0/1846822948 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.570+0000 7fde3fe00700 1 -- 172.21.15.16:0/1846822948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e42e0 msgr2=0x55f65b7d9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.570+0000 7fde3fe00700 1 --2- 172.21.15.16:0/1846822948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e42e0 0x55f65b7d9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.570+0000 7fde3fe00700 1 -- 172.21.15.16:0/1846822948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7ea180 msgr2=0x55f65b7e9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.570+0000 7fde3fe00700 1 --2- 172.21.15.16:0/1846822948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7ea180 0x55f65b7e9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.570+0000 7fde3fe00700 1 -- 172.21.15.16:0/1846822948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f65b6faea0 con 0x55f65b7e4ca0 2024-04-05T23:35:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.570+0000 7fde3fe00700 1 --2- 172.21.15.16:0/1846822948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 0x55f65b7e91e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fde30009230 tx=0x7fde30005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e6dcb0d7aa70874 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.570+0000 7fde3dc00700 1 -- 172.21.15.16:0/1846822948 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde30027b40 con 0x55f65b7e4ca0 2024-04-05T23:35:25.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.570+0000 7fde3dc00700 1 -- 172.21.15.16:0/1846822948 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde3001d040 con 0x55f65b7e4ca0 2024-04-05T23:35:25.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.570+0000 7fde3dc00700 1 -- 172.21.15.16:0/1846822948 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3000f960 con 0x55f65b7e4ca0 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 -- 172.21.15.16:0/1846822948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 msgr2=0x55f65b7e91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1846822948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 0x55f65b7e91e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fde30009230 tx=0x7fde30005dc0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 -- 172.21.15.16:0/1846822948 shutdown_connections 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1846822948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7ea180 0x55f65b7e9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1846822948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7e42e0 0x55f65b7d9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 --2- 172.21.15.16:0/1846822948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 0x55f65b7e91e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 -- 172.21.15.16:0/1846822948 >> 172.21.15.16:0/1846822948 conn(0x55f65b758220 msgr2=0x55f65b7df040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 -- 172.21.15.16:0/1846822948 shutdown_connections 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 -- 172.21.15.16:0/1846822948 wait complete. 2024-04-05T23:35:25.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.571+0000 7fde627a8b80 1 -- start start 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde627a8b80 1 Processor -- start 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde627a8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 0x55f65b851280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde627a8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b7f4300 con 0x55f65b7e42e0 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde627a8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b8517c0 con 0x55f65b7e4ca0 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde627a8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65b851900 con 0x55f65b7ea180 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde627a8b80 10 client.0 _ll_register_callbacks cb 0x7fde3801e7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde47e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde47e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36738/0 (socket says 172.21.15.16:36738) 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde47e00700 1 -- 172.21.15.16:0/8267159 learned_addr learned my addr 172.21.15.16:0/8267159 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde4cc00700 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 0x55f65b851280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.572+0000 7fde3fe00700 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde47e00700 1 -- 172.21.15.16:0/8267159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 msgr2=0x55f65b851280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde47e00700 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 0x55f65b851280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde47e00700 1 -- 172.21.15.16:0/8267159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 msgr2=0x55f65b7f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde47e00700 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde47e00700 1 -- 172.21.15.16:0/8267159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4810e5b0 con 0x55f65b7e42e0 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde47e00700 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f36e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fde38009350 tx=0x7fde38009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59bf78d2ec56cd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde35600700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde35600700 1 -- 172.21.15.16:0/8267159 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3800f970 con 0x55f65b7e42e0 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde627a8b80 1 -- 172.21.15.16:0/8267159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde40099d90 con 0x55f65b7e42e0 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde35600700 1 -- 172.21.15.16:0/8267159 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fde3800fad0 con 0x55f65b7e42e0 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde35600700 1 -- 172.21.15.16:0/8267159 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde3800fc90 con 0x55f65b7e42e0 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde627a8b80 10 client.15933 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.573+0000 7fde627a8b80 1 -- 172.21.15.16:0/8267159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55f65b5bacd0 con 0x55f65b7e42e0 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.574+0000 7fde35600700 1 -- 172.21.15.16:0/8267159 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde40099d90 con 0x55f65b7e42e0 2024-04-05T23:35:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.574+0000 7fde35600700 1 -- 172.21.15.16:0/8267159 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fde3801d440 con 0x55f65b7e42e0 2024-04-05T23:35:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.574+0000 7fde627a8b80 10 client.15933 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.574+0000 7fde627a8b80 10 client.15933 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-05T23:35:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.574+0000 7fde627a8b80 20 client.15933 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.574+0000 7fde627a8b80 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55f65b853110 0x55f65b855530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.574+0000 7fde627a8b80 4 client.15933 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:35:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.574+0000 7fde627a8b80 1 -- 172.21.15.16:0/8267159 --> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f65b583cf0 con 0x55f65b853110 2024-04-05T23:35:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.574+0000 7fde3fe00700 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55f65b853110 0x55f65b855530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde3fe00700 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55f65b853110 0x55f65b855530 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c5c99f9dd8f91765 server_cookie=315e45b982f19c4 in_seq=0 out_seq=0 2024-04-05T23:35:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde35600700 10 client.15933 ms_handle_connect on v2:172.21.15.130:6838/2970827239 2024-04-05T23:35:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde35600700 1 -- 172.21.15.16:0/8267159 <== mds.1 v2:172.21.15.130:6838/2970827239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f65b583cf0 con 0x55f65b853110 2024-04-05T23:35:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde35600700 10 client.15933 handle_command_reply: tid=0 2024-04-05T23:35:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde627a8b80 1 client.15933 shutdown 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde627a8b80 1 -- 172.21.15.16:0/8267159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 msgr2=0x55f65b7f36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde627a8b80 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f36e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fde38009350 tx=0x7fde38009380 comp rx=0 tx=0).stop 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde627a8b80 1 -- 172.21.15.16:0/8267159 shutdown_connections 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde627a8b80 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f65b7e42e0 0x55f65b7f36e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde627a8b80 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.130:6838/2970827239,v1:172.21.15.130:6839/2970827239] conn(0x55f65b853110 0x55f65b855530 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde627a8b80 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f65b7ea180 0x55f65b851280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde627a8b80 1 --2- 172.21.15.16:0/8267159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f65b7e4ca0 0x55f65b7f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde627a8b80 1 -- 172.21.15.16:0/8267159 >> 172.21.15.16:0/8267159 conn(0x55f65b758220 msgr2=0x55f65b7e91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.575+0000 7fde35600700 0 client.15933 ms_handle_reset on v2:172.21.15.130:6838/2970827239 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde627a8b80 1 -- 172.21.15.16:0/8267159 shutdown_connections 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde627a8b80 1 -- 172.21.15.16:0/8267159 wait complete. 2024-04-05T23:35:25.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde627a8b80 20 client.15933 trim_cache size 0 max 16384 2024-04-05T23:35:25.572 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:35:25.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 -- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fde2c0a1290 msgr2=0x7fde2c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fde2c0a1290 0x7fde2c0a3750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fde38009c70 tx=0x7fde38011720 comp rx=0 tx=0).stop 2024-04-05T23:35:25.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 -- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde48073c60 msgr2=0x7fde481a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde48073c60 0x7fde481a5590 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fde4000e3c0 tx=0x7fde4000e6d0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 -- 172.21.15.16:0/2093204805 shutdown_connections 2024-04-05T23:35:25.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fde2c0a1290 0x7fde2c0a3750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde48073c60 0x7fde481a5590 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde481010a0 0x7fde480731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 --2- 172.21.15.16:0/2093204805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde481061c0 0x7fde48073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.576+0000 7fde4ec00700 1 -- 172.21.15.16:0/2093204805 >> 172.21.15.16:0/2093204805 conn(0x7fde480fc900 msgr2=0x7fde480fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.577+0000 7fde4ec00700 1 -- 172.21.15.16:0/2093204805 shutdown_connections 2024-04-05T23:35:25.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.577+0000 7fde4ec00700 1 -- 172.21.15.16:0/2093204805 wait complete. 2024-04-05T23:35:25.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-05T23:35:25.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.799+0000 7f4e08600700 1 -- 172.21.15.16:0/819514812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073280 msgr2=0x7f4e00073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.799+0000 7f4e08600700 1 --2- 172.21.15.16:0/819514812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073280 0x7f4e00073660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4df40098f0 tx=0x7f4df4009c00 comp rx=0 tx=0).stop 2024-04-05T23:35:25.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.799+0000 7f4e08600700 1 -- 172.21.15.16:0/819514812 shutdown_connections 2024-04-05T23:35:25.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.799+0000 7f4e08600700 1 --2- 172.21.15.16:0/819514812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e00111e40 0x7f4e00114230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.799+0000 7f4e08600700 1 --2- 172.21.15.16:0/819514812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073280 0x7f4e00073660 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.799+0000 7f4e08600700 1 --2- 172.21.15.16:0/819514812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00073ba0 0x7f4e00111900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.799+0000 7f4e08600700 1 -- 172.21.15.16:0/819514812 >> 172.21.15.16:0/819514812 conn(0x7f4e000fc900 msgr2=0x7f4e000fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.799+0000 7f4e08600700 1 -- 172.21.15.16:0/819514812 shutdown_connections 2024-04-05T23:35:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.799+0000 7f4e08600700 1 -- 172.21.15.16:0/819514812 wait complete. 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.800+0000 7f4e08600700 1 Processor -- start 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e08600700 1 -- start start 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e08600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00073280 0x7f4e001a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e08600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073ba0 0x7f4e001a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e08600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e00111e40 0x7f4e0019d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e08600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0019de70 con 0x7f4e00111e40 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e08600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0019dfe0 con 0x7f4e00073280 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e08600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0019e1a0 con 0x7f4e00073ba0 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e05200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073ba0 0x7f4e001a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e06600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e00111e40 0x7f4e0019d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e05200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073ba0 0x7f4e001a3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:47876/0 (socket says 172.21.15.16:47876) 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.801+0000 7f4e05200700 1 -- 172.21.15.16:0/18916890 learned_addr learned my addr 172.21.15.16:0/18916890 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4e05c00700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00073280 0x7f4e001a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4e05200700 1 -- 172.21.15.16:0/18916890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00073280 msgr2=0x7f4e001a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4e05200700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00073280 0x7f4e001a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4e05200700 1 -- 172.21.15.16:0/18916890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e00111e40 msgr2=0x7f4e0019d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4e05200700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e00111e40 0x7f4e0019d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4e05200700 1 -- 172.21.15.16:0/18916890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df8009d00 con 0x7f4e00073ba0 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4e05c00700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00073280 0x7f4e001a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4e05200700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073ba0 0x7f4e001a3c10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e9a0 tx=0x7f4dec00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4dfe000700 1 -- 172.21.15.16:0/18916890 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec009790 con 0x7f4e00073ba0 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4dfe000700 1 -- 172.21.15.16:0/18916890 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4dec00cbd0 con 0x7f4e00073ba0 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4dfe000700 1 -- 172.21.15.16:0/18916890 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec018680 con 0x7f4e00073ba0 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.802+0000 7f4e08600700 1 -- 172.21.15.16:0/18916890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df4009520 con 0x7f4e00073ba0 2024-04-05T23:35:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.803+0000 7f4e08600700 1 -- 172.21.15.16:0/18916890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e001aa4e0 con 0x7f4e00073ba0 2024-04-05T23:35:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.807+0000 7f4dfe000700 1 -- 172.21.15.16:0/18916890 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f4dec00cd40 con 0x7f4e00073ba0 2024-04-05T23:35:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.808+0000 7f4e1c220b80 1 Processor -- start 2024-04-05T23:35:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.808+0000 7f4e1c220b80 1 -- start start 2024-04-05T23:35:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.808+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 0x562a487ac560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.808+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a62e0 0x562a487a6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.808+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.808+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a4870cbe0 con 0x562a487a6ca0 2024-04-05T23:35:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.808+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a486fd990 con 0x562a487ac180 2024-04-05T23:35:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.808+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a486fb350 con 0x562a487a62e0 2024-04-05T23:35:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4e05200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4e05c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a62e0 0x562a487a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4e06600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 0x562a487ac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4e05200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36768/0 (socket says 172.21.15.16:36768) 2024-04-05T23:35:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4e05200700 1 -- 172.21.15.16:0/2024080737 learned_addr learned my addr 172.21.15.16:0/2024080737 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4dfe000700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4de80a1290 0x7f4de80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4e05c00700 1 -- 172.21.15.16:0/2024080737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 msgr2=0x562a487ac560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4e05c00700 1 --2- 172.21.15.16:0/2024080737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 0x562a487ac560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4dfe000700 1 -- 172.21.15.16:0/18916890 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f4dec014070 con 0x7f4e00073ba0 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4e05c00700 1 -- 172.21.15.16:0/2024080737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 msgr2=0x562a487b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.809+0000 7f4e05c00700 1 --2- 172.21.15.16:0/2024080737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.810+0000 7f4e05c00700 1 -- 172.21.15.16:0/2024080737 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a485783b0 con 0x562a487a62e0 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.810+0000 7f4e05200700 1 --2- 172.21.15.16:0/2024080737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.810+0000 7f4e06600700 1 --2- 172.21.15.16:0/2024080737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 0x562a487ac560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.810+0000 7f4e06600700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4de80a1290 0x7f4de80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.810+0000 7f4e05c00700 1 --2- 172.21.15.16:0/2024080737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a62e0 0x562a487a6760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4df40098c0 tx=0x7f4df400bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0c91647a08a5d68 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.810+0000 7f4df2a00700 1 -- 172.21.15.16:0/2024080737 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df401c040 con 0x562a487a62e0 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.810+0000 7f4df2a00700 1 -- 172.21.15.16:0/2024080737 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4df400f920 con 0x562a487a62e0 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.810+0000 7f4df2a00700 1 -- 172.21.15.16:0/2024080737 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df40052a0 con 0x562a487a62e0 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2024080737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a62e0 msgr2=0x562a487a6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/2024080737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a62e0 0x562a487a6760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4df40098c0 tx=0x7f4df400bef0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e06600700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4de80a1290 0x7f4de80a3750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4e001a43d0 tx=0x7f4df8004ad0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2024080737 shutdown_connections 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/2024080737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/2024080737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 0x562a487ac560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/2024080737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a62e0 0x562a487a6760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2024080737 >> 172.21.15.16:0/2024080737 conn(0x562a4871a220 msgr2=0x562a487a1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2024080737 shutdown_connections 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.811+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2024080737 wait complete. 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.812+0000 7f4e1c220b80 1 -- start start 2024-04-05T23:35:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.812+0000 7f4e1c220b80 1 Processor -- start 2024-04-05T23:35:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.812+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a487b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.812+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.812+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 0x562a487b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.812+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a486f6c40 con 0x562a487a6ca0 2024-04-05T23:35:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.812+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a48692d80 con 0x562a487a62e0 2024-04-05T23:35:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.812+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a48692a20 con 0x562a487ac180 2024-04-05T23:35:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.812+0000 7f4e1c220b80 10 client.0 _ll_register_callbacks cb 0x562a484fe3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36782/0 (socket says 172.21.15.16:36782) 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05200700 1 -- 172.21.15.16:0/1200806051 learned_addr learned my addr 172.21.15.16:0/1200806051 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05c00700 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a487b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e06600700 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 0x562a487b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05200700 1 -- 172.21.15.16:0/1200806051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 msgr2=0x562a487b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05200700 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 0x562a487b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05200700 1 -- 172.21.15.16:0/1200806051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 msgr2=0x562a487b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05200700 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a487b2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05200700 1 -- 172.21.15.16:0/1200806051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df80091b0 con 0x562a487a6ca0 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05c00700 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a487b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4e05200700 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b01b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00c5b0 tx=0x7f4dec0cac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27f260f093e67cd server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.813+0000 7f4de7400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4de7400700 1 -- 172.21.15.16:0/1200806051 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec099bb0 con 0x562a487a6ca0 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1200806051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562a48799490 con 0x562a487a6ca0 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4de7400700 1 -- 172.21.15.16:0/1200806051 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f4dec09a770 con 0x562a487a6ca0 2024-04-05T23:35:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4de7400700 1 -- 172.21.15.16:0/1200806051 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4dec0b2020 con 0x562a487a6ca0 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4e1c220b80 10 client.15945 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1200806051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7f4e00114be0 con 0x562a487a6ca0 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4de7400700 1 -- 172.21.15.16:0/1200806051 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562a48799490 con 0x562a487a6ca0 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4de7400700 1 -- 172.21.15.16:0/1200806051 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f4dec0a2060 con 0x562a487a6ca0 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4e1c220b80 10 client.15945 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4e1c220b80 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4e1c220b80 20 client.15945 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a487c03d0 0x562a487c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4e1c220b80 4 client.15945 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.814+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1200806051 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a486ef700 con 0x562a487c03d0 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.815+0000 7f4e05c00700 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a487c03d0 0x562a487c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.815+0000 7f4e05c00700 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a487c03d0 0x562a487c27f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a817e090ced394e1 server_cookie=a60589edd3311ea3 in_seq=0 out_seq=0 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.815+0000 7f4de7400700 10 client.15945 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.816+0000 7f4de7400700 1 -- 172.21.15.16:0/1200806051 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562a486ef700 con 0x562a487c03d0 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.816+0000 7f4de7400700 10 client.15945 handle_command_reply: tid=0 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.816+0000 7f4e1c220b80 1 client.15945 shutdown 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.816+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1200806051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 msgr2=0x562a487b01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.816+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b01b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00c5b0 tx=0x7f4dec0cac10 comp rx=0 tx=0).stop 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.816+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1200806051 shutdown_connections 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.816+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a487c03d0 0x562a487c27f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.817+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 0x562a487b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.817+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a487b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.817+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1200806051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b01b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.817+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1200806051 >> 172.21.15.16:0/1200806051 conn(0x562a4871a220 msgr2=0x562a4879b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.817+0000 7f4de7400700 0 client.15945 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.817+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1200806051 shutdown_connections 2024-04-05T23:35:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.817+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1200806051 wait complete. 2024-04-05T23:35:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.817+0000 7f4e1c220b80 20 client.15945 trim_cache size 0 max 16384 2024-04-05T23:35:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.832+0000 7f0e57000700 1 -- 172.21.15.16:0/2201165622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e501019d0 msgr2=0x7f0e50105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.832+0000 7f0e57000700 1 --2- 172.21.15.16:0/2201165622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e501019d0 0x7f0e50105a10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c009990 tx=0x7f0e3c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.832+0000 7f0e57000700 1 -- 172.21.15.16:0/2201165622 shutdown_connections 2024-04-05T23:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.832+0000 7f0e57000700 1 --2- 172.21.15.16:0/2201165622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e50106140 0x7f0e5010dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.832+0000 7f0e57000700 1 --2- 172.21.15.16:0/2201165622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e50101020 0x7f0e50101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.832+0000 7f0e57000700 1 --2- 172.21.15.16:0/2201165622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e501019d0 0x7f0e50105a10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.832+0000 7f0e57000700 1 -- 172.21.15.16:0/2201165622 >> 172.21.15.16:0/2201165622 conn(0x7f0e500fc880 msgr2=0x7f0e500feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.832+0000 7f0e57000700 1 -- 172.21.15.16:0/2201165622 shutdown_connections 2024-04-05T23:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.832+0000 7f0e57000700 1 -- 172.21.15.16:0/2201165622 wait complete. 2024-04-05T23:35:25.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e1c220b80 1 Processor -- start 2024-04-05T23:35:25.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e1c220b80 1 -- start start 2024-04-05T23:35:25.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f0e57000700 1 Processor -- start 2024-04-05T23:35:25.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a4879b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 0x562a487a8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a487a9e10 con 0x562a487a6ca0 2024-04-05T23:35:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a4857a6a0 con 0x562a487a62e0 2024-04-05T23:35:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a4879bad0 con 0x562a487ac180 2024-04-05T23:35:25.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f0e57000700 1 -- start start 2024-04-05T23:35:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e05200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e05c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a4879b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e05c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a4879b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37978/0 (socket says 172.21.15.16:37978) 2024-04-05T23:35:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e05c00700 1 -- 172.21.15.16:0/2704009632 learned_addr learned my addr 172.21.15.16:0/2704009632 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.833+0000 7f4e06600700 1 --2- 172.21.15.16:0/2704009632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 0x562a487a8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e57000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e50101020 0x7f0e50199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f4e05200700 1 -- 172.21.15.16:0/2704009632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 msgr2=0x562a487a8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f4e05200700 1 --2- 172.21.15.16:0/2704009632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 0x562a487a8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f4e05200700 1 -- 172.21.15.16:0/2704009632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 msgr2=0x562a4879b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f4e05200700 1 --2- 172.21.15.16:0/2704009632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a4879b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f4e05200700 1 -- 172.21.15.16:0/2704009632 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a48524b60 con 0x562a487a6ca0 2024-04-05T23:35:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e57000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e50106140 0x7f0e50199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e57000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5019dde0 0x7f0e5019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e57000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5019e860 con 0x7f0e5019dde0 2024-04-05T23:35:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e57000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5019e9d0 con 0x7f0e50106140 2024-04-05T23:35:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e57000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5019eb90 con 0x7f0e50101020 2024-04-05T23:35:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f4e05c00700 1 --2- 172.21.15.16:0/2704009632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a4879b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e55000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5019dde0 0x7f0e5019e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e55000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5019dde0 0x7f0e5019e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36818/0 (socket says 172.21.15.16:36818) 2024-04-05T23:35:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e55000700 1 -- 172.21.15.16:0/1978428198 learned_addr learned my addr 172.21.15.16:0/1978428198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e4f400700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e50106140 0x7f0e50199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f4e05200700 1 --2- 172.21.15.16:0/2704009632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2950 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4dec096a90 tx=0x7f4dec099840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd6266c9592fde5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.834+0000 7f0e4fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e50101020 0x7f0e50199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f4df3400700 1 -- 172.21.15.16:0/2704009632 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0b2070 con 0x562a487a6ca0 2024-04-05T23:35:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e55000700 1 -- 172.21.15.16:0/1978428198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e50101020 msgr2=0x7f0e50199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e55000700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e50101020 0x7f0e50199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e55000700 1 -- 172.21.15.16:0/1978428198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e50106140 msgr2=0x7f0e50199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e55000700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e50106140 0x7f0e50199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e55000700 1 -- 172.21.15.16:0/1978428198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e40009520 con 0x7f0e5019dde0 2024-04-05T23:35:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f4df3400700 1 -- 172.21.15.16:0/2704009632 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4dec099bb0 con 0x562a487a6ca0 2024-04-05T23:35:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f4df3400700 1 -- 172.21.15.16:0/2704009632 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0a2060 con 0x562a487a6ca0 2024-04-05T23:35:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e50101020 0x7f0e50199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:35:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e55000700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5019dde0 0x7f0e5019e260 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800ef30 tx=0x7f0e4800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e4cc00700 1 -- 172.21.15.16:0/1978428198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48018070 con 0x7f0e5019dde0 2024-04-05T23:35:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e4cc00700 1 -- 172.21.15.16:0/1978428198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e4800f040 con 0x7f0e5019dde0 2024-04-05T23:35:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e57000700 1 -- 172.21.15.16:0/1978428198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c0095f0 con 0x7f0e5019dde0 2024-04-05T23:35:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e4cc00700 1 -- 172.21.15.16:0/1978428198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4800cdb0 con 0x7f0e5019dde0 2024-04-05T23:35:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.835+0000 7f0e57000700 1 -- 172.21.15.16:0/1978428198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e501a6430 con 0x7f0e5019dde0 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.836+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2704009632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 msgr2=0x562a487b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.836+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/2704009632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2950 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4dec096a90 tx=0x7f4dec099840 comp rx=0 tx=0).stop 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.836+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2704009632 shutdown_connections 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.836+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/2704009632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487ac180 0x562a487a8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.836+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/2704009632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487a62e0 0x562a4879b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.836+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/2704009632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a6ca0 0x562a487b2950 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.836+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2704009632 >> 172.21.15.16:0/2704009632 conn(0x562a4871a220 msgr2=0x562a487aeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.836+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2704009632 shutdown_connections 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.836+0000 7f4e1c220b80 1 -- 172.21.15.16:0/2704009632 wait complete. 2024-04-05T23:35:25.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.837+0000 7f4e1c220b80 1 -- start start 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.837+0000 7f4e1c220b80 1 Processor -- start 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.837+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a62e0 0x562a487a8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.837+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a6ca0 0x562a487a93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.837+0000 7f4e1c220b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 0x562a487a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.837+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a487a8070 con 0x562a487a62e0 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.837+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a487a81e0 con 0x562a487ac180 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.837+0000 7f4e1c220b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a487a83a0 con 0x562a487a6ca0 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.837+0000 7f4e1c220b80 10 client.0 _ll_register_callbacks cb 0x7f4dec096410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e05c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a62e0 0x562a487a8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e05c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a62e0 0x562a487a8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36822/0 (socket says 172.21.15.16:36822) 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e05c00700 1 -- 172.21.15.16:0/1920225519 learned_addr learned my addr 172.21.15.16:0/1920225519 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e05200700 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a6ca0 0x562a487a93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e06600700 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 0x562a487a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e05200700 1 -- 172.21.15.16:0/1920225519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 msgr2=0x562a487a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e05200700 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 0x562a487a7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e05200700 1 -- 172.21.15.16:0/1920225519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a62e0 msgr2=0x562a487a8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e05200700 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a62e0 0x562a487a8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.838+0000 7f4e05200700 1 -- 172.21.15.16:0/1920225519 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562a485783b0 con 0x562a487a6ca0 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e05200700 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a6ca0 0x562a487a93e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x562a487c00b0 tx=0x7f4dec0999d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a44dc1224850ef8d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4de7400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4de7400700 1 -- 172.21.15.16:0/1920225519 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0b6070 con 0x562a487a6ca0 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4de7400700 1 -- 172.21.15.16:0/1920225519 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f4dec09a770 con 0x562a487a6ca0 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1920225519 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4df400f920 con 0x562a487a6ca0 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4de7400700 1 -- 172.21.15.16:0/1920225519 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4dec0b2050 con 0x562a487a6ca0 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e1c220b80 10 client.26026 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1920225519 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7f4e00114be0 con 0x562a487a6ca0 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4de7400700 1 -- 172.21.15.16:0/1920225519 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4df400f920 con 0x562a487a6ca0 2024-04-05T23:35:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4de7400700 1 -- 172.21.15.16:0/1920225519 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f4dec09edc0 con 0x562a487a6ca0 2024-04-05T23:35:25.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f0e4cc00700 1 -- 172.21.15.16:0/1978428198 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f0e48003a00 con 0x7f0e5019dde0 2024-04-05T23:35:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e1c220b80 10 client.26026 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e1c220b80 10 client.26026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e1c220b80 20 client.26026 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a48814d90 0x562a488171b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e1c220b80 4 client.26026 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-05T23:35:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.839+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1920225519 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562a48545cf0 con 0x562a48814d90 2024-04-05T23:35:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.840+0000 7f4e05c00700 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a48814d90 0x562a488171b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.840+0000 7f4e05c00700 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a48814d90 0x562a488171b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7979c1e2eb6af977 server_cookie=1c751346a2b5465e in_seq=0 out_seq=0 2024-04-05T23:35:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.840+0000 7f4de7400700 10 client.26026 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.840+0000 7f4de7400700 1 -- 172.21.15.16:0/1920225519 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562a48545cf0 con 0x562a48814d90 2024-04-05T23:35:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4de7400700 10 client.26026 handle_command_reply: tid=0 2024-04-05T23:35:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 client.26026 shutdown 2024-04-05T23:35:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1920225519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a6ca0 msgr2=0x562a487a93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a6ca0 0x562a487a93e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x562a487c00b0 tx=0x7f4dec0999d0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1920225519 shutdown_connections 2024-04-05T23:35:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a48814d90 0x562a488171b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a487a62e0 0x562a487a8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a487ac180 0x562a487a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4de7400700 0 client.26026 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:25.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 --2- 172.21.15.16:0/1920225519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a487a6ca0 0x562a487a93e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1920225519 >> 172.21.15.16:0/1920225519 conn(0x562a4871a220 msgr2=0x562a487aea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f0e6aa85b80 1 Processor -- start 2024-04-05T23:35:25.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1920225519 shutdown_connections 2024-04-05T23:35:25.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 1 -- 172.21.15.16:0/1920225519 wait complete. 2024-04-05T23:35:25.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f4e1c220b80 20 client.26026 trim_cache size 0 max 16384 2024-04-05T23:35:25.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.841+0000 7f0e6aa85b80 1 -- start start 2024-04-05T23:35:25.847 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-04-05T23:35:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 -- 172.21.15.16:0/18916890 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4de80a1290 msgr2=0x7f4de80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4de80a1290 0x7f4de80a3750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4e001a43d0 tx=0x7f4df8004ad0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 -- 172.21.15.16:0/18916890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073ba0 msgr2=0x7f4e001a3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073ba0 0x7f4e001a3c10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e9a0 tx=0x7f4dec00ecb0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e4cc00700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e340a1310 0x7f0e340a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ccf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cc9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1c27b80 con 0x55dfe1ccf370 2024-04-05T23:35:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1c18930 con 0x55dfe1ccfc90 2024-04-05T23:35:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1c162f0 con 0x55dfe1cc9d80 2024-04-05T23:35:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e4cc00700 1 -- 172.21.15.16:0/1978428198 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f0e480cc7f0 con 0x7f0e5019dde0 2024-04-05T23:35:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 -- 172.21.15.16:0/18916890 shutdown_connections 2024-04-05T23:35:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4de80a1290 0x7f4de80a3750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e00111e40 0x7f4e0019d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00073280 0x7f4e001a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 --2- 172.21.15.16:0/18916890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00073ba0 0x7f4e001a3c10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 -- 172.21.15.16:0/18916890 >> 172.21.15.16:0/18916890 conn(0x7f4e000fc900 msgr2=0x7f4e000fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 -- 172.21.15.16:0/18916890 shutdown_connections 2024-04-05T23:35:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e55000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e4f400700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e340a1310 0x7f0e340a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e55000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:47944/0 (socket says 172.21.15.16:47944) 2024-04-05T23:35:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f0e55000700 1 -- 172.21.15.16:0/3021567433 learned_addr learned my addr 172.21.15.16:0/3021567433 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:35:25.842+0000 7f4e08600700 1 -- 172.21.15.16:0/18916890 wait complete. 2024-04-05T23:35:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e4f400700 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cc9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ccf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e55000700 1 -- 172.21.15.16:0/3021567433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 msgr2=0x55dfe1cc9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e55000700 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cc9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e55000700 1 -- 172.21.15.16:0/3021567433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 msgr2=0x55dfe1ccf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e55000700 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ccf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e55000700 1 -- 172.21.15.16:0/3021567433 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfe1a93350 con 0x55dfe1cc9d80 2024-04-05T23:35:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e4f400700 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cc9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:35:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ccf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:35:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e55000700 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cca1e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0e480a1820 tx=0x7f0e48099730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8fedfc48630d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e4f400700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e340a1310 0x7f0e340a37d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e5010e100 tx=0x7f0e3c00b5b0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e45600700 1 -- 172.21.15.16:0/3021567433 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48004ad0 con 0x55dfe1cc9d80 2024-04-05T23:35:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e45600700 1 -- 172.21.15.16:0/3021567433 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e4801b320 con 0x55dfe1cc9d80 2024-04-05T23:35:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.843+0000 7f0e45600700 1 -- 172.21.15.16:0/3021567433 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48097d00 con 0x55dfe1cc9d80 2024-04-05T23:35:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.844+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/3021567433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 msgr2=0x55dfe1cca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.844+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cca1e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0e480a1820 tx=0x7f0e48099730 comp rx=0 tx=0).stop 2024-04-05T23:35:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.844+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/3021567433 shutdown_connections 2024-04-05T23:35:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.844+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ccf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.844+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cca1e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.844+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/3021567433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cc9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.844+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/3021567433 >> 172.21.15.16:0/3021567433 conn(0x55dfe1c351c0 msgr2=0x55dfe1cbbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.844+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/3021567433 shutdown_connections 2024-04-05T23:35:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.844+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/3021567433 wait complete. 2024-04-05T23:35:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.845+0000 7f0e6aa85b80 1 -- start start 2024-04-05T23:35:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.845+0000 7f0e6aa85b80 1 Processor -- start 2024-04-05T23:35:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 0x55dfe1cd2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cd36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1c11be0 con 0x55dfe1cc9d80 2024-04-05T23:35:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1badd20 con 0x55dfe1ccf370 2024-04-05T23:35:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1bad9c0 con 0x55dfe1ccfc90 2024-04-05T23:35:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e6aa85b80 10 client.0 _ll_register_callbacks cb 0x55dfe1a19340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e4fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e4fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36842/0 (socket says 172.21.15.16:36842) 2024-04-05T23:35:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e4fe00700 1 -- 172.21.15.16:0/1302056588 learned_addr learned my addr 172.21.15.16:0/1302056588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e4f400700 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 0x55dfe1cd2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.846+0000 7f0e55000700 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cd36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e4fe00700 1 -- 172.21.15.16:0/1302056588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1ccfc90 msgr2=0x55dfe1cd36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cd36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e4fe00700 1 -- 172.21.15.16:0/1302056588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 msgr2=0x55dfe1cd2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 0x55dfe1cd2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e4fe00700 1 -- 172.21.15.16:0/1302056588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e501a6430 con 0x55dfe1cc9d80 2024-04-05T23:35:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd5d00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0e40012950 tx=0x7f0e40012c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c886da74f8d2a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e3a000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e3a000700 1 -- 172.21.15.16:0/1302056588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40010cd0 con 0x55dfe1cc9d80 2024-04-05T23:35:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e3a000700 1 -- 172.21.15.16:0/1302056588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f0e40010e30 con 0x55dfe1cc9d80 2024-04-05T23:35:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/1302056588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dfe1cb41e0 con 0x55dfe1cc9d80 2024-04-05T23:35:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e3a000700 1 -- 172.21.15.16:0/1302056588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e4001a430 con 0x55dfe1cc9d80 2024-04-05T23:35:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e6aa85b80 10 client.15960 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.847+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/1302056588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55dfe1a3fb00 con 0x55dfe1cc9d80 2024-04-05T23:35:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.848+0000 7f0e3a000700 1 -- 172.21.15.16:0/1302056588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e40027020 con 0x55dfe1cc9d80 2024-04-05T23:35:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.848+0000 7f0e3a000700 1 -- 172.21.15.16:0/1302056588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f0e4000e6f0 con 0x55dfe1cc9d80 2024-04-05T23:35:25.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.848+0000 7f0e6aa85b80 10 client.15960 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.848+0000 7f0e6aa85b80 10 client.15960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:25.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.848+0000 7f0e6aa85b80 20 client.15960 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.848+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dfe1ce3720 0x55dfe1ce5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.848+0000 7f0e6aa85b80 4 client.15960 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:25.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.848+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/1302056588 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dfe1c0a6a0 con 0x55dfe1ce3720 2024-04-05T23:35:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.848+0000 7f0e4f400700 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dfe1ce3720 0x55dfe1ce5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.849+0000 7f0e4f400700 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dfe1ce3720 0x55dfe1ce5b40 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bbd4481bec66cd0 server_cookie=f5ce85d8bc348a00 in_seq=0 out_seq=0 2024-04-05T23:35:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.849+0000 7f0e3a000700 10 client.15960 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e3a000700 1 -- 172.21.15.16:0/1302056588 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dfe1c0a6a0 con 0x55dfe1ce3720 2024-04-05T23:35:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e3a000700 10 client.15960 handle_command_reply: tid=0 2024-04-05T23:35:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 client.15960 shutdown 2024-04-05T23:35:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/1302056588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1cc9d80 msgr2=0x55dfe1cd5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd5d00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0e40012950 tx=0x7f0e40012c60 comp rx=0 tx=0).stop 2024-04-05T23:35:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/1302056588 shutdown_connections 2024-04-05T23:35:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dfe1ce3720 0x55dfe1ce5b40 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd5d00 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cd36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/1302056588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 0x55dfe1cd2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e3a000700 0 client.15960 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/1302056588 >> 172.21.15.16:0/1302056588 conn(0x55dfe1c351c0 msgr2=0x55dfe1cbb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/1302056588 shutdown_connections 2024-04-05T23:35:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.850+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/1302056588 wait complete. 2024-04-05T23:35:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.851+0000 7f0e6aa85b80 20 client.15960 trim_cache size 0 max 16384 2024-04-05T23:35:25.877 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 283048517, 'ino': 1099511628322, 'path': '/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP'}, {'damage_type': 'backtrace', 'id': 2136576956, 'ino': 1099511628323, 'path': '/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS'}, {'damage_type': 'backtrace', 'id': 2408267209, 'ino': 1099511628324, 'path': '/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP'}, {'damage_type': 'backtrace', 'id': 2573444147, 'ino': 1099511628327, 'path': '/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC'}, {'damage_type': 'backtrace', 'id': 3084814481, 'ino': 1099511628328, 'path': '/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC'}, {'damage_type': 'backtrace', 'id': 3667539661, 'ino': 1099511628326, 'path': '/volumes/qa/sv_1/10ec3c0c-949c-4b3b-8e27-cecc0f3357fa/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS'}] 2024-04-05T23:35:25.878 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-04-05T23:35:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.866+0000 7f0e6aa85b80 1 Processor -- start 2024-04-05T23:35:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.866+0000 7f0e6aa85b80 1 -- start start 2024-04-05T23:35:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ce5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cca160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1a95640 con 0x55dfe1ccf370 2024-04-05T23:35:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1ccd940 con 0x55dfe1ccfc90 2024-04-05T23:35:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1cdf7d0 con 0x55dfe1cc9d80 2024-04-05T23:35:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e4f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ce5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e4f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ce5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36858/0 (socket says 172.21.15.16:36858) 2024-04-05T23:35:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e4f400700 1 -- 172.21.15.16:0/2391075136 learned_addr learned my addr 172.21.15.16:0/2391075136 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e55000700 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e4f400700 1 -- 172.21.15.16:0/2391075136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 msgr2=0x55dfe1cd36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e4f400700 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e4f400700 1 -- 172.21.15.16:0/2391075136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 msgr2=0x55dfe1cca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e4f400700 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e4f400700 1 -- 172.21.15.16:0/2391075136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfe1bd7e40 con 0x55dfe1ccf370 2024-04-05T23:35:25.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.867+0000 7f0e55000700 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.868+0000 7f0e4f400700 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ce5b10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0e5006a630 tx=0x7f0e3c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a508de803e6e8779 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.868+0000 7f0e46000700 1 -- 172.21.15.16:0/2391075136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c02f040 con 0x55dfe1ccf370 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.868+0000 7f0e46000700 1 -- 172.21.15.16:0/2391075136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e3c01f050 con 0x55dfe1ccf370 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.868+0000 7f0e46000700 1 -- 172.21.15.16:0/2391075136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c01d030 con 0x55dfe1ccf370 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2391075136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 msgr2=0x55dfe1ce5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ce5b10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0e5006a630 tx=0x7f0e3c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2391075136 shutdown_connections 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2391075136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccf370 0x55dfe1ce5b10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2391075136 >> 172.21.15.16:0/2391075136 conn(0x55dfe1c351c0 msgr2=0x55dfe1cea460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2391075136 shutdown_connections 2024-04-05T23:35:25.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2391075136 wait complete. 2024-04-05T23:35:25.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.869+0000 7f0e6aa85b80 1 -- start start 2024-04-05T23:35:25.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.870+0000 7f0e6aa85b80 1 Processor -- start 2024-04-05T23:35:25.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.870+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.870+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 0x55dfe1cd9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.870+0000 7f0e6aa85b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cda0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.870+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1ccaf40 con 0x55dfe1ccfc90 2024-04-05T23:35:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.870+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1ccb0b0 con 0x55dfe1ccf370 2024-04-05T23:35:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.870+0000 7f0e6aa85b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfe1ccb270 con 0x55dfe1cc9d80 2024-04-05T23:35:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.870+0000 7f0e6aa85b80 10 client.0 _ll_register_callbacks cb 0x7f0e40009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e55000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cda0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e4fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e4f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 0x55dfe1cd9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e55000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cda0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36864/0 (socket says 172.21.15.16:36864) 2024-04-05T23:35:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e55000700 1 -- 172.21.15.16:0/2139282321 learned_addr learned my addr 172.21.15.16:0/2139282321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e55000700 1 -- 172.21.15.16:0/2139282321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 msgr2=0x55dfe1cd9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e55000700 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e55000700 1 -- 172.21.15.16:0/2139282321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 msgr2=0x55dfe1cd9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:25.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e55000700 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 0x55dfe1cd9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e55000700 1 -- 172.21.15.16:0/2139282321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dfe1a93350 con 0x55dfe1ccfc90 2024-04-05T23:35:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e4f400700 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 0x55dfe1cd9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e55000700 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cda0b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0e480090a0 tx=0x7f0e480995a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109c48dd4676bb38 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.871+0000 7f0e3a000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:25.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e3a000700 1 -- 172.21.15.16:0/2139282321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e480ca3a0 con 0x55dfe1ccfc90 2024-04-05T23:35:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2139282321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e4801b320 con 0x55dfe1ccfc90 2024-04-05T23:35:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e3a000700 1 -- 172.21.15.16:0/2139282321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f0e480a1d50 con 0x55dfe1ccfc90 2024-04-05T23:35:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e3a000700 1 -- 172.21.15.16:0/2139282321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e48099960 con 0x55dfe1ccfc90 2024-04-05T23:35:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e6aa85b80 10 client.15966 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2139282321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7f0e3c020040 con 0x55dfe1ccfc90 2024-04-05T23:35:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e3a000700 1 -- 172.21.15.16:0/2139282321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e4801b320 con 0x55dfe1ccfc90 2024-04-05T23:35:25.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e3a000700 1 -- 172.21.15.16:0/2139282321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f0e48099bc0 con 0x55dfe1ccfc90 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e6aa85b80 10 client.15966 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e6aa85b80 10 client.15966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e6aa85b80 20 client.15966 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dfe1ccc640 0x55dfe1d3a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e6aa85b80 4 client.15966 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.872+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2139282321 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55dfe1a60c90 con 0x55dfe1ccc640 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.873+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dfe1ccc640 0x55dfe1d3a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.873+0000 7f0e4fe00700 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dfe1ccc640 0x55dfe1d3a530 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b64f119da8e1b9de server_cookie=b9ca152e9d43f85a in_seq=0 out_seq=0 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.873+0000 7f0e3a000700 10 client.15966 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e3a000700 1 -- 172.21.15.16:0/2139282321 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55dfe1a60c90 con 0x55dfe1ccc640 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e3a000700 10 client.15966 handle_command_reply: tid=0 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 client.15966 shutdown 2024-04-05T23:35:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2139282321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccfc90 msgr2=0x55dfe1cda0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cda0b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0e480090a0 tx=0x7f0e480995a0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2139282321 shutdown_connections 2024-04-05T23:35:25.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55dfe1ccc640 0x55dfe1d3a530 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dfe1ccfc90 0x55dfe1cda0b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfe1cc9d80 0x55dfe1cd9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 --2- 172.21.15.16:0/2139282321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfe1ccf370 0x55dfe1cd9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e3a000700 0 client.15966 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2139282321 >> 172.21.15.16:0/2139282321 conn(0x55dfe1c351c0 msgr2=0x55dfe1cea460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2139282321 shutdown_connections 2024-04-05T23:35:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 1 -- 172.21.15.16:0/2139282321 wait complete. 2024-04-05T23:35:25.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.896+0000 7f0e6aa85b80 20 client.15966 trim_cache size 0 max 16384 2024-04-05T23:35:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:35:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:35:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:35:25.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:35:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 -- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e340a1310 msgr2=0x7f0e340a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e340a1310 0x7f0e340a37d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e5010e100 tx=0x7f0e3c00b5b0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 -- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5019dde0 msgr2=0x7f0e5019e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5019dde0 0x7f0e5019e260 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800ef30 tx=0x7f0e4800c3a0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 -- 172.21.15.16:0/1978428198 shutdown_connections 2024-04-05T23:35:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e340a1310 0x7f0e340a37d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e5019dde0 0x7f0e5019e260 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e50101020 0x7f0e50199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 --2- 172.21.15.16:0/1978428198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e50106140 0x7f0e50199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 -- 172.21.15.16:0/1978428198 >> 172.21.15.16:0/1978428198 conn(0x7f0e500fc880 msgr2=0x7f0e500fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 -- 172.21.15.16:0/1978428198 shutdown_connections 2024-04-05T23:35:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:25.897+0000 7f0e57000700 1 -- 172.21.15.16:0/1978428198 wait complete. 2024-04-05T23:35:25.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22479 3.06 MB/sec warmup 45 sec latency 4964.162 ms 2024-04-05T23:35:26.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:25 smithi027 ceph-mon[78597]: pgmap v59: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 651 KiB/s wr, 38 op/s 2024-04-05T23:35:26.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:25 smithi016 ceph-mon[38248]: pgmap v59: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 651 KiB/s wr, 38 op/s 2024-04-05T23:35:26.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:25 smithi130 ceph-mon[79417]: pgmap v59: 97 pgs: 97 active+clean; 29 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 651 KiB/s wr, 38 op/s 2024-04-05T23:35:26.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24203 3.19 MB/sec warmup 46 sec latency 391.269 ms 2024-04-05T23:35:27.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24203 3.13 MB/sec warmup 47 sec latency 1391.409 ms 2024-04-05T23:35:28.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:27 smithi027 ceph-mon[78597]: pgmap v60: 97 pgs: 97 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 663 KiB/s wr, 45 op/s 2024-04-05T23:35:28.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:27 smithi016 ceph-mon[38248]: pgmap v60: 97 pgs: 97 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 663 KiB/s wr, 45 op/s 2024-04-05T23:35:28.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:27 smithi130 ceph-mon[79417]: pgmap v60: 97 pgs: 97 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 663 KiB/s wr, 45 op/s 2024-04-05T23:35:28.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24203 3.06 MB/sec warmup 48 sec latency 2391.483 ms 2024-04-05T23:35:29.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24203 3.00 MB/sec warmup 49 sec latency 3391.596 ms 2024-04-05T23:35:30.219 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:29 smithi027 ceph-mon[78597]: pgmap v61: 97 pgs: 97 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 KiB/s wr, 16 op/s 2024-04-05T23:35:30.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:29 smithi016 ceph-mon[38248]: pgmap v61: 97 pgs: 97 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 KiB/s wr, 16 op/s 2024-04-05T23:35:30.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:29 smithi130 ceph-mon[79417]: pgmap v61: 97 pgs: 97 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 KiB/s wr, 16 op/s 2024-04-05T23:35:30.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24225 2.95 MB/sec warmup 50 sec latency 3499.899 ms 2024-04-05T23:35:31.984 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24225 2.89 MB/sec warmup 51 sec latency 1873.583 ms 2024-04-05T23:35:32.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:31 smithi027 ceph-mon[78597]: pgmap v62: 97 pgs: 97 active+clean; 30 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 72 KiB/s wr, 26 op/s 2024-04-05T23:35:32.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:31 smithi016 ceph-mon[38248]: pgmap v62: 97 pgs: 97 active+clean; 30 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 72 KiB/s wr, 26 op/s 2024-04-05T23:35:32.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:31 smithi130 ceph-mon[79417]: pgmap v62: 97 pgs: 97 active+clean; 30 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 72 KiB/s wr, 26 op/s 2024-04-05T23:35:32.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24254 2.85 MB/sec warmup 52 sec latency 2255.573 ms 2024-04-05T23:35:33.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24254 2.80 MB/sec warmup 53 sec latency 1597.136 ms 2024-04-05T23:35:34.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:33 smithi027 ceph-mon[78597]: pgmap v63: 97 pgs: 97 active+clean; 31 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 119 KiB/s wr, 38 op/s 2024-04-05T23:35:34.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:33 smithi016 ceph-mon[38248]: pgmap v63: 97 pgs: 97 active+clean; 31 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 119 KiB/s wr, 38 op/s 2024-04-05T23:35:34.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:33 smithi130 ceph-mon[79417]: pgmap v63: 97 pgs: 97 active+clean; 31 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 119 KiB/s wr, 38 op/s 2024-04-05T23:35:34.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24254 2.75 MB/sec warmup 54 sec latency 2597.263 ms 2024-04-05T23:35:35.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24532 2.71 MB/sec warmup 55 sec latency 2709.927 ms 2024-04-05T23:35:36.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:35 smithi027 ceph-mon[78597]: pgmap v64: 97 pgs: 97 active+clean; 31 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 107 KiB/s wr, 31 op/s 2024-04-05T23:35:36.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:35 smithi016 ceph-mon[38248]: pgmap v64: 97 pgs: 97 active+clean; 31 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 107 KiB/s wr, 31 op/s 2024-04-05T23:35:36.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:35 smithi130 ceph-mon[79417]: pgmap v64: 97 pgs: 97 active+clean; 31 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 107 KiB/s wr, 31 op/s 2024-04-05T23:35:36.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24532 2.66 MB/sec warmup 56 sec latency 1751.501 ms 2024-04-05T23:35:37.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24532 2.61 MB/sec warmup 57 sec latency 2751.647 ms 2024-04-05T23:35:38.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:37 smithi027 ceph-mon[78597]: pgmap v65: 97 pgs: 97 active+clean; 32 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 340 KiB/s wr, 49 op/s 2024-04-05T23:35:38.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:37 smithi016 ceph-mon[38248]: pgmap v65: 97 pgs: 97 active+clean; 32 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 340 KiB/s wr, 49 op/s 2024-04-05T23:35:38.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:37 smithi130 ceph-mon[79417]: pgmap v65: 97 pgs: 97 active+clean; 32 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 340 KiB/s wr, 49 op/s 2024-04-05T23:35:38.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24532 2.57 MB/sec warmup 58 sec latency 3751.731 ms 2024-04-05T23:35:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.059+0000 7f41fe000700 1 -- 172.21.15.16:0/663039558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 msgr2=0x7f41f810dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.059+0000 7f41fe000700 1 --2- 172.21.15.16:0/663039558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 0x7f41f810dcd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f41f000b0a0 tx=0x7f41f000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.059+0000 7f41fe000700 1 -- 172.21.15.16:0/663039558 shutdown_connections 2024-04-05T23:35:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.059+0000 7f41fe000700 1 --2- 172.21.15.16:0/663039558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 0x7f41f810dcd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.059+0000 7f41fe000700 1 --2- 172.21.15.16:0/663039558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f8101030 0x7f41f8101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.059+0000 7f41fe000700 1 --2- 172.21.15.16:0/663039558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41f81019e0 0x7f41f8105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.059+0000 7f41fe000700 1 -- 172.21.15.16:0/663039558 >> 172.21.15.16:0/663039558 conn(0x7f41f80fc8b0 msgr2=0x7f41f80fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.059+0000 7f41fe000700 1 -- 172.21.15.16:0/663039558 shutdown_connections 2024-04-05T23:35:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.059+0000 7f41fe000700 1 -- 172.21.15.16:0/663039558 wait complete. 2024-04-05T23:35:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.060+0000 7f41fe000700 1 Processor -- start 2024-04-05T23:35:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.060+0000 7f41fe000700 1 -- start start 2024-04-05T23:35:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41fe000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f8101030 0x7f41f81973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41fe000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 0x7f41f8197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41fe000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41f819bca0 0x7f41f819c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41fe000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f819c720 con 0x7f41f8106150 2024-04-05T23:35:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41fe000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f819c890 con 0x7f41f8101030 2024-04-05T23:35:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41fe000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f819ca50 con 0x7f41f819bca0 2024-04-05T23:35:39.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41efe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 0x7f41f8197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41f7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f8101030 0x7f41f81973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41efe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 0x7f41f8197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55158/0 (socket says 172.21.15.16:55158) 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41efe00700 1 -- 172.21.15.16:0/728924204 learned_addr learned my addr 172.21.15.16:0/728924204 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.061+0000 7f41f7e00700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41f819bca0 0x7f41f819c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41efe00700 1 -- 172.21.15.16:0/728924204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41f819bca0 msgr2=0x7f41f819c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41efe00700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41f819bca0 0x7f41f819c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41efe00700 1 -- 172.21.15.16:0/728924204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f8101030 msgr2=0x7f41f81973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41efe00700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f8101030 0x7f41f81973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41efe00700 1 -- 172.21.15.16:0/728924204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f0009d00 con 0x7f41f8106150 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41f7400700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f8101030 0x7f41f81973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41efe00700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 0x7f41f8197900 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f41e000e970 tx=0x7f41e000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41f4c00700 1 -- 172.21.15.16:0/728924204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0009790 con 0x7f41f8106150 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41f4c00700 1 -- 172.21.15.16:0/728924204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41e000c7e0 con 0x7f41f8106150 2024-04-05T23:35:39.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41f4c00700 1 -- 172.21.15.16:0/728924204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e00054e0 con 0x7f41f8106150 2024-04-05T23:35:39.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41fe000700 1 -- 172.21.15.16:0/728924204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f81a3fb0 con 0x7f41f8106150 2024-04-05T23:35:39.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.062+0000 7f41fe000700 1 -- 172.21.15.16:0/728924204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f81a4430 con 0x7f41f8106150 2024-04-05T23:35:39.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.065+0000 7f41f4c00700 1 -- 172.21.15.16:0/728924204 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f41e000c950 con 0x7f41f8106150 2024-04-05T23:35:39.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.067+0000 7f41f4c00700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f41dc0a1310 0x7f41dc0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.067+0000 7f41f4c00700 1 -- 172.21.15.16:0/728924204 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f41e0014070 con 0x7f41f8106150 2024-04-05T23:35:39.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.067+0000 7f4211b37b80 1 Processor -- start 2024-04-05T23:35:39.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.067+0000 7f41f7400700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f41dc0a1310 0x7f41dc0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.067+0000 7f4211b37b80 1 -- start start 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41f7400700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f41dc0a1310 0x7f41dc0a37d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f41e8011490 tx=0x7f41e8009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ec7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ec2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ec2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f4211b37b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5e28b80 con 0x5584e5ec25c0 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f4211b37b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5e19930 con 0x5584e5ec7b80 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f4211b37b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5e172f0 con 0x5584e5ec84a0 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41f7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ec2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41f7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ec2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55166/0 (socket says 172.21.15.16:55166) 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41f7e00700 1 -- 172.21.15.16:0/302815564 learned_addr learned my addr 172.21.15.16:0/302815564 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41f7400700 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ec7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41efe00700 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ec2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41efe00700 1 -- 172.21.15.16:0/302815564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 msgr2=0x5584e5ec7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41efe00700 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ec7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41efe00700 1 -- 172.21.15.16:0/302815564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 msgr2=0x5584e5ec2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41efe00700 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ec2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41efe00700 1 -- 172.21.15.16:0/302815564 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584e5c94350 con 0x5584e5ec84a0 2024-04-05T23:35:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.068+0000 7f41f7e00700 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ec2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.069+0000 7f41efe00700 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ec2030 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f41e00ca550 tx=0x7f41e000ccd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5625d518c2fcc331 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.069+0000 7f41e7e00700 1 -- 172.21.15.16:0/302815564 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e009dd40 con 0x5584e5ec84a0 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.069+0000 7f41e7e00700 1 -- 172.21.15.16:0/302815564 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41e00a2390 con 0x5584e5ec84a0 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.069+0000 7f41e7e00700 1 -- 172.21.15.16:0/302815564 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e009a170 con 0x5584e5ec84a0 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 -- 172.21.15.16:0/302815564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 msgr2=0x5584e5ec2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ec2030 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f41e00ca550 tx=0x7f41e000ccd0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 -- 172.21.15.16:0/302815564 shutdown_connections 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ec2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ec7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 --2- 172.21.15.16:0/302815564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ec2030 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 -- 172.21.15.16:0/302815564 >> 172.21.15.16:0/302815564 conn(0x5584e5e361c0 msgr2=0x5584e5ebcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 -- 172.21.15.16:0/302815564 shutdown_connections 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 -- 172.21.15.16:0/302815564 wait complete. 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.070+0000 7f4211b37b80 1 -- start start 2024-04-05T23:35:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.071+0000 7f4211b37b80 1 Processor -- start 2024-04-05T23:35:39.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.071+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ece4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ecb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ecbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f4211b37b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5e12be0 con 0x5584e5ec25c0 2024-04-05T23:35:39.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f4211b37b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5daed20 con 0x5584e5ec7b80 2024-04-05T23:35:39.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f4211b37b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5dae9c0 con 0x5584e5ec84a0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f4211b37b80 10 client.0 _ll_register_callbacks cb 0x5584e5c1a340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41f7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ece4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41f7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ece4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55178/0 (socket says 172.21.15.16:55178) 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41f7400700 1 -- 172.21.15.16:0/2246880764 learned_addr learned my addr 172.21.15.16:0/2246880764 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41f7e00700 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ecbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41efe00700 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ecb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41f7400700 1 -- 172.21.15.16:0/2246880764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 msgr2=0x5584e5ecbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41f7400700 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ecbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41f7400700 1 -- 172.21.15.16:0/2246880764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 msgr2=0x5584e5ecb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41f7400700 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ecb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.072+0000 7f41f7400700 1 -- 172.21.15.16:0/2246880764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f810e540 con 0x5584e5ec25c0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.073+0000 7f41f7400700 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ece4e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f41e8003ad0 tx=0x7f41e80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7849d1420b6d06b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.073+0000 7f41e4c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.073+0000 7f41e4c00700 1 -- 172.21.15.16:0/2246880764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8003ea0 con 0x5584e5ec25c0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.073+0000 7f41e4c00700 1 -- 172.21.15.16:0/2246880764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f41e801fa10 con 0x5584e5ec25c0 2024-04-05T23:35:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.073+0000 7f4211b37b80 1 -- 172.21.15.16:0/2246880764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5584e5eb51e0 con 0x5584e5ec25c0 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.073+0000 7f41e4c00700 1 -- 172.21.15.16:0/2246880764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41e801fbd0 con 0x5584e5ec25c0 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.073+0000 7f4211b37b80 10 client.15978 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.073+0000 7f4211b37b80 1 -- 172.21.15.16:0/2246880764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7f41f81a4430 con 0x5584e5ec25c0 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.073+0000 7f41e4c00700 1 -- 172.21.15.16:0/2246880764 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5584e5eb51e0 con 0x5584e5ec25c0 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.074+0000 7f41e4c00700 1 -- 172.21.15.16:0/2246880764 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f41e802f070 con 0x5584e5ec25c0 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.074+0000 7f4211b37b80 10 client.15978 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.074+0000 7f4211b37b80 10 client.15978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.074+0000 7f4211b37b80 20 client.15978 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.074+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5584e5edbfc0 0x5584e5ede3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.074+0000 7f4211b37b80 4 client.15978 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.074+0000 7f4211b37b80 1 -- 172.21.15.16:0/2246880764 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5584e5e0b6a0 con 0x5584e5edbfc0 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.074+0000 7f41efe00700 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5584e5edbfc0 0x5584e5ede3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.075+0000 7f41e4c00700 10 client.15978 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.075+0000 7f41efe00700 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5584e5edbfc0 0x5584e5ede3e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca865eb4cd511de7 server_cookie=d66165e84f50db19 in_seq=0 out_seq=0 2024-04-05T23:35:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.075+0000 7f41e4c00700 1 -- 172.21.15.16:0/2246880764 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5584e5e0b6a0 con 0x5584e5edbfc0 2024-04-05T23:35:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.075+0000 7f41e4c00700 10 client.15978 handle_command_reply: tid=0 2024-04-05T23:35:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.075+0000 7f4211b37b80 1 client.15978 shutdown 2024-04-05T23:35:39.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.075+0000 7f4211b37b80 1 -- 172.21.15.16:0/2246880764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 msgr2=0x5584e5ece4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.075+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ece4e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f41e8003ad0 tx=0x7f41e80047c0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f4211b37b80 1 -- 172.21.15.16:0/2246880764 shutdown_connections 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5584e5edbfc0 0x5584e5ede3e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ece4e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f41e4c00700 0 client.15978 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ecbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2246880764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ecb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f4211b37b80 1 -- 172.21.15.16:0/2246880764 >> 172.21.15.16:0/2246880764 conn(0x5584e5e361c0 msgr2=0x5584e5ebc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f4211b37b80 1 -- 172.21.15.16:0/2246880764 shutdown_connections 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f4211b37b80 1 -- 172.21.15.16:0/2246880764 wait complete. 2024-04-05T23:35:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.076+0000 7f4211b37b80 20 client.15978 trim_cache size 0 max 16384 2024-04-05T23:35:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.091+0000 7f4211b37b80 1 Processor -- start 2024-04-05T23:35:39.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.091+0000 7f4211b37b80 1 -- start start 2024-04-05T23:35:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ebca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ec5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ec5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f4211b37b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5c96640 con 0x5584e5ec25c0 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f4211b37b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5ed9620 con 0x5584e5ec7b80 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f4211b37b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5ec9dc0 con 0x5584e5ec84a0 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41f7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ebca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41f7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ebca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55182/0 (socket says 172.21.15.16:55182) 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41f7400700 1 -- 172.21.15.16:0/2752993687 learned_addr learned my addr 172.21.15.16:0/2752993687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41efe00700 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ec5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41f7e00700 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ec5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41f7400700 1 -- 172.21.15.16:0/2752993687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 msgr2=0x5584e5ec5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41f7400700 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ec5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41f7400700 1 -- 172.21.15.16:0/2752993687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 msgr2=0x5584e5ec5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41f7400700 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ec5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41f7400700 1 -- 172.21.15.16:0/2752993687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584e5c40b00 con 0x5584e5ec25c0 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.092+0000 7f41efe00700 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ec5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f41f7400700 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ebca00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f41e8000c00 tx=0x7f41e8004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f888607e20d07123 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f41ed200700 1 -- 172.21.15.16:0/2752993687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e800e8e0 con 0x5584e5ec25c0 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f41ed200700 1 -- 172.21.15.16:0/2752993687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41e801f6e0 con 0x5584e5ec25c0 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f41ed200700 1 -- 172.21.15.16:0/2752993687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e80286a0 con 0x5584e5ec25c0 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f4211b37b80 1 -- 172.21.15.16:0/2752993687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 msgr2=0x5584e5ebca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ebca00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f41e8000c00 tx=0x7f41e8004160 comp rx=0 tx=0).stop 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f4211b37b80 1 -- 172.21.15.16:0/2752993687 shutdown_connections 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec25c0 0x5584e5ebca00 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec84a0 0x5584e5ec5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.093+0000 7f4211b37b80 1 --2- 172.21.15.16:0/2752993687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec7b80 0x5584e5ec5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.094+0000 7f4211b37b80 1 -- 172.21.15.16:0/2752993687 >> 172.21.15.16:0/2752993687 conn(0x5584e5e361c0 msgr2=0x5584e5ee2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.094+0000 7f4211b37b80 1 -- 172.21.15.16:0/2752993687 shutdown_connections 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.094+0000 7f4211b37b80 1 -- 172.21.15.16:0/2752993687 wait complete. 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.094+0000 7f4211b37b80 1 -- start start 2024-04-05T23:35:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.094+0000 7f4211b37b80 1 Processor -- start 2024-04-05T23:35:39.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec25c0 0x5584e5f2c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec7b80 0x5584e5f2c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f4211b37b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec84a0 0x5584e5f27d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f4211b37b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5f28350 con 0x5584e5ec7b80 2024-04-05T23:35:39.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f4211b37b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5f28490 con 0x5584e5ec84a0 2024-04-05T23:35:39.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f4211b37b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584e5f28650 con 0x5584e5ec25c0 2024-04-05T23:35:39.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f4211b37b80 10 client.0 _ll_register_callbacks cb 0x5584e5c1a340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:39.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41efe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec7b80 0x5584e5f2c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec84a0 0x5584e5f27d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec84a0 0x5584e5f27d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:50686/0 (socket says 172.21.15.16:50686) 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec25c0 0x5584e5f2c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7e00700 1 -- 172.21.15.16:0/1239721275 learned_addr learned my addr 172.21.15.16:0/1239721275 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7400700 1 -- 172.21.15.16:0/1239721275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec84a0 msgr2=0x5584e5f27d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7400700 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec84a0 0x5584e5f27d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7400700 1 -- 172.21.15.16:0/1239721275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec7b80 msgr2=0x5584e5f2c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7400700 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec7b80 0x5584e5f2c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7400700 1 -- 172.21.15.16:0/1239721275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584e5c94350 con 0x5584e5ec25c0 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41f7e00700 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec84a0 0x5584e5f27d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.095+0000 7f41efe00700 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec7b80 0x5584e5f2c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.096+0000 7f41f7400700 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec25c0 0x5584e5f2c380 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009410 tx=0x7f41e8005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b980bf30010f9f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.096+0000 7f41e4c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.096+0000 7f41e4c00700 1 -- 172.21.15.16:0/1239721275 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e800eb20 con 0x5584e5ec25c0 2024-04-05T23:35:39.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.096+0000 7f4211b37b80 1 -- 172.21.15.16:0/1239721275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f41e00a2390 con 0x5584e5ec25c0 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.096+0000 7f41e4c00700 1 -- 172.21.15.16:0/1239721275 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f41e800ec80 con 0x5584e5ec25c0 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.096+0000 7f41e4c00700 1 -- 172.21.15.16:0/1239721275 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41e800ee80 con 0x5584e5ec25c0 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.096+0000 7f4211b37b80 10 client.26056 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.096+0000 7f4211b37b80 1 -- 172.21.15.16:0/1239721275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7f41f81a4430 con 0x5584e5ec25c0 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.096+0000 7f41e4c00700 1 -- 172.21.15.16:0/1239721275 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41e00a2390 con 0x5584e5ec25c0 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f41e4c00700 1 -- 172.21.15.16:0/1239721275 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f41e802f070 con 0x5584e5ec25c0 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f4211b37b80 10 client.26056 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f4211b37b80 10 client.26056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f4211b37b80 20 client.26056 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f4211b37b80 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5584e5f350a0 0x5584e5f374c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f4211b37b80 4 client.26056 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f4211b37b80 1 -- 172.21.15.16:0/1239721275 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5584e5c61c90 con 0x5584e5f350a0 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f41efe00700 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5584e5f350a0 0x5584e5f374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:39.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f41efe00700 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5584e5f350a0 0x5584e5f374c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb2292e8492f4472 server_cookie=37ce25819ec8d879 in_seq=0 out_seq=0 2024-04-05T23:35:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.097+0000 7f41e4c00700 10 client.26056 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f41e4c00700 1 -- 172.21.15.16:0/1239721275 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5584e5c61c90 con 0x5584e5f350a0 2024-04-05T23:35:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f41e4c00700 10 client.26056 handle_command_reply: tid=0 2024-04-05T23:35:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 client.26056 shutdown 2024-04-05T23:35:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 -- 172.21.15.16:0/1239721275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec25c0 msgr2=0x5584e5f2c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec25c0 0x5584e5f2c380 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009410 tx=0x7f41e8005880 comp rx=0 tx=0).stop 2024-04-05T23:35:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 -- 172.21.15.16:0/1239721275 shutdown_connections 2024-04-05T23:35:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5584e5f350a0 0x5584e5f374c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584e5ec84a0 0x5584e5f27d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5584e5ec25c0 0x5584e5f2c380 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f41e4c00700 0 client.26056 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 --2- 172.21.15.16:0/1239721275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5584e5ec7b80 0x5584e5f2c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 -- 172.21.15.16:0/1239721275 >> 172.21.15.16:0/1239721275 conn(0x5584e5e361c0 msgr2=0x5584e5eca040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 -- 172.21.15.16:0/1239721275 shutdown_connections 2024-04-05T23:35:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 1 -- 172.21.15.16:0/1239721275 wait complete. 2024-04-05T23:35:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.119+0000 7f4211b37b80 20 client.26056 trim_cache size 0 max 16384 2024-04-05T23:35:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:35:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:35:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:35:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:35:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 -- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f41dc0a1310 msgr2=0x7f41dc0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f41dc0a1310 0x7f41dc0a37d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f41e8011490 tx=0x7f41e8009260 comp rx=0 tx=0).stop 2024-04-05T23:35:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 -- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 msgr2=0x7f41f8197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 0x7f41f8197900 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f41e000e970 tx=0x7f41e000ec80 comp rx=0 tx=0).stop 2024-04-05T23:35:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 -- 172.21.15.16:0/728924204 shutdown_connections 2024-04-05T23:35:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f41dc0a1310 0x7f41dc0a37d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41f819bca0 0x7f41f819c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f8101030 0x7f41f81973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 --2- 172.21.15.16:0/728924204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8106150 0x7f41f8197900 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 -- 172.21.15.16:0/728924204 >> 172.21.15.16:0/728924204 conn(0x7f41f80fc8b0 msgr2=0x7f41f80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 -- 172.21.15.16:0/728924204 shutdown_connections 2024-04-05T23:35:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:39.120+0000 7f41fe000700 1 -- 172.21.15.16:0/728924204 wait complete. 2024-04-05T23:35:39.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 24532 2.53 MB/sec warmup 59 sec latency 4751.845 ms 2024-04-05T23:35:40.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:39 smithi027 ceph-mon[78597]: pgmap v66: 97 pgs: 97 active+clean; 32 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 329 KiB/s wr, 42 op/s 2024-04-05T23:35:40.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:39 smithi016 ceph-mon[38248]: pgmap v66: 97 pgs: 97 active+clean; 32 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 329 KiB/s wr, 42 op/s 2024-04-05T23:35:40.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:39 smithi130 ceph-mon[79417]: pgmap v66: 97 pgs: 97 active+clean; 32 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 329 KiB/s wr, 42 op/s 2024-04-05T23:35:40.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 25597 2.58 MB/sec warmup 60 sec latency 4864.726 ms 2024-04-05T23:35:41.985 INFO:tasks.workunit.client.0.smithi016.stdout: 1 25665 2.55 MB/sec warmup 61 sec latency 952.913 ms 2024-04-05T23:35:42.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:41 smithi027 ceph-mon[78597]: pgmap v67: 97 pgs: 97 active+clean; 32 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 370 KiB/s wr, 59 op/s 2024-04-05T23:35:42.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:41 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:35:42.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:41 smithi016 ceph-mon[38248]: pgmap v67: 97 pgs: 97 active+clean; 32 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 370 KiB/s wr, 59 op/s 2024-04-05T23:35:42.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:41 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:35:42.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:41 smithi130 ceph-mon[79417]: pgmap v67: 97 pgs: 97 active+clean; 32 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 370 KiB/s wr, 59 op/s 2024-04-05T23:35:42.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:41 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:35:42.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 25665 2.50 MB/sec warmup 62 sec latency 1953.015 ms 2024-04-05T23:35:43.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:42 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:35:43.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:42 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:35:43.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:42 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:35:43.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:42 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:35:43.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:42 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:35:43.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:42 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:35:43.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:42 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:35:43.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:42 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:35:43.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:42 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:35:43.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 25665 2.46 MB/sec warmup 63 sec latency 2953.098 ms 2024-04-05T23:35:44.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:43 smithi027 ceph-mon[78597]: pgmap v68: 97 pgs: 97 active+clean; 28 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 331 KiB/s wr, 54 op/s 2024-04-05T23:35:44.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:43 smithi016 ceph-mon[38248]: pgmap v68: 97 pgs: 97 active+clean; 28 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 331 KiB/s wr, 54 op/s 2024-04-05T23:35:44.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:43 smithi130 ceph-mon[79417]: pgmap v68: 97 pgs: 97 active+clean; 28 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 331 KiB/s wr, 54 op/s 2024-04-05T23:35:44.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 25665 2.43 MB/sec warmup 64 sec latency 3953.238 ms 2024-04-05T23:35:45.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 25818 2.39 MB/sec warmup 65 sec latency 4849.332 ms 2024-04-05T23:35:46.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:46 smithi027 ceph-mon[78597]: pgmap v69: 97 pgs: 97 active+clean; 28 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 284 KiB/s wr, 40 op/s 2024-04-05T23:35:46.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:46 smithi016 ceph-mon[38248]: pgmap v69: 97 pgs: 97 active+clean; 28 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 284 KiB/s wr, 40 op/s 2024-04-05T23:35:46.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:46 smithi130 ceph-mon[79417]: pgmap v69: 97 pgs: 97 active+clean; 28 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 284 KiB/s wr, 40 op/s 2024-04-05T23:35:46.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 28145 2.56 MB/sec warmup 66 sec latency 10.060 ms 2024-04-05T23:35:47.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 29207 2.60 MB/sec warmup 67 sec latency 146.114 ms 2024-04-05T23:35:48.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:48 smithi027 ceph-mon[78597]: pgmap v70: 97 pgs: 97 active+clean; 27 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 338 KiB/s wr, 56 op/s 2024-04-05T23:35:48.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:48 smithi016 ceph-mon[38248]: pgmap v70: 97 pgs: 97 active+clean; 27 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 338 KiB/s wr, 56 op/s 2024-04-05T23:35:48.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:48 smithi130 ceph-mon[79417]: pgmap v70: 97 pgs: 97 active+clean; 27 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 338 KiB/s wr, 56 op/s 2024-04-05T23:35:48.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 29207 2.56 MB/sec warmup 68 sec latency 1146.234 ms 2024-04-05T23:35:49.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 29207 2.53 MB/sec warmup 69 sec latency 2146.360 ms 2024-04-05T23:35:50.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:50 smithi027 ceph-mon[78597]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 104 KiB/s wr, 38 op/s 2024-04-05T23:35:50.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:50 smithi016 ceph-mon[38248]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 104 KiB/s wr, 38 op/s 2024-04-05T23:35:50.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:50 smithi130 ceph-mon[79417]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 104 KiB/s wr, 38 op/s 2024-04-05T23:35:50.986 INFO:tasks.workunit.client.0.smithi016.stdout: 1 29366 2.50 MB/sec warmup 70 sec latency 3041.818 ms 2024-04-05T23:35:51.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 31805 2.65 MB/sec warmup 71 sec latency 12.899 ms 2024-04-05T23:35:52.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.268+0000 7f20d2800700 1 -- 172.21.15.16:0/2098625598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20cc106f40 msgr2=0x7f20cc107320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:52.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.268+0000 7f20d2800700 1 --2- 172.21.15.16:0/2098625598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20cc106f40 0x7f20cc107320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f20bc009990 tx=0x7f20bc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.268+0000 7f20d2800700 1 -- 172.21.15.16:0/2098625598 shutdown_connections 2024-04-05T23:35:52.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.268+0000 7f20d2800700 1 --2- 172.21.15.16:0/2098625598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc068be0 0x7f20cc10dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.268+0000 7f20d2800700 1 --2- 172.21.15.16:0/2098625598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20cc106f40 0x7f20cc107320 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.268+0000 7f20d2800700 1 --2- 172.21.15.16:0/2098625598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20cc068220 0x7f20cc0686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.268+0000 7f20d2800700 1 -- 172.21.15.16:0/2098625598 >> 172.21.15.16:0/2098625598 conn(0x7f20cc076e50 msgr2=0x7f20cc077260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:52.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.268+0000 7f20d2800700 1 -- 172.21.15.16:0/2098625598 shutdown_connections 2024-04-05T23:35:52.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.269+0000 7f20d2800700 1 -- 172.21.15.16:0/2098625598 wait complete. 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.269+0000 7f20d2800700 1 Processor -- start 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20d2800700 1 -- start start 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20d2800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20cc068220 0x7f20cc0ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20d2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc068be0 0x7f20cc100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20d2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20cc106f40 0x7f20cc1007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20d2800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc1a55e0 con 0x7f20cc106f40 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20d2800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc1a5750 con 0x7f20cc068be0 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20d2800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc1a5910 con 0x7f20cc068220 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20cb400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20cc068220 0x7f20cc0ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20caa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc068be0 0x7f20cc100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20cb400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20cc068220 0x7f20cc0ffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:58162/0 (socket says 172.21.15.16:58162) 2024-04-05T23:35:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20caa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc068be0 0x7f20cc100290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52100/0 (socket says 172.21.15.16:52100) 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20cb400700 1 -- 172.21.15.16:0/307921518 learned_addr learned my addr 172.21.15.16:0/307921518 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.270+0000 7f20cbe00700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20cc106f40 0x7f20cc1007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20caa00700 1 -- 172.21.15.16:0/307921518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20cc068220 msgr2=0x7f20cc0ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20caa00700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20cc068220 0x7f20cc0ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20caa00700 1 -- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20cc106f40 msgr2=0x7f20cc1007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20caa00700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20cc106f40 0x7f20cc1007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20caa00700 1 -- 172.21.15.16:0/307921518 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc0095f0 con 0x7f20cc068be0 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20caa00700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc068be0 0x7f20cc100290 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f20b800e9f0 tx=0x7f20b800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20c3e00700 1 -- 172.21.15.16:0/307921518 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b800c9c0 con 0x7f20cc068be0 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20c3e00700 1 -- 172.21.15.16:0/307921518 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20b8010430 con 0x7f20cc068be0 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20d2800700 1 -- 172.21.15.16:0/307921518 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc1a5bf0 con 0x7f20cc068be0 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20c3e00700 1 -- 172.21.15.16:0/307921518 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b8018680 con 0x7f20cc068be0 2024-04-05T23:35:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.271+0000 7f20d2800700 1 -- 172.21.15.16:0/307921518 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc1a6150 con 0x7f20cc068be0 2024-04-05T23:35:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.276+0000 7f20c3e00700 1 -- 172.21.15.16:0/307921518 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f20b8010860 con 0x7f20cc068be0 2024-04-05T23:35:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.277+0000 7f20e63c5b80 1 Processor -- start 2024-04-05T23:35:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.277+0000 7f20c3e00700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f20b00a1310 0x7f20b00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.277+0000 7f20c3e00700 1 -- 172.21.15.16:0/307921518 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f20b8014070 con 0x7f20cc068be0 2024-04-05T23:35:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.277+0000 7f20e63c5b80 1 -- start start 2024-04-05T23:35:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20cb400700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f20b00a1310 0x7f20b00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b872994f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8729954a0 0x55b87298f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b87298f5c0 0x55b87298fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8728f5b80 con 0x55b8729954a0 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8728e6930 con 0x55b872994b80 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8728e42f0 con 0x55b87298f5c0 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20caa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8729954a0 0x55b87298f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20cbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b87298f5c0 0x55b87298fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20cb400700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f20b00a1310 0x7f20b00a37d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f20bc006010 tx=0x7f20bc00b260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20caa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8729954a0 0x55b87298f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45592/0 (socket says 172.21.15.16:45592) 2024-04-05T23:35:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20caa00700 1 -- 172.21.15.16:0/543229156 learned_addr learned my addr 172.21.15.16:0/543229156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.278+0000 7f20cb400700 1 --2- 172.21.15.16:0/543229156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b872994f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20caa00700 1 -- 172.21.15.16:0/543229156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b87298f5c0 msgr2=0x55b87298fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20caa00700 1 --2- 172.21.15.16:0/543229156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b87298f5c0 0x55b87298fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20caa00700 1 -- 172.21.15.16:0/543229156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 msgr2=0x55b872994f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20caa00700 1 --2- 172.21.15.16:0/543229156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b872994f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20caa00700 1 -- 172.21.15.16:0/543229156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b872761350 con 0x55b8729954a0 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20cbe00700 1 --2- 172.21.15.16:0/543229156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b87298f5c0 0x55b87298fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20caa00700 1 --2- 172.21.15.16:0/543229156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8729954a0 0x55b87298f030 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f20b80968d0 tx=0x7f20b80998c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aedf82396c41d1b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20c0c00700 1 -- 172.21.15.16:0/543229156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b809dc90 con 0x55b8729954a0 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20c0c00700 1 -- 172.21.15.16:0/543229156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20b80a22e0 con 0x55b8729954a0 2024-04-05T23:35:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.279+0000 7f20c0c00700 1 -- 172.21.15.16:0/543229156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b809a390 con 0x55b8729954a0 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.280+0000 7f20e63c5b80 1 -- 172.21.15.16:0/543229156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8729954a0 msgr2=0x55b87298f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.280+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/543229156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8729954a0 0x55b87298f030 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f20b80968d0 tx=0x7f20b80998c0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.280+0000 7f20e63c5b80 1 -- 172.21.15.16:0/543229156 shutdown_connections 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.280+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/543229156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b87298f5c0 0x55b87298fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.280+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/543229156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b872994f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.280+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/543229156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8729954a0 0x55b87298f030 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.280+0000 7f20e63c5b80 1 -- 172.21.15.16:0/543229156 >> 172.21.15.16:0/543229156 conn(0x55b8729031c0 msgr2=0x55b872989be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.280+0000 7f20e63c5b80 1 -- 172.21.15.16:0/543229156 shutdown_connections 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.280+0000 7f20e63c5b80 1 -- 172.21.15.16:0/543229156 wait complete. 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.281+0000 7f20e63c5b80 1 -- start start 2024-04-05T23:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.281+0000 7f20e63c5b80 1 Processor -- start 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.281+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87298f5c0 0x55b87299b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b872994b80 0x55b872998780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 0x55b872998e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8728dfbe0 con 0x55b872994b80 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b87287bd20 con 0x55b87298f5c0 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b87287b9c0 con 0x55b8729954a0 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20e63c5b80 10 client.0 _ll_register_callbacks cb 0x55b8726e7340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87298f5c0 0x55b87299b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87298f5c0 0x55b87299b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52108/0 (socket says 172.21.15.16:52108) 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cb400700 1 -- 172.21.15.16:0/1480712840 learned_addr learned my addr 172.21.15.16:0/1480712840 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cbe00700 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 0x55b872998e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20caa00700 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b872994b80 0x55b872998780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cb400700 1 -- 172.21.15.16:0/1480712840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 msgr2=0x55b872998e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cb400700 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 0x55b872998e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cb400700 1 -- 172.21.15.16:0/1480712840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b872994b80 msgr2=0x55b872998780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cb400700 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b872994b80 0x55b872998780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cb400700 1 -- 172.21.15.16:0/1480712840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c4011230 con 0x55b87298f5c0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.282+0000 7f20cb400700 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87298f5c0 0x55b87299b490 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00bef0 tx=0x7f20bc01a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1048b77ceaea537d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20b5600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20b5600700 1 -- 172.21.15.16:0/1480712840 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc00f660 con 0x55b87298f5c0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20b5600700 1 -- 172.21.15.16:0/1480712840 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f20bc004e80 con 0x55b87298f5c0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1480712840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b8729821e0 con 0x55b87298f5c0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20b5600700 1 -- 172.21.15.16:0/1480712840 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20bc00f7c0 con 0x55b87298f5c0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20e63c5b80 10 client.26033 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1480712840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7f20cc1a6150 con 0x55b87298f5c0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20b5600700 1 -- 172.21.15.16:0/1480712840 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20bc03c020 con 0x55b87298f5c0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20b5600700 1 -- 172.21.15.16:0/1480712840 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f20bc024910 con 0x55b87298f5c0 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20e63c5b80 10 client.26033 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.283+0000 7f20e63c5b80 10 client.26033 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.284+0000 7f20e63c5b80 20 client.26033 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.284+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b8729a8ef0 0x55b8729ab310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.284+0000 7f20e63c5b80 4 client.26033 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.284+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1480712840 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8728d86a0 con 0x55b8729a8ef0 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.284+0000 7f20caa00700 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b8729a8ef0 0x55b8729ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.284+0000 7f20caa00700 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b8729a8ef0 0x55b8729ab310 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e11f78b22deb3cb server_cookie=3965a9d41d4b931a in_seq=0 out_seq=0 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.284+0000 7f20b5600700 10 client.26033 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20b5600700 1 -- 172.21.15.16:0/1480712840 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b8728d86a0 con 0x55b8729a8ef0 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20b5600700 10 client.26033 handle_command_reply: tid=0 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 client.26033 shutdown 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1480712840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87298f5c0 msgr2=0x55b87299b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87298f5c0 0x55b87299b490 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00bef0 tx=0x7f20bc01a050 comp rx=0 tx=0).stop 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1480712840 shutdown_connections 2024-04-05T23:35:52.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b8729a8ef0 0x55b8729ab310 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 0x55b872998e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b87298f5c0 0x55b87299b490 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1480712840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b872994b80 0x55b872998780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20b5600700 0 client.26033 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1480712840 >> 172.21.15.16:0/1480712840 conn(0x55b8729031c0 msgr2=0x55b872989280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1480712840 shutdown_connections 2024-04-05T23:35:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.285+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1480712840 wait complete. 2024-04-05T23:35:52.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.286+0000 7f20e63c5b80 20 client.26033 trim_cache size 0 max 16384 2024-04-05T23:35:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.302+0000 7f20e63c5b80 1 Processor -- start 2024-04-05T23:35:52.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.302+0000 7f20e63c5b80 1 -- start start 2024-04-05T23:35:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87298f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b872994b80 0x55b87299fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b8729954a0 0x55b872989a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b872763640 con 0x55b87298f5c0 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8729a03d0 con 0x55b8729954a0 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b872999400 con 0x55b872994b80 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87298f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b8729954a0 0x55b872989a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87298f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45612/0 (socket says 172.21.15.16:45612) 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20caa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b872994b80 0x55b87299fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b8729954a0 0x55b872989a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52112/0 (socket says 172.21.15.16:52112) 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cb400700 1 -- 172.21.15.16:0/1650332751 learned_addr learned my addr 172.21.15.16:0/1650332751 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cbe00700 1 -- 172.21.15.16:0/1650332751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b872994b80 msgr2=0x55b87299fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cbe00700 1 --2- 172.21.15.16:0/1650332751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b872994b80 0x55b87299fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cbe00700 1 -- 172.21.15.16:0/1650332751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 msgr2=0x55b87298f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cbe00700 1 --2- 172.21.15.16:0/1650332751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87298f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cbe00700 1 -- 172.21.15.16:0/1650332751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b87270db00 con 0x55b8729954a0 2024-04-05T23:35:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.303+0000 7f20cb400700 1 --2- 172.21.15.16:0/1650332751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87298f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20cbe00700 1 --2- 172.21.15.16:0/1650332751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b8729954a0 0x55b872989a00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20c4003ab0 tx=0x7f20c4009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca087171b8813f08 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20c1600700 1 -- 172.21.15.16:0/1650332751 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c400e040 con 0x55b8729954a0 2024-04-05T23:35:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20c1600700 1 -- 172.21.15.16:0/1650332751 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20c400f040 con 0x55b8729954a0 2024-04-05T23:35:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20c1600700 1 -- 172.21.15.16:0/1650332751 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c401f690 con 0x55b8729954a0 2024-04-05T23:35:52.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1650332751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b8729954a0 msgr2=0x55b872989a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:52.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1650332751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b8729954a0 0x55b872989a00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20c4003ab0 tx=0x7f20c4009920 comp rx=0 tx=0).stop 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1650332751 shutdown_connections 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1650332751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87298f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1650332751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b8729954a0 0x55b872989a00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1650332751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b872994b80 0x55b87299fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.304+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1650332751 >> 172.21.15.16:0/1650332751 conn(0x55b8729031c0 msgr2=0x55b872997050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.305+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1650332751 shutdown_connections 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.305+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1650332751 wait complete. 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.305+0000 7f20e63c5b80 1 -- start start 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.305+0000 7f20e63c5b80 1 Processor -- start 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87299fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b87299c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20e63c5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 0x55b87299cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b87299d290 con 0x55b87298f5c0 2024-04-05T23:35:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8729fcd20 con 0x55b872994b80 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20e63c5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8729fce60 con 0x55b8729954a0 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20e63c5b80 10 client.0 _ll_register_callbacks cb 0x7f20bc00baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20caa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b87299c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87299fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20caa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b87299c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52124/0 (socket says 172.21.15.16:52124) 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20caa00700 1 -- 172.21.15.16:0/1153542116 learned_addr learned my addr 172.21.15.16:0/1153542116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20cbe00700 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 0x55b87299cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.306+0000 7f20caa00700 1 -- 172.21.15.16:0/1153542116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 msgr2=0x55b87299cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20caa00700 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 0x55b87299cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20caa00700 1 -- 172.21.15.16:0/1153542116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 msgr2=0x55b87299fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20caa00700 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87299fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20caa00700 1 -- 172.21.15.16:0/1153542116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc1a6150 con 0x55b872994b80 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20cb400700 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87299fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20cbe00700 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 0x55b87299cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:35:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20caa00700 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b87299c490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f20b800c5b0 tx=0x7f20b8009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22ae149e1adf768b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20b5600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20b5600700 1 -- 172.21.15.16:0/1153542116 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b80b8170 con 0x55b872994b80 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20b5600700 1 -- 172.21.15.16:0/1153542116 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f20b8099350 con 0x55b872994b80 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1153542116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20b80a22e0 con 0x55b872994b80 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20b5600700 1 -- 172.21.15.16:0/1153542116 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20b8099510 con 0x55b872994b80 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20e63c5b80 10 client.26039 fetch_fsmap learned FSMap version 37 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1153542116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7f20c4009310 con 0x55b872994b80 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.307+0000 7f20b5600700 1 -- 172.21.15.16:0/1153542116 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20b80a22e0 con 0x55b872994b80 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20b5600700 1 -- 172.21.15.16:0/1153542116 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f20b80a1e50 con 0x55b872994b80 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20e63c5b80 10 client.26039 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20e63c5b80 10 client.26039 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20e63c5b80 20 client.26039 populate_metadata read hostname 'smithi016' 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b8729fe360 0x55b872a00780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:35:52.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20e63c5b80 4 client.26039 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:35:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1153542116 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b87272ec90 con 0x55b8729fe360 2024-04-05T23:35:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20cb400700 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b8729fe360 0x55b872a00780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:35:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20cb400700 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b8729fe360 0x55b872a00780 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f90064e7adc3b49d server_cookie=d845842e16bf2da0 in_seq=0 out_seq=0 2024-04-05T23:35:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.308+0000 7f20b5600700 10 client.26039 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.333+0000 7f20b5600700 1 -- 172.21.15.16:0/1153542116 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b87272ec90 con 0x55b8729fe360 2024-04-05T23:35:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.333+0000 7f20b5600700 10 client.26039 handle_command_reply: tid=0 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.333+0000 7f20e63c5b80 1 client.26039 shutdown 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1153542116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 msgr2=0x55b87299c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b87299c490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f20b800c5b0 tx=0x7f20b8009ef0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1153542116 shutdown_connections 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b8729fe360 0x55b872a00780 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b87298f5c0 0x55b87299fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b8729954a0 0x55b87299cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 --2- 172.21.15.16:0/1153542116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b872994b80 0x55b87299c490 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20b5600700 0 client.26039 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1153542116 >> 172.21.15.16:0/1153542116 conn(0x55b8729031c0 msgr2=0x55b87299a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1153542116 shutdown_connections 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 1 -- 172.21.15.16:0/1153542116 wait complete. 2024-04-05T23:35:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.334+0000 7f20e63c5b80 20 client.26039 trim_cache size 0 max 16384 2024-04-05T23:35:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:35:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:35:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:35:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 -- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f20b00a1310 msgr2=0x7f20b00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f20b00a1310 0x7f20b00a37d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f20bc006010 tx=0x7f20bc00b260 comp rx=0 tx=0).stop 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 -- 172.21.15.16:0/307921518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc068be0 msgr2=0x7f20cc100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc068be0 0x7f20cc100290 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f20b800e9f0 tx=0x7f20b800ed00 comp rx=0 tx=0).stop 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 -- 172.21.15.16:0/307921518 shutdown_connections 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f20b00a1310 0x7f20b00a37d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20cc106f40 0x7f20cc1007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20cc068220 0x7f20cc0ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 --2- 172.21.15.16:0/307921518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc068be0 0x7f20cc100290 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 -- 172.21.15.16:0/307921518 >> 172.21.15.16:0/307921518 conn(0x7f20cc076e50 msgr2=0x7f20cc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 -- 172.21.15.16:0/307921518 shutdown_connections 2024-04-05T23:35:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:35:52.335+0000 7f20d2800700 1 -- 172.21.15.16:0/307921518 wait complete. 2024-04-05T23:35:52.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:52 smithi027 ceph-mon[78597]: pgmap v72: 97 pgs: 97 active+clean; 29 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 289 KiB/s wr, 72 op/s 2024-04-05T23:35:52.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:52 smithi016 ceph-mon[38248]: pgmap v72: 97 pgs: 97 active+clean; 29 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 289 KiB/s wr, 72 op/s 2024-04-05T23:35:52.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:52 smithi130 ceph-mon[79417]: pgmap v72: 97 pgs: 97 active+clean; 29 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 289 KiB/s wr, 72 op/s 2024-04-05T23:35:52.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 33100 2.71 MB/sec warmup 72 sec latency 19.370 ms 2024-04-05T23:35:53.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 35649 2.89 MB/sec warmup 73 sec latency 10.341 ms 2024-04-05T23:35:54.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:54 smithi027 ceph-mon[78597]: pgmap v73: 97 pgs: 97 active+clean; 29 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 289 KiB/s wr, 69 op/s 2024-04-05T23:35:54.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:54 smithi016 ceph-mon[38248]: pgmap v73: 97 pgs: 97 active+clean; 29 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 289 KiB/s wr, 69 op/s 2024-04-05T23:35:54.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:54 smithi130 ceph-mon[79417]: pgmap v73: 97 pgs: 97 active+clean; 29 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 289 KiB/s wr, 69 op/s 2024-04-05T23:35:54.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 37038 2.94 MB/sec warmup 74 sec latency 11.784 ms 2024-04-05T23:35:55.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 39323 3.08 MB/sec warmup 75 sec latency 10.787 ms 2024-04-05T23:35:56.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:56 smithi027 ceph-mon[78597]: pgmap v74: 97 pgs: 97 active+clean; 29 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 280 KiB/s wr, 63 op/s 2024-04-05T23:35:56.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:56 smithi016 ceph-mon[38248]: pgmap v74: 97 pgs: 97 active+clean; 29 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 280 KiB/s wr, 63 op/s 2024-04-05T23:35:56.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:56 smithi130 ceph-mon[79417]: pgmap v74: 97 pgs: 97 active+clean; 29 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 280 KiB/s wr, 63 op/s 2024-04-05T23:35:56.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 40883 3.11 MB/sec warmup 76 sec latency 37.951 ms 2024-04-05T23:35:57.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 42870 3.25 MB/sec warmup 77 sec latency 38.145 ms 2024-04-05T23:35:58.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:35:58 smithi027 ceph-mon[78597]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 835 KiB/s wr, 166 op/s 2024-04-05T23:35:58.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:35:58 smithi016 ceph-mon[38248]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 835 KiB/s wr, 166 op/s 2024-04-05T23:35:58.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:35:58 smithi130 ceph-mon[79417]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 835 KiB/s wr, 166 op/s 2024-04-05T23:35:58.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 44440 3.28 MB/sec warmup 78 sec latency 15.992 ms 2024-04-05T23:35:59.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 46456 3.41 MB/sec warmup 79 sec latency 12.420 ms 2024-04-05T23:36:00.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:00 smithi016 ceph-mon[38248]: pgmap v76: 97 pgs: 97 active+clean; 35 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 781 KiB/s wr, 150 op/s 2024-04-05T23:36:00.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:00 smithi130 ceph-mon[79417]: pgmap v76: 97 pgs: 97 active+clean; 35 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 781 KiB/s wr, 150 op/s 2024-04-05T23:36:00.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:00 smithi027 ceph-mon[78597]: pgmap v76: 97 pgs: 97 active+clean; 35 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 781 KiB/s wr, 150 op/s 2024-04-05T23:36:00.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 47273 3.39 MB/sec warmup 80 sec latency 23.825 ms 2024-04-05T23:36:01.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 49806 3.55 MB/sec warmup 81 sec latency 10.399 ms 2024-04-05T23:36:02.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:02 smithi016 ceph-mon[38248]: pgmap v77: 97 pgs: 97 active+clean; 39 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 230 op/s 2024-04-05T23:36:02.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:02 smithi130 ceph-mon[79417]: pgmap v77: 97 pgs: 97 active+clean; 39 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 230 op/s 2024-04-05T23:36:02.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:02 smithi027 ceph-mon[78597]: pgmap v77: 97 pgs: 97 active+clean; 39 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 230 op/s 2024-04-05T23:36:02.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 50949 3.55 MB/sec warmup 82 sec latency 13.164 ms 2024-04-05T23:36:03.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 53454 3.70 MB/sec warmup 83 sec latency 38.285 ms 2024-04-05T23:36:04.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:04 smithi016 ceph-mon[38248]: pgmap v78: 97 pgs: 97 active+clean; 40 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 211 op/s 2024-04-05T23:36:04.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:04 smithi130 ceph-mon[79417]: pgmap v78: 97 pgs: 97 active+clean; 40 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 211 op/s 2024-04-05T23:36:04.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:04 smithi027 ceph-mon[78597]: pgmap v78: 97 pgs: 97 active+clean; 40 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 211 op/s 2024-04-05T23:36:04.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 54729 3.72 MB/sec warmup 84 sec latency 11.748 ms 2024-04-05T23:36:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.502+0000 7f36c2200700 1 -- 172.21.15.16:0/2244354252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc101830 msgr2=0x7f36bc105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.502+0000 7f36c2200700 1 --2- 172.21.15.16:0/2244354252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc101830 0x7f36bc105870 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f36a8009990 tx=0x7f36a8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.502+0000 7f36c2200700 1 -- 172.21.15.16:0/2244354252 shutdown_connections 2024-04-05T23:36:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.502+0000 7f36c2200700 1 --2- 172.21.15.16:0/2244354252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc100e80 0x7f36bc101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.502+0000 7f36c2200700 1 --2- 172.21.15.16:0/2244354252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36bc105fa0 0x7f36bc10db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.502+0000 7f36c2200700 1 --2- 172.21.15.16:0/2244354252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc101830 0x7f36bc105870 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.502+0000 7f36c2200700 1 -- 172.21.15.16:0/2244354252 >> 172.21.15.16:0/2244354252 conn(0x7f36bc0fc740 msgr2=0x7f36bc0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.502+0000 7f36c2200700 1 -- 172.21.15.16:0/2244354252 shutdown_connections 2024-04-05T23:36:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.502+0000 7f36c2200700 1 -- 172.21.15.16:0/2244354252 wait complete. 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.503+0000 7f36c2200700 1 Processor -- start 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.503+0000 7f36c2200700 1 -- start start 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c2200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc100e80 0x7f36bc199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c2200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc105fa0 0x7f36bc1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c2200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36bc19dc40 0x7f36bc19e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c2200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc19e6c0 con 0x7f36bc100e80 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c2200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc19e830 con 0x7f36bc105fa0 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c2200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc19e9f0 con 0x7f36bc19dc40 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c0e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc100e80 0x7f36bc199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c1800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36bc19dc40 0x7f36bc19e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c0e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc100e80 0x7f36bc199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38866/0 (socket says 172.21.15.16:38866) 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c0e00700 1 -- 172.21.15.16:0/790645426 learned_addr learned my addr 172.21.15.16:0/790645426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36bbe00700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc105fa0 0x7f36bc1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c1800700 1 -- 172.21.15.16:0/790645426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc105fa0 msgr2=0x7f36bc1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c1800700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc105fa0 0x7f36bc1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c1800700 1 -- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc100e80 msgr2=0x7f36bc199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c1800700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc100e80 0x7f36bc199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c1800700 1 -- 172.21.15.16:0/790645426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a80095f0 con 0x7f36bc19dc40 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.504+0000 7f36c0e00700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc100e80 0x7f36bc199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.505+0000 7f36c1800700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36bc19dc40 0x7f36bc19e0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f36b4007f10 tx=0x7f36b400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.505+0000 7f36b9600700 1 -- 172.21.15.16:0/790645426 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4009ea0 con 0x7f36bc19dc40 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.505+0000 7f36b9600700 1 -- 172.21.15.16:0/790645426 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36b400f040 con 0x7f36bc19dc40 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.505+0000 7f36c2200700 1 -- 172.21.15.16:0/790645426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36bc1a5f50 con 0x7f36bc19dc40 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.505+0000 7f36b9600700 1 -- 172.21.15.16:0/790645426 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4004350 con 0x7f36bc19dc40 2024-04-05T23:36:05.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.505+0000 7f36c2200700 1 -- 172.21.15.16:0/790645426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc1a6420 con 0x7f36bc19dc40 2024-04-05T23:36:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.509+0000 7f36b9600700 1 -- 172.21.15.16:0/790645426 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f36b400db20 con 0x7f36bc19dc40 2024-04-05T23:36:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.510+0000 7f36b9600700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f36a40a1290 0x7f36a40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.510+0000 7f36b9600700 1 -- 172.21.15.16:0/790645426 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f36b40cc510 con 0x7f36bc19dc40 2024-04-05T23:36:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.510+0000 7f36c0e00700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f36a40a1290 0x7f36a40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36d73c4b80 1 Processor -- start 2024-04-05T23:36:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36d73c4b80 1 -- start start 2024-04-05T23:36:05.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36c0e00700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f36a40a1290 0x7f36a40a3750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f36b0011550 tx=0x7f36b0009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:05.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8220 0x55e17ccd8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8b40 0x55e17ccdc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17cc3eb80 con 0x55e17ccd8220 2024-04-05T23:36:05.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17cc2f930 con 0x55e17ccdcec0 2024-04-05T23:36:05.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17cc2d2f0 con 0x55e17ccd8b40 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36c1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.511+0000 7f36c0e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8220 0x55e17ccd8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36bbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8b40 0x55e17ccdc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36c1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:54466/0 (socket says 172.21.15.16:54466) 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36c1800700 1 -- 172.21.15.16:0/1279272565 learned_addr learned my addr 172.21.15.16:0/1279272565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36c1800700 1 -- 172.21.15.16:0/1279272565 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8b40 msgr2=0x55e17ccdc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36c1800700 1 --2- 172.21.15.16:0/1279272565 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8b40 0x55e17ccdc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36c1800700 1 -- 172.21.15.16:0/1279272565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8220 msgr2=0x55e17ccd8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36c1800700 1 --2- 172.21.15.16:0/1279272565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8220 0x55e17ccd8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36c1800700 1 -- 172.21.15.16:0/1279272565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e17caaa350 con 0x55e17ccdcec0 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36c1800700 1 --2- 172.21.15.16:0/1279272565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdd300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f36b40a1540 tx=0x7f36b4095a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=686aa274bff64686 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36ae000700 1 -- 172.21.15.16:0/1279272565 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b40b90d0 con 0x55e17ccdcec0 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36ae000700 1 -- 172.21.15.16:0/1279272565 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36b4099cf0 con 0x55e17ccdcec0 2024-04-05T23:36:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.512+0000 7f36ae000700 1 -- 172.21.15.16:0/1279272565 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4003a90 con 0x55e17ccdcec0 2024-04-05T23:36:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.513+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1279272565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 msgr2=0x55e17ccdd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.513+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1279272565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdd300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f36b40a1540 tx=0x7f36b4095a50 comp rx=0 tx=0).stop 2024-04-05T23:36:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.513+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1279272565 shutdown_connections 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.513+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1279272565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8220 0x55e17ccd8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.513+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1279272565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdd300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.513+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1279272565 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8b40 0x55e17ccdc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.513+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1279272565 >> 172.21.15.16:0/1279272565 conn(0x55e17cc4c1c0 msgr2=0x55e17ccd3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.513+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1279272565 shutdown_connections 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.513+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1279272565 wait complete. 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.514+0000 7f36d73c4b80 1 -- start start 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.514+0000 7f36d73c4b80 1 Processor -- start 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.514+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccd8220 0x55e17cce46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.514+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17cce19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.514+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccdcec0 0x55e17cce2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.514+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17cc28be0 con 0x55e17ccd8b40 2024-04-05T23:36:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.514+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17cbc4d20 con 0x55e17ccd8220 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.514+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17cbc49c0 con 0x55e17ccdcec0 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.514+0000 7f36d73c4b80 10 client.0 _ll_register_callbacks cb 0x55e17ca30340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccd8220 0x55e17cce46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccd8220 0x55e17cce46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:54472/0 (socket says 172.21.15.16:54472) 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c0e00700 1 -- 172.21.15.16:0/2450023175 learned_addr learned my addr 172.21.15.16:0/2450023175 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c1800700 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccdcec0 0x55e17cce2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36bbe00700 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17cce19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c0e00700 1 -- 172.21.15.16:0/2450023175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccdcec0 msgr2=0x55e17cce2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c0e00700 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccdcec0 0x55e17cce2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c0e00700 1 -- 172.21.15.16:0/2450023175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 msgr2=0x55e17cce19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c0e00700 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17cce19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c0e00700 1 -- 172.21.15.16:0/2450023175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc1a6420 con 0x55e17ccd8220 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c1800700 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccdcec0 0x55e17cce2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36c0e00700 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccd8220 0x55e17cce46f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009ec0 tx=0x7f36b00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a1f2fba02a5ad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.515+0000 7f36a2a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.516+0000 7f36a2a00700 1 -- 172.21.15.16:0/2450023175 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b0022070 con 0x55e17ccd8220 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.516+0000 7f36a2a00700 1 -- 172.21.15.16:0/2450023175 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f36b001ddc0 con 0x55e17ccd8220 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.516+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2450023175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e17cccb1e0 con 0x55e17ccd8220 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.516+0000 7f36a2a00700 1 -- 172.21.15.16:0/2450023175 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36b000ee70 con 0x55e17ccd8220 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.516+0000 7f36d73c4b80 10 client.26051 fetch_fsmap learned FSMap version 37 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.516+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2450023175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55e17ca56b00 con 0x55e17ccd8220 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.516+0000 7f36a2a00700 1 -- 172.21.15.16:0/2450023175 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e17cccb1e0 con 0x55e17ccd8220 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.516+0000 7f36a2a00700 1 -- 172.21.15.16:0/2450023175 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f36b0005510 con 0x55e17ccd8220 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.517+0000 7f36d73c4b80 10 client.26051 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:36:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.517+0000 7f36d73c4b80 10 client.26051 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.517+0000 7f36d73c4b80 20 client.26051 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.517+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e17ccf2280 0x55e17ccf46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.517+0000 7f36d73c4b80 4 client.26051 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.517+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2450023175 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e17cc216a0 con 0x55e17ccf2280 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.517+0000 7f36bbe00700 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e17ccf2280 0x55e17ccf46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.517+0000 7f36bbe00700 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e17ccf2280 0x55e17ccf46a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6078e2fc3f1a3c0b server_cookie=cb9ecdcb1f39bf08 in_seq=0 out_seq=0 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.517+0000 7f36a2a00700 10 client.26051 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36a2a00700 1 -- 172.21.15.16:0/2450023175 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e17cc216a0 con 0x55e17ccf2280 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36a2a00700 10 client.26051 handle_command_reply: tid=0 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36d73c4b80 1 client.26051 shutdown 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2450023175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccd8220 msgr2=0x55e17cce46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccd8220 0x55e17cce46f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009ec0 tx=0x7f36b00047c0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2450023175 shutdown_connections 2024-04-05T23:36:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e17ccf2280 0x55e17ccf46a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccdcec0 0x55e17cce2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccd8220 0x55e17cce46f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2450023175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17cce19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36a2a00700 0 client.26051 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.518+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2450023175 >> 172.21.15.16:0/2450023175 conn(0x55e17cc4c1c0 msgr2=0x55e17ccd2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.519+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2450023175 shutdown_connections 2024-04-05T23:36:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.519+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2450023175 wait complete. 2024-04-05T23:36:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.519+0000 7f36d73c4b80 20 client.26051 trim_cache size 0 max 16384 2024-04-05T23:36:05.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.534+0000 7f36d73c4b80 1 Processor -- start 2024-04-05T23:36:05.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36d73c4b80 1 -- start start 2024-04-05T23:36:05.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17cce9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17ccde2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17caac640 con 0x55e17ccd8b40 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17ccd4720 con 0x55e17ccdcec0 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17ccef860 con 0x55e17ccd8220 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36c0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17cce9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36c0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17cce9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:52906/0 (socket says 172.21.15.16:52906) 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36c0e00700 1 -- 172.21.15.16:0/2745351086 learned_addr learned my addr 172.21.15.16:0/2745351086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36bbe00700 1 --2- 172.21.15.16:0/2745351086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17ccde2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36c1800700 1 --2- 172.21.15.16:0/2745351086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36c0e00700 1 -- 172.21.15.16:0/2745351086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 msgr2=0x55e17ccdb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.535+0000 7f36c0e00700 1 --2- 172.21.15.16:0/2745351086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.536+0000 7f36c0e00700 1 -- 172.21.15.16:0/2745351086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 msgr2=0x55e17ccde2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.536+0000 7f36c0e00700 1 --2- 172.21.15.16:0/2745351086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17ccde2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.536+0000 7f36c0e00700 1 -- 172.21.15.16:0/2745351086 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e17cbeee40 con 0x55e17ccd8220 2024-04-05T23:36:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.536+0000 7f36c0e00700 1 --2- 172.21.15.16:0/2745351086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17cce9100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f36b001db80 tx=0x7f36b0009350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e3e2345ca04cdf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.536+0000 7f36aea00700 1 -- 172.21.15.16:0/2745351086 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b0022050 con 0x55e17ccd8220 2024-04-05T23:36:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.536+0000 7f36aea00700 1 -- 172.21.15.16:0/2745351086 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36b001c640 con 0x55e17ccd8220 2024-04-05T23:36:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.536+0000 7f36aea00700 1 -- 172.21.15.16:0/2745351086 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b002f7b0 con 0x55e17ccd8220 2024-04-05T23:36:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2745351086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 msgr2=0x55e17cce9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2745351086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17cce9100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f36b001db80 tx=0x7f36b0009350 comp rx=0 tx=0).stop 2024-04-05T23:36:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2745351086 shutdown_connections 2024-04-05T23:36:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2745351086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17ccdb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2745351086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17cce9100 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/2745351086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17ccde2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2745351086 >> 172.21.15.16:0/2745351086 conn(0x55e17cc4c1c0 msgr2=0x55e17cce4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2745351086 shutdown_connections 2024-04-05T23:36:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 -- 172.21.15.16:0/2745351086 wait complete. 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.537+0000 7f36d73c4b80 1 -- start start 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36d73c4b80 1 Processor -- start 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17ccddb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17ccde0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36d73c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17cce3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17ccde5f0 con 0x55e17ccd8b40 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17ccde760 con 0x55e17ccdcec0 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36d73c4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e17ccde920 con 0x55e17ccd8220 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36d73c4b80 10 client.0 _ll_register_callbacks cb 0x7f36b000ea50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36c0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17ccddb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36bbe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17ccde0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36c0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17ccddb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:52908/0 (socket says 172.21.15.16:52908) 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36c0e00700 1 -- 172.21.15.16:0/1259020531 learned_addr learned my addr 172.21.15.16:0/1259020531 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36bbe00700 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17ccde0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38896/0 (socket says 172.21.15.16:38896) 2024-04-05T23:36:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.538+0000 7f36c1800700 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17cce3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36c1800700 1 -- 172.21.15.16:0/1259020531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 msgr2=0x55e17ccddb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36c1800700 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17ccddb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36c1800700 1 -- 172.21.15.16:0/1259020531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 msgr2=0x55e17ccde0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36c1800700 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17ccde0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36c1800700 1 -- 172.21.15.16:0/1259020531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e17ca56b00 con 0x55e17ccdcec0 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36c0e00700 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17ccddb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36c1800700 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17cce3280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f36b4016000 tx=0x7f36b4099e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5414d8e8f9074b32 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36a2a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36a2a00700 1 -- 172.21.15.16:0/1259020531 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b40995f0 con 0x55e17ccdcec0 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1259020531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36b4099cf0 con 0x55e17ccdcec0 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36a2a00700 1 -- 172.21.15.16:0/1259020531 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f36b40a1a70 con 0x55e17ccdcec0 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36a2a00700 1 -- 172.21.15.16:0/1259020531 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36b409d420 con 0x55e17ccdcec0 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36d73c4b80 10 client.26057 fetch_fsmap learned FSMap version 37 2024-04-05T23:36:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.539+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1259020531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55e17caaec70 con 0x55e17ccdcec0 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.540+0000 7f36a2a00700 1 -- 172.21.15.16:0/1259020531 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36b4099cf0 con 0x55e17ccdcec0 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.540+0000 7f36a2a00700 1 -- 172.21.15.16:0/1259020531 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f36b40b7be0 con 0x55e17ccdcec0 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.540+0000 7f36d73c4b80 10 client.26057 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.540+0000 7f36d73c4b80 10 client.26057 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.540+0000 7f36d73c4b80 20 client.26057 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.540+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e17cd46890 0x55e17cd48cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.540+0000 7f36d73c4b80 4 client.26057 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.540+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1259020531 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e17ca77c90 con 0x55e17cd46890 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.540+0000 7f36c0e00700 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e17cd46890 0x55e17cd48cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.541+0000 7f36c0e00700 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e17cd46890 0x55e17cd48cb0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67eea809aafb2219 server_cookie=456f7faba090f7e5 in_seq=0 out_seq=0 2024-04-05T23:36:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.541+0000 7f36a2a00700 10 client.26057 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36a2a00700 1 -- 172.21.15.16:0/1259020531 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e17ca77c90 con 0x55e17cd46890 2024-04-05T23:36:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36a2a00700 10 client.26057 handle_command_reply: tid=0 2024-04-05T23:36:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 client.26057 shutdown 2024-04-05T23:36:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1259020531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 msgr2=0x55e17cce3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17cce3280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f36b4016000 tx=0x7f36b4099e60 comp rx=0 tx=0).stop 2024-04-05T23:36:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1259020531 shutdown_connections 2024-04-05T23:36:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e17cd46890 0x55e17cd48cb0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e17ccdcec0 0x55e17cce3280 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e17ccd8220 0x55e17ccddb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36a2a00700 0 client.26057 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 --2- 172.21.15.16:0/1259020531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e17ccd8b40 0x55e17ccde0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1259020531 >> 172.21.15.16:0/1259020531 conn(0x55e17cc4c1c0 msgr2=0x55e17cce0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1259020531 shutdown_connections 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.578+0000 7f36d73c4b80 1 -- 172.21.15.16:0/1259020531 wait complete. 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36d73c4b80 20 client.26057 trim_cache size 0 max 16384 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:36:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:36:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:36:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:36:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 -- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f36a40a1290 msgr2=0x7f36a40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f36a40a1290 0x7f36a40a3750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f36b0011550 tx=0x7f36b0009260 comp rx=0 tx=0).stop 2024-04-05T23:36:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 -- 172.21.15.16:0/790645426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36bc19dc40 msgr2=0x7f36bc19e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36bc19dc40 0x7f36bc19e0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f36b4007f10 tx=0x7f36b400d3b0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 -- 172.21.15.16:0/790645426 shutdown_connections 2024-04-05T23:36:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f36a40a1290 0x7f36a40a3750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc100e80 0x7f36bc199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36bc19dc40 0x7f36bc19e0c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 --2- 172.21.15.16:0/790645426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc105fa0 0x7f36bc1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.579+0000 7f36c2200700 1 -- 172.21.15.16:0/790645426 >> 172.21.15.16:0/790645426 conn(0x7f36bc0fc740 msgr2=0x7f36bc0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.580+0000 7f36c2200700 1 -- 172.21.15.16:0/790645426 shutdown_connections 2024-04-05T23:36:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:05.580+0000 7f36c2200700 1 -- 172.21.15.16:0/790645426 wait complete. 2024-04-05T23:36:05.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 57016 3.84 MB/sec warmup 85 sec latency 27.925 ms 2024-04-05T23:36:06.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:06 smithi016 ceph-mon[38248]: pgmap v79: 97 pgs: 97 active+clean; 40 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 198 op/s 2024-04-05T23:36:06.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:06 smithi130 ceph-mon[79417]: pgmap v79: 97 pgs: 97 active+clean; 40 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 198 op/s 2024-04-05T23:36:06.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:06 smithi027 ceph-mon[78597]: pgmap v79: 97 pgs: 97 active+clean; 40 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 198 op/s 2024-04-05T23:36:06.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 58310 3.86 MB/sec warmup 86 sec latency 11.472 ms 2024-04-05T23:36:07.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 60581 3.97 MB/sec warmup 87 sec latency 14.284 ms 2024-04-05T23:36:08.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:08 smithi016 ceph-mon[38248]: pgmap v80: 97 pgs: 97 active+clean; 41 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 345 op/s 2024-04-05T23:36:08.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:08 smithi130 ceph-mon[79417]: pgmap v80: 97 pgs: 97 active+clean; 41 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 345 op/s 2024-04-05T23:36:08.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:08 smithi027 ceph-mon[78597]: pgmap v80: 97 pgs: 97 active+clean; 41 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 345 op/s 2024-04-05T23:36:08.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 61835 3.99 MB/sec warmup 88 sec latency 12.258 ms 2024-04-05T23:36:09.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64150 4.10 MB/sec warmup 89 sec latency 14.049 ms 2024-04-05T23:36:10.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:10 smithi016 ceph-mon[38248]: pgmap v81: 97 pgs: 97 active+clean; 41 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-04-05T23:36:10.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:10 smithi130 ceph-mon[79417]: pgmap v81: 97 pgs: 97 active+clean; 41 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-04-05T23:36:10.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:10 smithi027 ceph-mon[78597]: pgmap v81: 97 pgs: 97 active+clean; 41 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-04-05T23:36:10.987 INFO:tasks.workunit.client.0.smithi016.stdout: 1 65565 4.12 MB/sec warmup 90 sec latency 13.273 ms 2024-04-05T23:36:11.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 67689 4.22 MB/sec warmup 91 sec latency 11.263 ms 2024-04-05T23:36:12.764 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:12 smithi027 ceph-mon[78597]: pgmap v82: 97 pgs: 97 active+clean; 46 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 322 op/s 2024-04-05T23:36:12.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:12 smithi016 ceph-mon[38248]: pgmap v82: 97 pgs: 97 active+clean; 46 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 322 op/s 2024-04-05T23:36:12.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:12 smithi130 ceph-mon[79417]: pgmap v82: 97 pgs: 97 active+clean; 46 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 322 op/s 2024-04-05T23:36:12.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 69032 4.23 MB/sec warmup 92 sec latency 12.363 ms 2024-04-05T23:36:13.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71233 4.33 MB/sec warmup 93 sec latency 15.174 ms 2024-04-05T23:36:14.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:14 smithi016 ceph-mon[38248]: pgmap v83: 97 pgs: 97 active+clean; 47 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 256 op/s 2024-04-05T23:36:14.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:14 smithi130 ceph-mon[79417]: pgmap v83: 97 pgs: 97 active+clean; 47 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 256 op/s 2024-04-05T23:36:14.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:14 smithi027 ceph-mon[78597]: pgmap v83: 97 pgs: 97 active+clean; 47 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 256 op/s 2024-04-05T23:36:14.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 72655 4.35 MB/sec warmup 94 sec latency 17.288 ms 2024-04-05T23:36:15.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 74820 4.44 MB/sec warmup 95 sec latency 11.814 ms 2024-04-05T23:36:16.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:16 smithi016 ceph-mon[38248]: pgmap v84: 97 pgs: 97 active+clean; 47 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 241 op/s 2024-04-05T23:36:16.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:16 smithi130 ceph-mon[79417]: pgmap v84: 97 pgs: 97 active+clean; 47 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 241 op/s 2024-04-05T23:36:16.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:16 smithi027 ceph-mon[78597]: pgmap v84: 97 pgs: 97 active+clean; 47 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 241 op/s 2024-04-05T23:36:16.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 76522 4.46 MB/sec warmup 96 sec latency 16.366 ms 2024-04-05T23:36:17.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 78445 4.55 MB/sec warmup 97 sec latency 11.420 ms 2024-04-05T23:36:18.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.750+0000 7effe2400700 1 -- 172.21.15.16:0/4211665283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc101040 msgr2=0x7effdc101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:18.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.750+0000 7effe2400700 1 --2- 172.21.15.16:0/4211665283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc101040 0x7effdc101420 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7effcc009990 tx=0x7effcc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.750+0000 7effe2400700 1 -- 172.21.15.16:0/4211665283 shutdown_connections 2024-04-05T23:36:18.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.750+0000 7effe2400700 1 --2- 172.21.15.16:0/4211665283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc106160 0x7effdc10dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.750+0000 7effe2400700 1 --2- 172.21.15.16:0/4211665283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc101040 0x7effdc101420 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.750+0000 7effe2400700 1 --2- 172.21.15.16:0/4211665283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effdc1019f0 0x7effdc105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.750+0000 7effe2400700 1 -- 172.21.15.16:0/4211665283 >> 172.21.15.16:0/4211665283 conn(0x7effdc0fc8e0 msgr2=0x7effdc0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:18.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.750+0000 7effe2400700 1 -- 172.21.15.16:0/4211665283 shutdown_connections 2024-04-05T23:36:18.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.750+0000 7effe2400700 1 -- 172.21.15.16:0/4211665283 wait complete. 2024-04-05T23:36:18.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.751+0000 7effe2400700 1 Processor -- start 2024-04-05T23:36:18.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.751+0000 7effe2400700 1 -- start start 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effe2400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effdc101040 0x7effdc0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effe2400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc106160 0x7effdc073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effe2400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc073c60 0x7effdc1a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effe2400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1a5bb0 con 0x7effdc073c60 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effe2400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1a5d20 con 0x7effdc106160 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effe2400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1a5ee0 con 0x7effdc101040 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc106160 0x7effdc073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdb400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effdc101040 0x7effdc0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc106160 0x7effdc073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56810/0 (socket says 172.21.15.16:56810) 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdaa00700 1 -- 172.21.15.16:0/250030469 learned_addr learned my addr 172.21.15.16:0/250030469 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdbe00700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc073c60 0x7effdc1a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdaa00700 1 -- 172.21.15.16:0/250030469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effdc101040 msgr2=0x7effdc0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdaa00700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effdc101040 0x7effdc0731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdaa00700 1 -- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc073c60 msgr2=0x7effdc1a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdaa00700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc073c60 0x7effdc1a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.752+0000 7effdaa00700 1 -- 172.21.15.16:0/250030469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc0095f0 con 0x7effdc106160 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.753+0000 7effdaa00700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc106160 0x7effdc073720 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7effc800e970 tx=0x7effc800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.753+0000 7effd3e00700 1 -- 172.21.15.16:0/250030469 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8009790 con 0x7effdc106160 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.753+0000 7effe2400700 1 -- 172.21.15.16:0/250030469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc1a61c0 con 0x7effdc106160 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.753+0000 7effd3e00700 1 -- 172.21.15.16:0/250030469 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7effc800eec0 con 0x7effdc106160 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.753+0000 7effd3e00700 1 -- 172.21.15.16:0/250030469 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8005590 con 0x7effdc106160 2024-04-05T23:36:18.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.753+0000 7effe2400700 1 -- 172.21.15.16:0/250030469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc1a6720 con 0x7effdc106160 2024-04-05T23:36:18.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.756+0000 7effd3e00700 1 -- 172.21.15.16:0/250030469 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7effc800cbd0 con 0x7effdc106160 2024-04-05T23:36:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.758+0000 7effd3e00700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7effc00a1310 0x7effc00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7effd3e00700 1 -- 172.21.15.16:0/250030469 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7effc8014070 con 0x7effdc106160 2024-04-05T23:36:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7efff5e83b80 1 Processor -- start 2024-04-05T23:36:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7effdb400700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7effc00a1310 0x7effc00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7efff5e83b80 1 -- start start 2024-04-05T23:36:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 0x555bfd4b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4af030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4af5c0 0x555bfd4afa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7efff5e83b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd415b80 con 0x555bfd4b4b80 2024-04-05T23:36:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7efff5e83b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd406930 con 0x555bfd4b54a0 2024-04-05T23:36:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.759+0000 7efff5e83b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd4042f0 con 0x555bfd4af5c0 2024-04-05T23:36:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdb400700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7effc00a1310 0x7effc00a37d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7effcc00b2e0 tx=0x7effcc005fd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4af030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56818/0 (socket says 172.21.15.16:56818) 2024-04-05T23:36:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdaa00700 1 -- 172.21.15.16:0/1288061237 learned_addr learned my addr 172.21.15.16:0/1288061237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdb400700 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 0x555bfd4b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdbe00700 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4af5c0 0x555bfd4afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdbe00700 1 -- 172.21.15.16:0/1288061237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 msgr2=0x555bfd4af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdbe00700 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdbe00700 1 -- 172.21.15.16:0/1288061237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 msgr2=0x555bfd4b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdbe00700 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 0x555bfd4b4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdbe00700 1 -- 172.21.15.16:0/1288061237 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bfd281350 con 0x555bfd4af5c0 2024-04-05T23:36:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.760+0000 7effdaa00700 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.761+0000 7effdbe00700 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4af5c0 0x555bfd4afa20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7effd4009d10 tx=0x7effd40077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17bcde0782365eab server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:18.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.761+0000 7effd0c00700 1 -- 172.21.15.16:0/1288061237 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4018070 con 0x555bfd4af5c0 2024-04-05T23:36:18.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.761+0000 7effd0c00700 1 -- 172.21.15.16:0/1288061237 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7effd400f050 con 0x555bfd4af5c0 2024-04-05T23:36:18.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.761+0000 7effd0c00700 1 -- 172.21.15.16:0/1288061237 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4004420 con 0x555bfd4af5c0 2024-04-05T23:36:18.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.762+0000 7efff5e83b80 1 -- 172.21.15.16:0/1288061237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4af5c0 msgr2=0x555bfd4afa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.762+0000 7efff5e83b80 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4af5c0 0x555bfd4afa20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7effd4009d10 tx=0x7effd40077e0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.762+0000 7efff5e83b80 1 -- 172.21.15.16:0/1288061237 shutdown_connections 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.762+0000 7efff5e83b80 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 0x555bfd4b4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.762+0000 7efff5e83b80 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4af5c0 0x555bfd4afa20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.762+0000 7efff5e83b80 1 --2- 172.21.15.16:0/1288061237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.762+0000 7efff5e83b80 1 -- 172.21.15.16:0/1288061237 >> 172.21.15.16:0/1288061237 conn(0x555bfd4231c0 msgr2=0x555bfd4a9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.762+0000 7efff5e83b80 1 -- 172.21.15.16:0/1288061237 shutdown_connections 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.762+0000 7efff5e83b80 1 -- 172.21.15.16:0/1288061237 wait complete. 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.763+0000 7efff5e83b80 1 -- start start 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.763+0000 7efff5e83b80 1 Processor -- start 2024-04-05T23:36:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4af5c0 0x555bfd4bb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 0x555bfd4b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7efff5e83b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd3ffbe0 con 0x555bfd4b4b80 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7efff5e83b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd39bd20 con 0x555bfd4af5c0 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7efff5e83b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd39b9c0 con 0x555bfd4b54a0 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7efff5e83b80 10 client.0 _ll_register_callbacks cb 0x555bfd207340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7effdb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4af5c0 0x555bfd4bb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7effdb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4af5c0 0x555bfd4bb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56832/0 (socket says 172.21.15.16:56832) 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7effdb400700 1 -- 172.21.15.16:0/3031910931 learned_addr learned my addr 172.21.15.16:0/3031910931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7effdaa00700 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 0x555bfd4b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.764+0000 7effdbe00700 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effdb400700 1 -- 172.21.15.16:0/3031910931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 msgr2=0x555bfd4b8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effdb400700 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effdb400700 1 -- 172.21.15.16:0/3031910931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 msgr2=0x555bfd4b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effdb400700 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 0x555bfd4b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effdb400700 1 -- 172.21.15.16:0/3031910931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc1a6720 con 0x555bfd4af5c0 2024-04-05T23:36:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effdaa00700 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 0x555bfd4b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:36:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effdbe00700 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:36:18.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effdb400700 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4af5c0 0x555bfd4bb4e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7effcc009990 tx=0x7effcc01a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1761c1e1c8d415 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effc5600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effc5600700 1 -- 172.21.15.16:0/3031910931 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc00f660 con 0x555bfd4af5c0 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7efff5e83b80 1 -- 172.21.15.16:0/3031910931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555bfd4a21e0 con 0x555bfd4af5c0 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effc5600700 1 -- 172.21.15.16:0/3031910931 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7effcc00f7c0 con 0x555bfd4af5c0 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.765+0000 7effc5600700 1 -- 172.21.15.16:0/3031910931 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7effcc00f980 con 0x555bfd4af5c0 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7efff5e83b80 10 client.26069 fetch_fsmap learned FSMap version 37 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7efff5e83b80 1 -- 172.21.15.16:0/3031910931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7effd4009940 con 0x555bfd4af5c0 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7effc5600700 1 -- 172.21.15.16:0/3031910931 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7effcc03c020 con 0x555bfd4af5c0 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7effc5600700 1 -- 172.21.15.16:0/3031910931 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7effcc024ac0 con 0x555bfd4af5c0 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7efff5e83b80 10 client.26069 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7efff5e83b80 10 client.26069 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7efff5e83b80 20 client.26069 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7efff5e83b80 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555bfd4c8fe0 0x555bfd4cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7efff5e83b80 4 client.26069 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7efff5e83b80 1 -- 172.21.15.16:0/3031910931 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555bfd3f86a0 con 0x555bfd4c8fe0 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.766+0000 7effdaa00700 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555bfd4c8fe0 0x555bfd4cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.767+0000 7effdaa00700 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555bfd4c8fe0 0x555bfd4cb400 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cb932acb278a321 server_cookie=621790a769d6f222 in_seq=0 out_seq=0 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.767+0000 7effc5600700 10 client.26069 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7effc5600700 1 -- 172.21.15.16:0/3031910931 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555bfd3f86a0 con 0x555bfd4c8fe0 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7effc5600700 10 client.26069 handle_command_reply: tid=0 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 client.26069 shutdown 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 -- 172.21.15.16:0/3031910931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4af5c0 msgr2=0x555bfd4bb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4af5c0 0x555bfd4bb4e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7effcc009990 tx=0x7effcc01a050 comp rx=0 tx=0).stop 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 -- 172.21.15.16:0/3031910931 shutdown_connections 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555bfd4c8fe0 0x555bfd4cb400 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4af5c0 0x555bfd4bb4e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 --2- 172.21.15.16:0/3031910931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4b4b80 0x555bfd4b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 -- 172.21.15.16:0/3031910931 >> 172.21.15.16:0/3031910931 conn(0x555bfd4231c0 msgr2=0x555bfd4a9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:18.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7effc5600700 0 client.26069 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:18.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.768+0000 7efff5e83b80 1 -- 172.21.15.16:0/3031910931 shutdown_connections 2024-04-05T23:36:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.769+0000 7efff5e83b80 1 -- 172.21.15.16:0/3031910931 wait complete. 2024-04-05T23:36:18.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.769+0000 7efff5e83b80 20 client.26069 trim_cache size 0 max 16384 2024-04-05T23:36:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.787+0000 7efff5e83b80 1 Processor -- start 2024-04-05T23:36:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.787+0000 7efff5e83b80 1 -- start start 2024-04-05T23:36:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.787+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4af030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.787+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b4b80 0x555bfd4a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.787+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.787+0000 7efff5e83b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd283640 con 0x555bfd4af5c0 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.787+0000 7efff5e83b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd4b6d20 con 0x555bfd4b54a0 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.787+0000 7efff5e83b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd4b3450 con 0x555bfd4b4b80 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4b28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56834/0 (socket says 172.21.15.16:56834) 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdbe00700 1 -- 172.21.15.16:0/2853672780 learned_addr learned my addr 172.21.15.16:0/2853672780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdaa00700 1 --2- 172.21.15.16:0/2853672780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b4b80 0x555bfd4a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdbe00700 1 -- 172.21.15.16:0/2853672780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b4b80 msgr2=0x555bfd4a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdbe00700 1 --2- 172.21.15.16:0/2853672780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b4b80 0x555bfd4a9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdbe00700 1 -- 172.21.15.16:0/2853672780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 msgr2=0x555bfd4af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdbe00700 1 --2- 172.21.15.16:0/2853672780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdbe00700 1 -- 172.21.15.16:0/2853672780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bfd22db00 con 0x555bfd4b54a0 2024-04-05T23:36:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdb400700 1 --2- 172.21.15.16:0/2853672780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.788+0000 7effdbe00700 1 --2- 172.21.15.16:0/2853672780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4b28e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7effd4017000 tx=0x7effd4006000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e77f2da8ff359bf server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:18.786 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.789+0000 7effd1600700 1 -- 172.21.15.16:0/2853672780 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4018070 con 0x555bfd4b54a0 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.789+0000 7effd1600700 1 -- 172.21.15.16:0/2853672780 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7effd4019040 con 0x555bfd4b54a0 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.789+0000 7effd1600700 1 -- 172.21.15.16:0/2853672780 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4004390 con 0x555bfd4b54a0 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.789+0000 7efff5e83b80 1 -- 172.21.15.16:0/2853672780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 msgr2=0x555bfd4b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.789+0000 7efff5e83b80 1 --2- 172.21.15.16:0/2853672780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4b28e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7effd4017000 tx=0x7effd4006000 comp rx=0 tx=0).stop 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.790+0000 7efff5e83b80 1 -- 172.21.15.16:0/2853672780 shutdown_connections 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.790+0000 7efff5e83b80 1 --2- 172.21.15.16:0/2853672780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.790+0000 7efff5e83b80 1 --2- 172.21.15.16:0/2853672780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b54a0 0x555bfd4b28e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.790+0000 7efff5e83b80 1 --2- 172.21.15.16:0/2853672780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b4b80 0x555bfd4a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.790+0000 7efff5e83b80 1 -- 172.21.15.16:0/2853672780 >> 172.21.15.16:0/2853672780 conn(0x555bfd4231c0 msgr2=0x555bfd4bb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.790+0000 7efff5e83b80 1 -- 172.21.15.16:0/2853672780 shutdown_connections 2024-04-05T23:36:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.790+0000 7efff5e83b80 1 -- 172.21.15.16:0/2853672780 wait complete. 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.790+0000 7efff5e83b80 1 -- start start 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.790+0000 7efff5e83b80 1 Processor -- start 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4b1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b4b80 0x555bfd4b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7efff5e83b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7efff5e83b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd4b22f0 con 0x555bfd4af5c0 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7efff5e83b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd4b2460 con 0x555bfd4b4b80 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7efff5e83b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bfd4b0bc0 con 0x555bfd4b54a0 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7efff5e83b80 10 client.0 _ll_register_callbacks cb 0x7effcc00bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7effdb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7effdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b4b80 0x555bfd4b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7effdbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7effdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b4b80 0x555bfd4b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56848/0 (socket says 172.21.15.16:56848) 2024-04-05T23:36:18.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7effdb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59682/0 (socket says 172.21.15.16:59682) 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.791+0000 7effdaa00700 1 -- 172.21.15.16:0/604325566 learned_addr learned my addr 172.21.15.16:0/604325566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effdbe00700 1 -- 172.21.15.16:0/604325566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b4b80 msgr2=0x555bfd4b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effdbe00700 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b4b80 0x555bfd4b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effdbe00700 1 -- 172.21.15.16:0/604325566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 msgr2=0x555bfd4b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effdbe00700 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effdbe00700 1 -- 172.21.15.16:0/604325566 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd4009940 con 0x555bfd4b54a0 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effdb400700 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effdbe00700 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b0600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7effd4006070 tx=0x7effd4004400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52def2200a59b8da server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effc5600700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effc5600700 1 -- 172.21.15.16:0/604325566 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4018070 con 0x555bfd4b54a0 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effc5600700 1 -- 172.21.15.16:0/604325566 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7effd40047c0 con 0x555bfd4b54a0 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7efff5e83b80 1 -- 172.21.15.16:0/604325566 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effd400f050 con 0x555bfd4b54a0 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7effc5600700 1 -- 172.21.15.16:0/604325566 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7effd4003ea0 con 0x555bfd4b54a0 2024-04-05T23:36:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7efff5e83b80 10 client.26110 fetch_fsmap learned FSMap version 37 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.792+0000 7efff5e83b80 1 -- 172.21.15.16:0/604325566 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x555bfd281350 con 0x555bfd4b54a0 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.793+0000 7effc5600700 1 -- 172.21.15.16:0/604325566 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7effd400f050 con 0x555bfd4b54a0 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.793+0000 7effc5600700 1 -- 172.21.15.16:0/604325566 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7effd4031070 con 0x555bfd4b54a0 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.793+0000 7efff5e83b80 10 client.26110 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.793+0000 7efff5e83b80 10 client.26110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.793+0000 7efff5e83b80 20 client.26110 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.793+0000 7efff5e83b80 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555bfd51d820 0x555bfd51fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.793+0000 7efff5e83b80 4 client.26110 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.793+0000 7efff5e83b80 1 -- 172.21.15.16:0/604325566 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555bfd24ec90 con 0x555bfd51d820 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.793+0000 7effdb400700 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555bfd51d820 0x555bfd51fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.794+0000 7effdb400700 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555bfd51d820 0x555bfd51fc40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eb456c2e888a794 server_cookie=3e747052232372b6 in_seq=0 out_seq=0 2024-04-05T23:36:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.794+0000 7effc5600700 10 client.26110 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:18.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:18 smithi016 ceph-mon[38248]: pgmap v85: 97 pgs: 97 active+clean; 53 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 338 op/s 2024-04-05T23:36:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.840+0000 7effc5600700 1 -- 172.21.15.16:0/604325566 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555bfd24ec90 con 0x555bfd51d820 2024-04-05T23:36:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.840+0000 7effc5600700 10 client.26110 handle_command_reply: tid=0 2024-04-05T23:36:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.840+0000 7efff5e83b80 1 client.26110 shutdown 2024-04-05T23:36:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 -- 172.21.15.16:0/604325566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 msgr2=0x555bfd4b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b0600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7effd4006070 tx=0x7effd4004400 comp rx=0 tx=0).stop 2024-04-05T23:36:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 -- 172.21.15.16:0/604325566 shutdown_connections 2024-04-05T23:36:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555bfd51d820 0x555bfd51fc40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555bfd4af5c0 0x555bfd4b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bfd4b54a0 0x555bfd4b0600 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 --2- 172.21.15.16:0/604325566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555bfd4b4b80 0x555bfd4b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 -- 172.21.15.16:0/604325566 >> 172.21.15.16:0/604325566 conn(0x555bfd4231c0 msgr2=0x555bfd4b7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7effc5600700 0 client.26110 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 -- 172.21.15.16:0/604325566 shutdown_connections 2024-04-05T23:36:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 1 -- 172.21.15.16:0/604325566 wait complete. 2024-04-05T23:36:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.841+0000 7efff5e83b80 20 client.26110 trim_cache size 0 max 16384 2024-04-05T23:36:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:36:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:36:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:36:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:36:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 -- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7effc00a1310 msgr2=0x7effc00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7effc00a1310 0x7effc00a37d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7effcc00b2e0 tx=0x7effcc005fd0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 -- 172.21.15.16:0/250030469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc106160 msgr2=0x7effdc073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc106160 0x7effdc073720 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7effc800e970 tx=0x7effc800ec80 comp rx=0 tx=0).stop 2024-04-05T23:36:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 -- 172.21.15.16:0/250030469 shutdown_connections 2024-04-05T23:36:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7effc00a1310 0x7effc00a37d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effdc073c60 0x7effdc1a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effdc101040 0x7effdc0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 --2- 172.21.15.16:0/250030469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effdc106160 0x7effdc073720 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.842+0000 7effe2400700 1 -- 172.21.15.16:0/250030469 >> 172.21.15.16:0/250030469 conn(0x7effdc0fc8e0 msgr2=0x7effdc0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.843+0000 7effe2400700 1 -- 172.21.15.16:0/250030469 shutdown_connections 2024-04-05T23:36:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:18.843+0000 7effe2400700 1 -- 172.21.15.16:0/250030469 wait complete. 2024-04-05T23:36:18.841 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:18 smithi130 ceph-mon[79417]: pgmap v85: 97 pgs: 97 active+clean; 53 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 338 op/s 2024-04-05T23:36:18.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:18 smithi027 ceph-mon[78597]: pgmap v85: 97 pgs: 97 active+clean; 53 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 338 op/s 2024-04-05T23:36:18.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 80268 4.57 MB/sec warmup 98 sec latency 16.627 ms 2024-04-05T23:36:19.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 82006 4.65 MB/sec warmup 99 sec latency 13.847 ms 2024-04-05T23:36:20.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:20 smithi016 ceph-mon[38248]: pgmap v86: 97 pgs: 97 active+clean; 53 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 191 op/s 2024-04-05T23:36:20.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:20 smithi130 ceph-mon[79417]: pgmap v86: 97 pgs: 97 active+clean; 53 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 191 op/s 2024-04-05T23:36:20.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:20 smithi027 ceph-mon[78597]: pgmap v86: 97 pgs: 97 active+clean; 53 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 191 op/s 2024-04-05T23:36:20.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 84031 4.68 MB/sec warmup 100 sec latency 10.709 ms 2024-04-05T23:36:21.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 85587 4.75 MB/sec warmup 101 sec latency 11.614 ms 2024-04-05T23:36:22.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:22 smithi027 ceph-mon[78597]: pgmap v87: 97 pgs: 97 active+clean; 49 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 307 op/s 2024-04-05T23:36:22.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:22 smithi016 ceph-mon[38248]: pgmap v87: 97 pgs: 97 active+clean; 49 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 307 op/s 2024-04-05T23:36:22.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:22 smithi130 ceph-mon[79417]: pgmap v87: 97 pgs: 97 active+clean; 49 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 307 op/s 2024-04-05T23:36:22.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 87618 4.78 MB/sec warmup 102 sec latency 11.475 ms 2024-04-05T23:36:23.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 89165 4.84 MB/sec warmup 103 sec latency 27.261 ms 2024-04-05T23:36:24.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:24 smithi016 ceph-mon[38248]: pgmap v88: 97 pgs: 97 active+clean; 50 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 242 op/s 2024-04-05T23:36:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:24 smithi130 ceph-mon[79417]: pgmap v88: 97 pgs: 97 active+clean; 50 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 242 op/s 2024-04-05T23:36:24.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:24 smithi027 ceph-mon[78597]: pgmap v88: 97 pgs: 97 active+clean; 50 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 242 op/s 2024-04-05T23:36:24.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 91322 4.88 MB/sec warmup 104 sec latency 11.069 ms 2024-04-05T23:36:25.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 92829 4.94 MB/sec warmup 105 sec latency 11.183 ms 2024-04-05T23:36:26.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:26 smithi016 ceph-mon[38248]: pgmap v89: 97 pgs: 97 active+clean; 50 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 228 op/s 2024-04-05T23:36:26.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:26 smithi130 ceph-mon[79417]: pgmap v89: 97 pgs: 97 active+clean; 50 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 228 op/s 2024-04-05T23:36:26.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:26 smithi027 ceph-mon[78597]: pgmap v89: 97 pgs: 97 active+clean; 50 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 228 op/s 2024-04-05T23:36:26.988 INFO:tasks.workunit.client.0.smithi016.stdout: 1 95198 5.01 MB/sec warmup 106 sec latency 13.473 ms 2024-04-05T23:36:27.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 96539 5.02 MB/sec warmup 107 sec latency 12.149 ms 2024-04-05T23:36:28.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:28 smithi016 ceph-mon[38248]: pgmap v90: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 340 op/s 2024-04-05T23:36:28.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:28 smithi130 ceph-mon[79417]: pgmap v90: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 340 op/s 2024-04-05T23:36:28.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:28 smithi027 ceph-mon[78597]: pgmap v90: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 340 op/s 2024-04-05T23:36:28.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 98804 5.09 MB/sec warmup 108 sec latency 26.152 ms 2024-04-05T23:36:29.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100142 5.11 MB/sec warmup 109 sec latency 13.361 ms 2024-04-05T23:36:30.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:30 smithi016 ceph-mon[38248]: pgmap v91: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 243 op/s 2024-04-05T23:36:30.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:30 smithi130 ceph-mon[79417]: pgmap v91: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 243 op/s 2024-04-05T23:36:30.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:30 smithi027 ceph-mon[78597]: pgmap v91: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 243 op/s 2024-04-05T23:36:30.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 102569 5.18 MB/sec warmup 110 sec latency 14.385 ms 2024-04-05T23:36:31.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 103823 5.19 MB/sec warmup 111 sec latency 13.000 ms 2024-04-05T23:36:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:31.999+0000 7fa06f200700 1 -- 172.21.15.16:0/562612640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 msgr2=0x7fa068101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:31.999+0000 7fa06f200700 1 --2- 172.21.15.16:0/562612640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 0x7fa068101420 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa058009990 tx=0x7fa058009ca0 comp rx=0 tx=0).stop 2024-04-05T23:36:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:31.999+0000 7fa06f200700 1 -- 172.21.15.16:0/562612640 shutdown_connections 2024-04-05T23:36:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:31.999+0000 7fa06f200700 1 --2- 172.21.15.16:0/562612640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa068106160 0x7fa06810dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:31.999+0000 7fa06f200700 1 --2- 172.21.15.16:0/562612640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 0x7fa068101420 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:31.999+0000 7fa06f200700 1 --2- 172.21.15.16:0/562612640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0681019f0 0x7fa068105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:31.999+0000 7fa06f200700 1 -- 172.21.15.16:0/562612640 >> 172.21.15.16:0/562612640 conn(0x7fa0680fc8e0 msgr2=0x7fa0680fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:31.999+0000 7fa06f200700 1 -- 172.21.15.16:0/562612640 shutdown_connections 2024-04-05T23:36:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:31.999+0000 7fa06f200700 1 -- 172.21.15.16:0/562612640 wait complete. 2024-04-05T23:36:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.000+0000 7fa06f200700 1 Processor -- start 2024-04-05T23:36:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.000+0000 7fa06f200700 1 -- start start 2024-04-05T23:36:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.000+0000 7fa06f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 0x7fa0681972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.000+0000 7fa06f200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa068106160 0x7fa068197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.000+0000 7fa06f200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06819bb70 0x7fa06819bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.000+0000 7fa06f200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06819c5f0 con 0x7fa06819bb70 2024-04-05T23:36:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.000+0000 7fa06f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06819c760 con 0x7fa068101040 2024-04-05T23:36:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.000+0000 7fa06f200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06819c920 con 0x7fa068106160 2024-04-05T23:36:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 0x7fa0681972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 0x7fa0681972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51012/0 (socket says 172.21.15.16:51012) 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067e00700 1 -- 172.21.15.16:0/1683881461 learned_addr learned my addr 172.21.15.16:0/1683881461 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067400700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa068106160 0x7fa068197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa06d200700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06819bb70 0x7fa06819bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067e00700 1 -- 172.21.15.16:0/1683881461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa068106160 msgr2=0x7fa068197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067e00700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa068106160 0x7fa068197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067e00700 1 -- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06819bb70 msgr2=0x7fa06819bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067e00700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06819bb70 0x7fa06819bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067e00700 1 -- 172.21.15.16:0/1683881461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0580095f0 con 0x7fa068101040 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.001+0000 7fa067e00700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 0x7fa0681972e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa058005910 tx=0x7fa058004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.002+0000 7fa064c00700 1 -- 172.21.15.16:0/1683881461 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05801d070 con 0x7fa068101040 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.002+0000 7fa06f200700 1 -- 172.21.15.16:0/1683881461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0681a3e20 con 0x7fa068101040 2024-04-05T23:36:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.002+0000 7fa06f200700 1 -- 172.21.15.16:0/1683881461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0681a4250 con 0x7fa068101040 2024-04-05T23:36:31.995 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.002+0000 7fa064c00700 1 -- 172.21.15.16:0/1683881461 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa05802b3b0 con 0x7fa068101040 2024-04-05T23:36:31.995 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.002+0000 7fa064c00700 1 -- 172.21.15.16:0/1683881461 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800e640 con 0x7fa068101040 2024-04-05T23:36:31.996 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.005+0000 7fa064c00700 1 -- 172.21.15.16:0/1683881461 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fa05800fac0 con 0x7fa068101040 2024-04-05T23:36:31.998 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.006+0000 7fa064c00700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa04c0a1310 0x7fa04c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:31.998 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa082ceeb80 1 Processor -- start 2024-04-05T23:36:31.998 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa064c00700 1 -- 172.21.15.16:0/1683881461 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa0580cdc80 con 0x7fa068101040 2024-04-05T23:36:31.998 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa082ceeb80 1 -- start start 2024-04-05T23:36:31.998 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa067400700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa04c0a1310 0x7fa04c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 0x5564c0fcff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 0x5564c0fca030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fcaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0f30b80 con 0x5564c0fcfb80 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0f21930 con 0x5564c0fca5c0 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0f1f2f0 con 0x5564c0fd04a0 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa067e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 0x5564c0fcff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.007+0000 7fa06d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fcaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa067e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 0x5564c0fcff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36530/0 (socket says 172.21.15.16:36530) 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa067e00700 1 -- 172.21.15.16:0/363840701 learned_addr learned my addr 172.21.15.16:0/363840701 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa067400700 1 --2- 172.21.15.16:0/363840701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 0x5564c0fca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa06d200700 1 -- 172.21.15.16:0/363840701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 msgr2=0x5564c0fca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa06d200700 1 --2- 172.21.15.16:0/363840701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 0x5564c0fca030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa06d200700 1 -- 172.21.15.16:0/363840701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 msgr2=0x5564c0fcff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa06d200700 1 --2- 172.21.15.16:0/363840701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 0x5564c0fcff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa06d200700 1 -- 172.21.15.16:0/363840701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564c0d9c350 con 0x5564c0fca5c0 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa067400700 1 --2- 172.21.15.16:0/363840701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 0x5564c0fca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa067400700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa04c0a1310 0x7fa04c0a37d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa0681987e0 tx=0x7fa054009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa06d200700 1 --2- 172.21.15.16:0/363840701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fcaa20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa060011580 tx=0x7fa060009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82b0fc9c58abf16 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa05d600700 1 -- 172.21.15.16:0/363840701 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06000e040 con 0x5564c0fca5c0 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.008+0000 7fa05d600700 1 -- 172.21.15.16:0/363840701 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa06000f040 con 0x5564c0fca5c0 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa05d600700 1 -- 172.21.15.16:0/363840701 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060004450 con 0x5564c0fca5c0 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa082ceeb80 1 -- 172.21.15.16:0/363840701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 msgr2=0x5564c0fcaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/363840701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fcaa20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa060011580 tx=0x7fa060009920 comp rx=0 tx=0).stop 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa082ceeb80 1 -- 172.21.15.16:0/363840701 shutdown_connections 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/363840701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 0x5564c0fcff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/363840701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fcaa20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/363840701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 0x5564c0fca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa082ceeb80 1 -- 172.21.15.16:0/363840701 >> 172.21.15.16:0/363840701 conn(0x5564c0f3e1c0 msgr2=0x5564c0fc4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa082ceeb80 1 -- 172.21.15.16:0/363840701 shutdown_connections 2024-04-05T23:36:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.009+0000 7fa082ceeb80 1 -- 172.21.15.16:0/363840701 wait complete. 2024-04-05T23:36:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.010+0000 7fa082ceeb80 1 -- start start 2024-04-05T23:36:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.010+0000 7fa082ceeb80 1 Processor -- start 2024-04-05T23:36:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fd3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 0x5564c0fd3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0f1abe0 con 0x5564c0fca5c0 2024-04-05T23:36:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0eb6d20 con 0x5564c0fd04a0 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0eb69c0 con 0x5564c0fcfb80 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa082ceeb80 10 client.0 _ll_register_callbacks cb 0x5564c0d22340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fd3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36538/0 (socket says 172.21.15.16:36538) 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fd3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:51870/0 (socket says 172.21.15.16:51870) 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067e00700 1 -- 172.21.15.16:0/3281993340 learned_addr learned my addr 172.21.15.16:0/3281993340 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa06d200700 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 0x5564c0fd3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067400700 1 -- 172.21.15.16:0/3281993340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 msgr2=0x5564c0fd3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067400700 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 0x5564c0fd3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067400700 1 -- 172.21.15.16:0/3281993340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 msgr2=0x5564c0fd6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067400700 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.011+0000 7fa067400700 1 -- 172.21.15.16:0/3281993340 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa060011230 con 0x5564c0fcfb80 2024-04-05T23:36:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.012+0000 7fa067400700 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fd3790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa0540097c0 tx=0x7fa0540047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87497d32b6515e69 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.012+0000 7fa052000700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.012+0000 7fa052000700 1 -- 172.21.15.16:0/3281993340 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054025400 con 0x5564c0fcfb80 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.012+0000 7fa052000700 1 -- 172.21.15.16:0/3281993340 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa054025560 con 0x5564c0fcfb80 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.012+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3281993340 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564c0fbd1e0 con 0x5564c0fcfb80 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.012+0000 7fa052000700 1 -- 172.21.15.16:0/3281993340 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa054025720 con 0x5564c0fcfb80 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.012+0000 7fa082ceeb80 10 client.26122 fetch_fsmap learned FSMap version 37 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.012+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3281993340 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x5564c0d48b00 con 0x5564c0fcfb80 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.012+0000 7fa052000700 1 -- 172.21.15.16:0/3281993340 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa054032020 con 0x5564c0fcfb80 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa052000700 1 -- 172.21.15.16:0/3281993340 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fa05400e440 con 0x5564c0fcfb80 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa082ceeb80 10 client.26122 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa082ceeb80 10 client.26122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa082ceeb80 20 client.26122 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5564c0fec890 0x5564c0feecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa082ceeb80 4 client.26122 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3281993340 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564c0f136a0 con 0x5564c0fec890 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa067e00700 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5564c0fec890 0x5564c0feecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa067e00700 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5564c0fec890 0x5564c0feecb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca606542d3b1accc server_cookie=770cc3d1f1e6d310 in_seq=0 out_seq=0 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.013+0000 7fa052000700 10 client.26122 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa052000700 1 -- 172.21.15.16:0/3281993340 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5564c0f136a0 con 0x5564c0fec890 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa052000700 10 client.26122 handle_command_reply: tid=0 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa082ceeb80 1 client.26122 shutdown 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3281993340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 msgr2=0x5564c0fd3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fd3790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa0540097c0 tx=0x7fa0540047c0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3281993340 shutdown_connections 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5564c0fec890 0x5564c0feecb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 0x5564c0fd3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa052000700 0 client.26122 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3281993340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fd3790 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.014+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3281993340 >> 172.21.15.16:0/3281993340 conn(0x5564c0f3e1c0 msgr2=0x5564c0fc4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.015+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3281993340 shutdown_connections 2024-04-05T23:36:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.015+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3281993340 wait complete. 2024-04-05T23:36:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.015+0000 7fa082ceeb80 20 client.26122 trim_cache size 0 max 16384 2024-04-05T23:36:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.031+0000 7fa082ceeb80 1 Processor -- start 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.031+0000 7fa082ceeb80 1 -- start start 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.031+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.031+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fc4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.031+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 0x5564c0fce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.031+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0d9e640 con 0x5564c0fca5c0 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.031+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0fd1d20 con 0x5564c0fd04a0 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.031+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0fce450 con 0x5564c0fcfb80 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa06d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 0x5564c0fce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36544/0 (socket says 172.21.15.16:36544) 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067e00700 1 -- 172.21.15.16:0/3819457306 learned_addr learned my addr 172.21.15.16:0/3819457306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067400700 1 --2- 172.21.15.16:0/3819457306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fc4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067e00700 1 -- 172.21.15.16:0/3819457306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 msgr2=0x5564c0fc4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067e00700 1 --2- 172.21.15.16:0/3819457306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fc4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067e00700 1 -- 172.21.15.16:0/3819457306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 msgr2=0x5564c0fce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067e00700 1 --2- 172.21.15.16:0/3819457306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 0x5564c0fce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067e00700 1 -- 172.21.15.16:0/3819457306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564c0ee0e40 con 0x5564c0fca5c0 2024-04-05T23:36:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa06d200700 1 --2- 172.21.15.16:0/3819457306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 0x5564c0fce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa067e00700 1 --2- 172.21.15.16:0/3819457306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd5fd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa05800f740 tx=0x7fa05800b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68c64b8dc23350a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.032+0000 7fa05e000700 1 -- 172.21.15.16:0/3819457306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058097130 con 0x5564c0fca5c0 2024-04-05T23:36:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa05e000700 1 -- 172.21.15.16:0/3819457306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa05809b4f0 con 0x5564c0fca5c0 2024-04-05T23:36:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa05e000700 1 -- 172.21.15.16:0/3819457306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800fd70 con 0x5564c0fca5c0 2024-04-05T23:36:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3819457306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 msgr2=0x5564c0fd5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3819457306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd5fd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa05800f740 tx=0x7fa05800b610 comp rx=0 tx=0).stop 2024-04-05T23:36:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3819457306 shutdown_connections 2024-04-05T23:36:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3819457306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fca5c0 0x5564c0fd5fd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3819457306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fd04a0 0x5564c0fce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/3819457306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fcfb80 0x5564c0fc4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3819457306 >> 172.21.15.16:0/3819457306 conn(0x5564c0f3e1c0 msgr2=0x5564c0fca9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.033+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3819457306 shutdown_connections 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.034+0000 7fa082ceeb80 1 -- 172.21.15.16:0/3819457306 wait complete. 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.034+0000 7fa082ceeb80 1 -- start start 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.034+0000 7fa082ceeb80 1 Processor -- start 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.034+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fe8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.034+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 0x5564c0fe93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.034+0000 7fa082ceeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 0x5564c0fcb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.034+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0fe9ae0 con 0x5564c0fcfb80 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0fcbc20 con 0x5564c0fca5c0 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa082ceeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564c0fcbde0 con 0x5564c0fd04a0 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa082ceeb80 10 client.0 _ll_register_callbacks cb 0x7fa05401f040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fe8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fe8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:50748/0 (socket says 172.21.15.16:50748) 2024-04-05T23:36:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067e00700 1 -- 172.21.15.16:0/4243908262 learned_addr learned my addr 172.21.15.16:0/4243908262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa06d200700 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 0x5564c0fcb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067400700 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 0x5564c0fe93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067e00700 1 -- 172.21.15.16:0/4243908262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 msgr2=0x5564c0fcb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067e00700 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 0x5564c0fcb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067e00700 1 -- 172.21.15.16:0/4243908262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 msgr2=0x5564c0fe93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067e00700 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 0x5564c0fe93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067e00700 1 -- 172.21.15.16:0/4243908262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564c0d9c350 con 0x5564c0fca5c0 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa067e00700 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fe8e90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa05809aaf0 tx=0x7fa058097250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9272de1c1cdee9db server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.035+0000 7fa052000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa052000700 1 -- 172.21.15.16:0/4243908262 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0580a31e0 con 0x5564c0fca5c0 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa052000700 1 -- 172.21.15.16:0/4243908262 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fa05800fd70 con 0x5564c0fca5c0 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa082ceeb80 1 -- 172.21.15.16:0/4243908262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa06000f040 con 0x5564c0fca5c0 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa052000700 1 -- 172.21.15.16:0/4243908262 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa05809b050 con 0x5564c0fca5c0 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa082ceeb80 10 client.26093 fetch_fsmap learned FSMap version 37 2024-04-05T23:36:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa082ceeb80 1 -- 172.21.15.16:0/4243908262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x7fa0681a4250 con 0x5564c0fca5c0 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa052000700 1 -- 172.21.15.16:0/4243908262 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa06000f040 con 0x5564c0fca5c0 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa052000700 1 -- 172.21.15.16:0/4243908262 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fa0580a2d50 con 0x5564c0fca5c0 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa082ceeb80 10 client.26093 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa082ceeb80 10 client.26093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa082ceeb80 20 client.26093 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5564c0fcd210 0x5564c10436d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa082ceeb80 4 client.26093 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa082ceeb80 1 -- 172.21.15.16:0/4243908262 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5564c0d69c90 con 0x5564c0fcd210 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.036+0000 7fa067400700 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5564c0fcd210 0x5564c10436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.037+0000 7fa067400700 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5564c0fcd210 0x5564c10436d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=406116b84ab201d2 server_cookie=440ad03533ad1e1 in_seq=0 out_seq=0 2024-04-05T23:36:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.037+0000 7fa052000700 10 client.26093 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.074+0000 7fa052000700 1 -- 172.21.15.16:0/4243908262 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5564c0d69c90 con 0x5564c0fcd210 2024-04-05T23:36:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.074+0000 7fa052000700 10 client.26093 handle_command_reply: tid=0 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.074+0000 7fa082ceeb80 1 client.26093 shutdown 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.074+0000 7fa082ceeb80 1 -- 172.21.15.16:0/4243908262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 msgr2=0x5564c0fe8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.074+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fe8e90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa05809aaf0 tx=0x7fa058097250 comp rx=0 tx=0).stop 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa082ceeb80 1 -- 172.21.15.16:0/4243908262 shutdown_connections 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5564c0fcd210 0x5564c10436d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564c0fd04a0 0x5564c0fcb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5564c0fca5c0 0x5564c0fe8e90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa082ceeb80 1 --2- 172.21.15.16:0/4243908262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564c0fcfb80 0x5564c0fe93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa052000700 0 client.26093 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa082ceeb80 1 -- 172.21.15.16:0/4243908262 >> 172.21.15.16:0/4243908262 conn(0x5564c0f3e1c0 msgr2=0x5564c0fd0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa082ceeb80 1 -- 172.21.15.16:0/4243908262 shutdown_connections 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa082ceeb80 1 -- 172.21.15.16:0/4243908262 wait complete. 2024-04-05T23:36:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.075+0000 7fa082ceeb80 20 client.26093 trim_cache size 0 max 16384 2024-04-05T23:36:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:36:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:36:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:36:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:36:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 -- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa04c0a1310 msgr2=0x7fa04c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa04c0a1310 0x7fa04c0a37d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa0681987e0 tx=0x7fa054009260 comp rx=0 tx=0).stop 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 -- 172.21.15.16:0/1683881461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 msgr2=0x7fa0681972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 0x7fa0681972e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa058005910 tx=0x7fa058004730 comp rx=0 tx=0).stop 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 -- 172.21.15.16:0/1683881461 shutdown_connections 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa04c0a1310 0x7fa04c0a37d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06819bb70 0x7fa06819bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa068101040 0x7fa0681972e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 --2- 172.21.15.16:0/1683881461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa068106160 0x7fa068197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 -- 172.21.15.16:0/1683881461 >> 172.21.15.16:0/1683881461 conn(0x7fa0680fc8e0 msgr2=0x7fa0680fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 -- 172.21.15.16:0/1683881461 shutdown_connections 2024-04-05T23:36:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:32.076+0000 7fa06f200700 1 -- 172.21.15.16:0/1683881461 wait complete. 2024-04-05T23:36:32.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:32 smithi016 ceph-mon[38248]: pgmap v92: 97 pgs: 97 active+clean; 58 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 348 op/s 2024-04-05T23:36:32.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:32 smithi130 ceph-mon[79417]: pgmap v92: 97 pgs: 97 active+clean; 58 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 348 op/s 2024-04-05T23:36:32.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:32 smithi027 ceph-mon[78597]: pgmap v92: 97 pgs: 97 active+clean; 58 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 348 op/s 2024-04-05T23:36:32.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 106258 5.27 MB/sec warmup 112 sec latency 34.248 ms 2024-04-05T23:36:33.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 107353 5.27 MB/sec warmup 113 sec latency 34.396 ms 2024-04-05T23:36:34.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:34 smithi016 ceph-mon[38248]: pgmap v93: 97 pgs: 97 active+clean; 59 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2024-04-05T23:36:34.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:34 smithi130 ceph-mon[79417]: pgmap v93: 97 pgs: 97 active+clean; 59 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2024-04-05T23:36:34.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:34 smithi027 ceph-mon[78597]: pgmap v93: 97 pgs: 97 active+clean; 59 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2024-04-05T23:36:34.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 109713 5.34 MB/sec warmup 114 sec latency 18.923 ms 2024-04-05T23:36:35.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 110934 5.34 MB/sec warmup 115 sec latency 12.702 ms 2024-04-05T23:36:36.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:36 smithi016 ceph-mon[38248]: pgmap v94: 97 pgs: 97 active+clean; 59 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 232 op/s 2024-04-05T23:36:36.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:36 smithi130 ceph-mon[79417]: pgmap v94: 97 pgs: 97 active+clean; 59 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 232 op/s 2024-04-05T23:36:36.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:36 smithi027 ceph-mon[78597]: pgmap v94: 97 pgs: 97 active+clean; 59 MiB data, 694 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 232 op/s 2024-04-05T23:36:36.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 113516 5.44 MB/sec warmup 116 sec latency 12.161 ms 2024-04-05T23:36:37.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 114487 5.42 MB/sec warmup 117 sec latency 48.702 ms 2024-04-05T23:36:38.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:38 smithi016 ceph-mon[38248]: pgmap v95: 97 pgs: 97 active+clean; 60 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 375 op/s 2024-04-05T23:36:38.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:38 smithi130 ceph-mon[79417]: pgmap v95: 97 pgs: 97 active+clean; 60 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 375 op/s 2024-04-05T23:36:38.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:38 smithi027 ceph-mon[78597]: pgmap v95: 97 pgs: 97 active+clean; 60 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 375 op/s 2024-04-05T23:36:38.989 INFO:tasks.workunit.client.0.smithi016.stdout: 1 116822 5.49 MB/sec warmup 118 sec latency 23.045 ms 2024-04-05T23:36:39.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 117963 5.48 MB/sec warmup 119 sec latency 18.051 ms 2024-04-05T23:36:40.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:40 smithi016 ceph-mon[38248]: pgmap v96: 97 pgs: 97 active+clean; 60 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 263 op/s 2024-04-05T23:36:40.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:40 smithi130 ceph-mon[79417]: pgmap v96: 97 pgs: 97 active+clean; 60 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 263 op/s 2024-04-05T23:36:40.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:40 smithi027 ceph-mon[78597]: pgmap v96: 97 pgs: 97 active+clean; 60 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 263 op/s 2024-04-05T23:36:41.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 121547 5.42 MB/sec execute 1 sec latency 15.320 ms 2024-04-05T23:36:42.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:42 smithi016 ceph-mon[38248]: pgmap v97: 97 pgs: 97 active+clean; 64 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 377 op/s 2024-04-05T23:36:42.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:42 smithi130 ceph-mon[79417]: pgmap v97: 97 pgs: 97 active+clean; 64 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 377 op/s 2024-04-05T23:36:42.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:42 smithi027 ceph-mon[78597]: pgmap v97: 97 pgs: 97 active+clean; 64 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 377 op/s 2024-04-05T23:36:42.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124008 9.60 MB/sec execute 2 sec latency 11.481 ms 2024-04-05T23:36:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:43 smithi016 ceph-mon[38248]: pgmap v98: 97 pgs: 97 active+clean; 64 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 273 op/s 2024-04-05T23:36:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:43 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:36:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:43 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:36:43.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:43 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:36:43.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:43 smithi130 ceph-mon[79417]: pgmap v98: 97 pgs: 97 active+clean; 64 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 273 op/s 2024-04-05T23:36:43.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:43 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:36:43.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:43 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:36:43.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:43 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:36:43.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:43 smithi027 ceph-mon[78597]: pgmap v98: 97 pgs: 97 active+clean; 64 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 273 op/s 2024-04-05T23:36:43.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:43 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:36:43.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:43 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:36:43.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:43 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:36:43.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 125095 8.18 MB/sec execute 3 sec latency 13.135 ms 2024-04-05T23:36:44.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:36:44.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:36:44.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:36:44.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:36:44.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:36:44.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:36:44.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:36:44.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:36:44.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:36:44.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 127354 9.25 MB/sec execute 4 sec latency 33.144 ms 2024-04-05T23:36:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.235+0000 7f9193800700 1 -- 172.21.15.16:0/571374146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f918c1019e0 msgr2=0x7f918c105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.235+0000 7f9193800700 1 --2- 172.21.15.16:0/571374146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f918c1019e0 0x7f918c105a20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9178009990 tx=0x7f9178009ca0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.235+0000 7f9193800700 1 -- 172.21.15.16:0/571374146 shutdown_connections 2024-04-05T23:36:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.235+0000 7f9193800700 1 --2- 172.21.15.16:0/571374146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f918c101030 0x7f918c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.235+0000 7f9193800700 1 --2- 172.21.15.16:0/571374146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f918c106150 0x7f918c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.235+0000 7f9193800700 1 --2- 172.21.15.16:0/571374146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f918c1019e0 0x7f918c105a20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.235+0000 7f9193800700 1 -- 172.21.15.16:0/571374146 >> 172.21.15.16:0/571374146 conn(0x7f918c0fc8b0 msgr2=0x7f918c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.235+0000 7f9193800700 1 -- 172.21.15.16:0/571374146 shutdown_connections 2024-04-05T23:36:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.235+0000 7f9193800700 1 -- 172.21.15.16:0/571374146 wait complete. 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f9193800700 1 Processor -- start 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f9193800700 1 -- start start 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f9193800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f918c101030 0x7f918c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f9193800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f918c106150 0x7f918c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f9193800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f918c073c60 0x7f918c1a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f9193800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c1a5a70 con 0x7f918c101030 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f9193800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c1a5bb0 con 0x7f918c073c60 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f9193800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c1a5d70 con 0x7f918c106150 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f918be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f918c106150 0x7f918c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.237+0000 7f9190e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f918c101030 0x7f918c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f918be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f918c106150 0x7f918c073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44242/0 (socket says 172.21.15.16:44242) 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f9191800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f918c073c60 0x7f918c1a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f918be00700 1 -- 172.21.15.16:0/2429195002 learned_addr learned my addr 172.21.15.16:0/2429195002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f9191800700 1 -- 172.21.15.16:0/2429195002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f918c106150 msgr2=0x7f918c073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f9191800700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f918c106150 0x7f918c073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f9191800700 1 -- 172.21.15.16:0/2429195002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f918c101030 msgr2=0x7f918c0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f9191800700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f918c101030 0x7f918c0731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f9191800700 1 -- 172.21.15.16:0/2429195002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91780095f0 con 0x7f918c073c60 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f9191800700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f918c073c60 0x7f918c1a54a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f918400efc0 tx=0x7f918400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f9189600700 1 -- 172.21.15.16:0/2429195002 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184018070 con 0x7f918c073c60 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.238+0000 7f9189600700 1 -- 172.21.15.16:0/2429195002 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f918400f040 con 0x7f918c073c60 2024-04-05T23:36:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.239+0000 7f9193800700 1 -- 172.21.15.16:0/2429195002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c1a5ff0 con 0x7f918c073c60 2024-04-05T23:36:45.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.239+0000 7f9189600700 1 -- 172.21.15.16:0/2429195002 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918400cd40 con 0x7f918c073c60 2024-04-05T23:36:45.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.239+0000 7f9193800700 1 -- 172.21.15.16:0/2429195002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c1a6520 con 0x7f918c073c60 2024-04-05T23:36:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.242+0000 7f9189600700 1 -- 172.21.15.16:0/2429195002 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f9184003a00 con 0x7f918c073c60 2024-04-05T23:36:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.243+0000 7f91a73f5b80 1 Processor -- start 2024-04-05T23:36:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.243+0000 7f91a73f5b80 1 -- start start 2024-04-05T23:36:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.243+0000 7f9189600700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f91740a1310 0x7f91740a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.243+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d572f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.243+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 0x557b9d56d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.243+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d56da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.243+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d4d3b80 con 0x557b9d5734a0 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.243+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d4c4930 con 0x557b9d56d5c0 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.243+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d4c22f0 con 0x557b9d572b80 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9189600700 1 -- 172.21.15.16:0/2429195002 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f91840cc7a0 con 0x7f918c073c60 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9190e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d572f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9191800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d56da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9191800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d56da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:41544/0 (socket says 172.21.15.16:41544) 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9190e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d572f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44258/0 (socket says 172.21.15.16:44258) 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9191800700 1 -- 172.21.15.16:0/1440478258 learned_addr learned my addr 172.21.15.16:0/1440478258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f918be00700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f91740a1310 0x7f91740a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9191800700 1 -- 172.21.15.16:0/1440478258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 msgr2=0x557b9d572f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9191800700 1 --2- 172.21.15.16:0/1440478258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d572f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9191800700 1 -- 172.21.15.16:0/1440478258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 msgr2=0x557b9d56d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9191800700 1 --2- 172.21.15.16:0/1440478258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 0x557b9d56d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9191800700 1 -- 172.21.15.16:0/1440478258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b9d33f350 con 0x557b9d56d5c0 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f9191800700 1 --2- 172.21.15.16:0/1440478258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d56da20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f91840090a0 tx=0x7f9184095d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d049d7d1feccc23 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.244+0000 7f917e000700 1 -- 172.21.15.16:0/1440478258 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184099270 con 0x557b9d56d5c0 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f917e000700 1 -- 172.21.15.16:0/1440478258 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91840211b0 con 0x557b9d56d5c0 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f918be00700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f91740a1310 0x7f91740a37d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x557b9d573900 tx=0x7f917801a040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f917e000700 1 -- 172.21.15.16:0/1440478258 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918409d740 con 0x557b9d56d5c0 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f91a73f5b80 1 -- 172.21.15.16:0/1440478258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 msgr2=0x557b9d56da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/1440478258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d56da20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f91840090a0 tx=0x7f9184095d50 comp rx=0 tx=0).stop 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f91a73f5b80 1 -- 172.21.15.16:0/1440478258 shutdown_connections 2024-04-05T23:36:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/1440478258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d56da20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/1440478258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d572f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/1440478258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 0x557b9d56d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f91a73f5b80 1 -- 172.21.15.16:0/1440478258 >> 172.21.15.16:0/1440478258 conn(0x557b9d4e11c0 msgr2=0x557b9d567be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.245+0000 7f91a73f5b80 1 -- 172.21.15.16:0/1440478258 shutdown_connections 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.246+0000 7f91a73f5b80 1 -- 172.21.15.16:0/1440478258 wait complete. 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.246+0000 7f91a73f5b80 1 -- start start 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.246+0000 7f91a73f5b80 1 Processor -- start 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d579460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d572b80 0x557b9d5767f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 0x557b9d576e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d4bdbe0 con 0x557b9d572b80 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d459d20 con 0x557b9d56d5c0 2024-04-05T23:36:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d4599c0 con 0x557b9d5734a0 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f91a73f5b80 10 client.0 _ll_register_callbacks cb 0x557b9d2c5340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f918be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d572b80 0x557b9d5767f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f9190e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d579460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f9190e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d579460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:41556/0 (socket says 172.21.15.16:41556) 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f9190e00700 1 -- 172.21.15.16:0/4016547169 learned_addr learned my addr 172.21.15.16:0/4016547169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f9191800700 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 0x557b9d576e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f9190e00700 1 -- 172.21.15.16:0/4016547169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 msgr2=0x557b9d576e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f9190e00700 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 0x557b9d576e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f9190e00700 1 -- 172.21.15.16:0/4016547169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d572b80 msgr2=0x557b9d5767f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f9190e00700 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d572b80 0x557b9d5767f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.247+0000 7f9190e00700 1 -- 172.21.15.16:0/4016547169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91840b7e70 con 0x557b9d56d5c0 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.248+0000 7f9190e00700 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d579460 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9180012860 tx=0x7f9180012b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c8aba545d73caea server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.248+0000 7f9172a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:36:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.248+0000 7f9172a00700 1 -- 172.21.15.16:0/4016547169 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9180017070 con 0x557b9d56d5c0 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.248+0000 7f91a73f5b80 1 -- 172.21.15.16:0/4016547169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b9d5601e0 con 0x557b9d56d5c0 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.248+0000 7f9172a00700 1 -- 172.21.15.16:0/4016547169 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f918000f540 con 0x557b9d56d5c0 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.248+0000 7f9172a00700 1 -- 172.21.15.16:0/4016547169 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91800122e0 con 0x557b9d56d5c0 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.248+0000 7f91a73f5b80 10 client.26105 fetch_fsmap learned FSMap version 37 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.248+0000 7f91a73f5b80 1 -- 172.21.15.16:0/4016547169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x557b9d2ebb00 con 0x557b9d56d5c0 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.248+0000 7f9172a00700 1 -- 172.21.15.16:0/4016547169 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9180028080 con 0x557b9d56d5c0 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f9172a00700 1 -- 172.21.15.16:0/4016547169 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f9180010880 con 0x557b9d56d5c0 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f91a73f5b80 10 client.26105 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f91a73f5b80 10 client.26105 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f91a73f5b80 20 client.26105 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557b9d586ed0 0x557b9d5892f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f91a73f5b80 4 client.26105 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:36:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f91a73f5b80 1 -- 172.21.15.16:0/4016547169 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b9d4b66a0 con 0x557b9d586ed0 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f918be00700 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557b9d586ed0 0x557b9d5892f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f9172a00700 10 client.26105 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.249+0000 7f918be00700 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557b9d586ed0 0x557b9d5892f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d969d6ed819718a7 server_cookie=fc81deef72a45ed in_seq=0 out_seq=0 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f9172a00700 1 -- 172.21.15.16:0/4016547169 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557b9d4b66a0 con 0x557b9d586ed0 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f9172a00700 10 client.26105 handle_command_reply: tid=0 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f91a73f5b80 1 client.26105 shutdown 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f91a73f5b80 1 -- 172.21.15.16:0/4016547169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 msgr2=0x557b9d579460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d579460 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9180012860 tx=0x7f9180012b70 comp rx=0 tx=0).stop 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f91a73f5b80 1 -- 172.21.15.16:0/4016547169 shutdown_connections 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557b9d586ed0 0x557b9d5892f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 0x557b9d576e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f9172a00700 0 client.26105 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d579460 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/4016547169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d572b80 0x557b9d5767f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.251+0000 7f91a73f5b80 1 -- 172.21.15.16:0/4016547169 >> 172.21.15.16:0/4016547169 conn(0x557b9d4e11c0 msgr2=0x557b9d567280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.252+0000 7f91a73f5b80 1 -- 172.21.15.16:0/4016547169 shutdown_connections 2024-04-05T23:36:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.252+0000 7f91a73f5b80 1 -- 172.21.15.16:0/4016547169 wait complete. 2024-04-05T23:36:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.252+0000 7f91a73f5b80 20 client.26105 trim_cache size 0 max 16384 2024-04-05T23:36:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.268+0000 7f91a73f5b80 1 Processor -- start 2024-04-05T23:36:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.268+0000 7f91a73f5b80 1 -- start start 2024-04-05T23:36:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.268+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d56d5c0 0x557b9d567a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.268+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d572b80 0x557b9d578f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.268+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 0x557b9d5825c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.268+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d341640 con 0x557b9d56d5c0 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.268+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d574d40 con 0x557b9d572b80 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.268+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d571480 con 0x557b9d5734a0 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f918be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d572b80 0x557b9d578f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f9190e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d56d5c0 0x557b9d567a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f9191800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 0x557b9d5825c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f918be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d572b80 0x557b9d578f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:41568/0 (socket says 172.21.15.16:41568) 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f918be00700 1 -- 172.21.15.16:0/3648469487 learned_addr learned my addr 172.21.15.16:0/3648469487 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f918be00700 1 -- 172.21.15.16:0/3648469487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 msgr2=0x557b9d5825c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f918be00700 1 --2- 172.21.15.16:0/3648469487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 0x557b9d5825c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f918be00700 1 -- 172.21.15.16:0/3648469487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d56d5c0 msgr2=0x557b9d567a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f918be00700 1 --2- 172.21.15.16:0/3648469487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d56d5c0 0x557b9d567a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f918be00700 1 -- 172.21.15.16:0/3648469487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b9d483e40 con 0x557b9d572b80 2024-04-05T23:36:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f9190e00700 1 --2- 172.21.15.16:0/3648469487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d56d5c0 0x557b9d567a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f9191800700 1 --2- 172.21.15.16:0/3648469487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 0x557b9d5825c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f918be00700 1 --2- 172.21.15.16:0/3648469487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d572b80 0x557b9d578f90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9178005cd0 tx=0x7f9178020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b043a07b8eca4be server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f917ea00700 1 -- 172.21.15.16:0/3648469487 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917802b040 con 0x557b9d572b80 2024-04-05T23:36:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f917ea00700 1 -- 172.21.15.16:0/3648469487 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f917802c040 con 0x557b9d572b80 2024-04-05T23:36:45.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.269+0000 7f917ea00700 1 -- 172.21.15.16:0/3648469487 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91780047f0 con 0x557b9d572b80 2024-04-05T23:36:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.270+0000 7f91a73f5b80 1 -- 172.21.15.16:0/3648469487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d572b80 msgr2=0x557b9d578f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.270+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/3648469487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d572b80 0x557b9d578f90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9178005cd0 tx=0x7f9178020000 comp rx=0 tx=0).stop 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.270+0000 7f91a73f5b80 1 -- 172.21.15.16:0/3648469487 shutdown_connections 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.270+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/3648469487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d56d5c0 0x557b9d567a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.270+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/3648469487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d5734a0 0x557b9d5825c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.270+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/3648469487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d572b80 0x557b9d578f90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.270+0000 7f91a73f5b80 1 -- 172.21.15.16:0/3648469487 >> 172.21.15.16:0/3648469487 conn(0x557b9d4e11c0 msgr2=0x557b9d58dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.270+0000 7f91a73f5b80 1 -- 172.21.15.16:0/3648469487 shutdown_connections 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.270+0000 7f91a73f5b80 1 -- 172.21.15.16:0/3648469487 wait complete. 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.271+0000 7f91a73f5b80 1 -- start start 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.271+0000 7f91a73f5b80 1 Processor -- start 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.271+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d57c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.271+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d57c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.271+0000 7f91a73f5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 0x557b9d5d9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.271+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d57cc60 con 0x557b9d5734a0 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.271+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d5da260 con 0x557b9d56d5c0 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.271+0000 7f91a73f5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9d5da3a0 con 0x557b9d572b80 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.271+0000 7f91a73f5b80 10 client.0 _ll_register_callbacks cb 0x7f9180009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9190e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d57c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f918be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d57c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9190e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d57c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:41580/0 (socket says 172.21.15.16:41580) 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9191800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 0x557b9d5d9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9190e00700 1 -- 172.21.15.16:0/474510797 learned_addr learned my addr 172.21.15.16:0/474510797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f918be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d57c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44288/0 (socket says 172.21.15.16:44288) 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9190e00700 1 -- 172.21.15.16:0/474510797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 msgr2=0x557b9d57c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9190e00700 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d57c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9190e00700 1 -- 172.21.15.16:0/474510797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 msgr2=0x557b9d5d9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9190e00700 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 0x557b9d5d9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9190e00700 1 -- 172.21.15.16:0/474510797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b9d2ebb00 con 0x557b9d56d5c0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9191800700 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 0x557b9d5d9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9190e00700 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d57c040 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9180005a80 tx=0x7f91800122e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42ef5f0856d3db91 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.272+0000 7f9172a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f9172a00700 1 -- 172.21.15.16:0/474510797 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918002b020 con 0x557b9d56d5c0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f9172a00700 1 -- 172.21.15.16:0/474510797 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f9180000e40 con 0x557b9d56d5c0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f91a73f5b80 1 -- 172.21.15.16:0/474510797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91840211b0 con 0x557b9d56d5c0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f9172a00700 1 -- 172.21.15.16:0/474510797 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9180017040 con 0x557b9d56d5c0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f91a73f5b80 10 client.26111 fetch_fsmap learned FSMap version 37 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f91a73f5b80 1 -- 172.21.15.16:0/474510797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x557b9d33f350 con 0x557b9d56d5c0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f9172a00700 1 -- 172.21.15.16:0/474510797 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91840211b0 con 0x557b9d56d5c0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f9172a00700 1 -- 172.21.15.16:0/474510797 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f9180005460 con 0x557b9d56d5c0 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f91a73f5b80 10 client.26111 fetch_fsmap finished waiting for FSMap version 37 2024-04-05T23:36:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f91a73f5b80 10 client.26111 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f91a73f5b80 20 client.26111 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557b9d586ed0 0x557b9d5ddc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f91a73f5b80 4 client.26111 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:36:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.273+0000 7f91a73f5b80 1 -- 172.21.15.16:0/474510797 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557b9d30cc90 con 0x557b9d586ed0 2024-04-05T23:36:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.274+0000 7f918be00700 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557b9d586ed0 0x557b9d5ddc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.274+0000 7f918be00700 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557b9d586ed0 0x557b9d5ddc00 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91e8cc57cdde7aa6 server_cookie=d6df85dafe1f2f33 in_seq=0 out_seq=0 2024-04-05T23:36:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.274+0000 7f9172a00700 10 client.26111 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f9172a00700 1 -- 172.21.15.16:0/474510797 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557b9d30cc90 con 0x557b9d586ed0 2024-04-05T23:36:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f9172a00700 10 client.26111 handle_command_reply: tid=0 2024-04-05T23:36:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 client.26111 shutdown 2024-04-05T23:36:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 -- 172.21.15.16:0/474510797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 msgr2=0x557b9d57c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d57c040 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9180005a80 tx=0x7f91800122e0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 -- 172.21.15.16:0/474510797 shutdown_connections 2024-04-05T23:36:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557b9d586ed0 0x557b9d5ddc00 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557b9d5734a0 0x557b9d5d9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557b9d56d5c0 0x557b9d57c040 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 --2- 172.21.15.16:0/474510797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557b9d572b80 0x557b9d57c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f9172a00700 0 client.26111 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 -- 172.21.15.16:0/474510797 >> 172.21.15.16:0/474510797 conn(0x557b9d4e11c0 msgr2=0x557b9d5825c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 -- 172.21.15.16:0/474510797 shutdown_connections 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.319+0000 7f91a73f5b80 1 -- 172.21.15.16:0/474510797 wait complete. 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f91a73f5b80 20 client.26111 trim_cache size 0 max 16384 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:36:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:36:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:36:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:36:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 -- 172.21.15.16:0/2429195002 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f91740a1310 msgr2=0x7f91740a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f91740a1310 0x7f91740a37d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x557b9d573900 tx=0x7f917801a040 comp rx=0 tx=0).stop 2024-04-05T23:36:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 -- 172.21.15.16:0/2429195002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f918c073c60 msgr2=0x7f918c1a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f918c073c60 0x7f918c1a54a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f918400efc0 tx=0x7f918400c3a0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 -- 172.21.15.16:0/2429195002 shutdown_connections 2024-04-05T23:36:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f91740a1310 0x7f91740a37d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f918c101030 0x7f918c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f918c073c60 0x7f918c1a54a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 --2- 172.21.15.16:0/2429195002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f918c106150 0x7f918c073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.320+0000 7f9193800700 1 -- 172.21.15.16:0/2429195002 >> 172.21.15.16:0/2429195002 conn(0x7f918c0fc8b0 msgr2=0x7f918c0fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.321+0000 7f9193800700 1 -- 172.21.15.16:0/2429195002 shutdown_connections 2024-04-05T23:36:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:45.321+0000 7f9193800700 1 -- 172.21.15.16:0/2429195002 wait complete. 2024-04-05T23:36:45.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:45 smithi016 ceph-mon[38248]: pgmap v99: 97 pgs: 97 active+clean; 64 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 257 op/s 2024-04-05T23:36:45.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:45 smithi130 ceph-mon[79417]: pgmap v99: 97 pgs: 97 active+clean; 64 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 257 op/s 2024-04-05T23:36:45.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:45 smithi027 ceph-mon[78597]: pgmap v99: 97 pgs: 97 active+clean; 64 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 257 op/s 2024-04-05T23:36:45.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 128547 8.69 MB/sec execute 5 sec latency 30.121 ms 2024-04-05T23:36:46.334 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:36:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: ignoring --setuser ceph since I am not root 2024-04-05T23:36:46.334 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:36:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: ignoring --setgroup ceph since I am not root 2024-04-05T23:36:46.334 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:36:46 smithi130 ceph-mds[121021]: -- 172.21.15.130:0/1122609995 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b0ab5bc1a0 con 0x55b0ab592800 2024-04-05T23:36:46.334 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:36:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: starting mds.f at 2024-04-05T23:36:46.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:46 smithi016 ceph-mon[38248]: daemon mds.f restarted 2024-04-05T23:36:46.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:46 smithi016 ceph-mon[38248]: osdmap e84: 12 total, 12 up, 12 in 2024-04-05T23:36:46.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:46 smithi130 ceph-mon[79417]: daemon mds.f restarted 2024-04-05T23:36:46.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:46 smithi130 ceph-mon[79417]: osdmap e84: 12 total, 12 up, 12 in 2024-04-05T23:36:46.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:46 smithi027 ceph-mon[78597]: daemon mds.f restarted 2024-04-05T23:36:46.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:46 smithi027 ceph-mon[78597]: osdmap e84: 12 total, 12 up, 12 in 2024-04-05T23:36:46.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 131091 9.57 MB/sec execute 6 sec latency 10.733 ms 2024-04-05T23:36:47.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:47 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6834/2562620223,v1:172.21.15.130:6835/2562620223] up:boot 2024-04-05T23:36:47.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:47 smithi016 ceph-mon[38248]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-05T23:36:47.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:47 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:36:47.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:47 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:36:47.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:47 smithi016 ceph-mon[38248]: pgmap v101: 97 pgs: 97 active+clean; 60 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 307 op/s 2024-04-05T23:36:47.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:47 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6834/2562620223,v1:172.21.15.130:6835/2562620223] up:boot 2024-04-05T23:36:47.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:47 smithi130 ceph-mon[79417]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-05T23:36:47.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:47 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:36:47.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:47 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:36:47.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:47 smithi130 ceph-mon[79417]: pgmap v101: 97 pgs: 97 active+clean; 60 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 307 op/s 2024-04-05T23:36:47.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:47 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6834/2562620223,v1:172.21.15.130:6835/2562620223] up:boot 2024-04-05T23:36:47.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:47 smithi027 ceph-mon[78597]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-05T23:36:47.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:47 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:36:47.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:47 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:36:47.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:47 smithi027 ceph-mon[78597]: pgmap v101: 97 pgs: 97 active+clean; 60 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 307 op/s 2024-04-05T23:36:47.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 132276 9.00 MB/sec execute 7 sec latency 10.683 ms 2024-04-05T23:36:48.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134904 9.97 MB/sec execute 8 sec latency 10.434 ms 2024-04-05T23:36:49.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:49 smithi027 ceph-mon[78597]: pgmap v102: 97 pgs: 97 active+clean; 60 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 307 op/s 2024-04-05T23:36:49.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 135892 9.24 MB/sec execute 9 sec latency 60.419 ms 2024-04-05T23:36:50.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:49 smithi016 ceph-mon[38248]: pgmap v102: 97 pgs: 97 active+clean; 60 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 307 op/s 2024-04-05T23:36:50.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:49 smithi130 ceph-mon[79417]: pgmap v102: 97 pgs: 97 active+clean; 60 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 307 op/s 2024-04-05T23:36:50.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 138428 9.88 MB/sec execute 10 sec latency 9.949 ms 2024-04-05T23:36:51.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:51 smithi027 ceph-mon[78597]: pgmap v103: 97 pgs: 97 active+clean; 64 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 313 op/s 2024-04-05T23:36:51.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 139546 9.31 MB/sec execute 11 sec latency 11.726 ms 2024-04-05T23:36:52.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:51 smithi016 ceph-mon[38248]: pgmap v103: 97 pgs: 97 active+clean; 64 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 313 op/s 2024-04-05T23:36:52.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:51 smithi130 ceph-mon[79417]: pgmap v103: 97 pgs: 97 active+clean; 64 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 313 op/s 2024-04-05T23:36:52.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 142070 9.84 MB/sec execute 12 sec latency 12.323 ms 2024-04-05T23:36:53.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:53 smithi027 ceph-mon[78597]: pgmap v104: 97 pgs: 97 active+clean; 64 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 313 op/s 2024-04-05T23:36:53.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 143364 9.58 MB/sec execute 13 sec latency 14.135 ms 2024-04-05T23:36:54.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:53 smithi016 ceph-mon[38248]: pgmap v104: 97 pgs: 97 active+clean; 64 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 313 op/s 2024-04-05T23:36:54.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:53 smithi130 ceph-mon[79417]: pgmap v104: 97 pgs: 97 active+clean; 64 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 313 op/s 2024-04-05T23:36:54.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 145658 9.87 MB/sec execute 14 sec latency 63.003 ms 2024-04-05T23:36:55.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:55 smithi027 ceph-mon[78597]: pgmap v105: 97 pgs: 97 active+clean; 64 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 313 op/s 2024-04-05T23:36:55.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 147018 9.58 MB/sec execute 15 sec latency 12.373 ms 2024-04-05T23:36:56.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:55 smithi016 ceph-mon[38248]: pgmap v105: 97 pgs: 97 active+clean; 64 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 313 op/s 2024-04-05T23:36:56.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:55 smithi130 ceph-mon[79417]: pgmap v105: 97 pgs: 97 active+clean; 64 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 313 op/s 2024-04-05T23:36:56.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149289 9.83 MB/sec execute 16 sec latency 12.283 ms 2024-04-05T23:36:57.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:57 smithi027 ceph-mon[78597]: pgmap v106: 97 pgs: 97 active+clean; 68 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 268 op/s 2024-04-05T23:36:57.990 INFO:tasks.workunit.client.0.smithi016.stdout: 1 150979 9.60 MB/sec execute 17 sec latency 26.478 ms 2024-04-05T23:36:58.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:57 smithi016 ceph-mon[38248]: pgmap v106: 97 pgs: 97 active+clean; 68 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 268 op/s 2024-04-05T23:36:58.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:57 smithi130 ceph-mon[79417]: pgmap v106: 97 pgs: 97 active+clean; 68 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 268 op/s 2024-04-05T23:36:58.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.476+0000 7fdc7a800700 1 -- 172.21.15.16:0/3858611145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc74101840 msgr2=0x7fdc74105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:58.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.476+0000 7fdc7a800700 1 --2- 172.21.15.16:0/3858611145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc74101840 0x7fdc74105880 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009990 tx=0x7fdc60009ca0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.476+0000 7fdc7a800700 1 -- 172.21.15.16:0/3858611145 shutdown_connections 2024-04-05T23:36:58.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.476+0000 7fdc7a800700 1 --2- 172.21.15.16:0/3858611145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc74105fb0 0x7fdc7410db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.476+0000 7fdc7a800700 1 --2- 172.21.15.16:0/3858611145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc74100e90 0x7fdc74101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.476+0000 7fdc7a800700 1 --2- 172.21.15.16:0/3858611145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc74101840 0x7fdc74105880 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.476+0000 7fdc7a800700 1 -- 172.21.15.16:0/3858611145 >> 172.21.15.16:0/3858611145 conn(0x7fdc740fc770 msgr2=0x7fdc740feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:58.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.476+0000 7fdc7a800700 1 -- 172.21.15.16:0/3858611145 shutdown_connections 2024-04-05T23:36:58.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.477+0000 7fdc7a800700 1 -- 172.21.15.16:0/3858611145 wait complete. 2024-04-05T23:36:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc7a800700 1 Processor -- start 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc7a800700 1 -- start start 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc7a800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc74100e90 0x7fdc741971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc7a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc74105fb0 0x7fdc741976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc7a800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc7419ba80 0x7fdc7419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc7a800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7419c500 con 0x7fdc7419ba80 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc7a800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7419c670 con 0x7fdc74100e90 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc7a800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7419c830 con 0x7fdc74105fb0 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc74100e90 0x7fdc741971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc78a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc74105fb0 0x7fdc741976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc74100e90 0x7fdc741971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:46802/0 (socket says 172.21.15.16:46802) 2024-04-05T23:36:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.478+0000 7fdc79400700 1 -- 172.21.15.16:0/2381596955 learned_addr learned my addr 172.21.15.16:0/2381596955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.479+0000 7fdc79e00700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc7419ba80 0x7fdc7419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.479+0000 7fdc79400700 1 -- 172.21.15.16:0/2381596955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc74105fb0 msgr2=0x7fdc741976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.479+0000 7fdc79400700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc74105fb0 0x7fdc741976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.479+0000 7fdc79400700 1 -- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc7419ba80 msgr2=0x7fdc7419bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.479+0000 7fdc79400700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc7419ba80 0x7fdc7419bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.479+0000 7fdc79400700 1 -- 172.21.15.16:0/2381596955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc600095f0 con 0x7fdc74100e90 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.479+0000 7fdc78a00700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc74105fb0 0x7fdc741976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.479+0000 7fdc79400700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc74100e90 0x7fdc741971a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800e9a0 tx=0x7fdc6800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.479+0000 7fdc72000700 1 -- 172.21.15.16:0/2381596955 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68009790 con 0x7fdc74100e90 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.480+0000 7fdc72000700 1 -- 172.21.15.16:0/2381596955 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc68010430 con 0x7fdc74100e90 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.480+0000 7fdc72000700 1 -- 172.21.15.16:0/2381596955 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68018680 con 0x7fdc74100e90 2024-04-05T23:36:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.480+0000 7fdc7a800700 1 -- 172.21.15.16:0/2381596955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc741a3d90 con 0x7fdc74100e90 2024-04-05T23:36:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.480+0000 7fdc7a800700 1 -- 172.21.15.16:0/2381596955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc741a4210 con 0x7fdc74100e90 2024-04-05T23:36:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.484+0000 7fdc72000700 1 -- 172.21.15.16:0/2381596955 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fdc680105a0 con 0x7fdc74100e90 2024-04-05T23:36:58.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.486+0000 7fdc8f87cb80 1 Processor -- start 2024-04-05T23:36:58.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.486+0000 7fdc72000700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc5c0a1310 0x7fdc5c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.486+0000 7fdc8f87cb80 1 -- start start 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.486+0000 7fdc72000700 1 -- 172.21.15.16:0/2381596955 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fdc68014070 con 0x7fdc74100e90 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.486+0000 7fdc79e00700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc5c0a1310 0x7fdc5c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.486+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b07a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b0bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 0x55e605b0c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605a69b80 con 0x55e605b07f50 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605a5a930 con 0x55e605b07630 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605a582f0 con 0x55e605b0c2d0 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b07a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b07a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:46816/0 (socket says 172.21.15.16:46816) 2024-04-05T23:36:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc79400700 1 -- 172.21.15.16:0/2587304018 learned_addr learned my addr 172.21.15.16:0/2587304018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc78a00700 1 --2- 172.21.15.16:0/2587304018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b0bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc79400700 1 -- 172.21.15.16:0/2587304018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 msgr2=0x55e605b0c730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc79400700 1 --2- 172.21.15.16:0/2587304018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 0x55e605b0c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc79400700 1 -- 172.21.15.16:0/2587304018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 msgr2=0x55e605b0bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc79400700 1 --2- 172.21.15.16:0/2587304018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b0bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc79400700 1 -- 172.21.15.16:0/2587304018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6058d5350 con 0x55e605b07630 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.487+0000 7fdc79400700 1 --2- 172.21.15.16:0/2587304018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b07a10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdc680969b0 tx=0x7fdc68099bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b9e2b4fcbe6dd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.488+0000 7fdc79e00700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc5c0a1310 0x7fdc5c0a37d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00f590 tx=0x7fdc6c00b040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.488+0000 7fdc66600700 1 -- 172.21.15.16:0/2587304018 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc680a23c0 con 0x55e605b07630 2024-04-05T23:36:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.488+0000 7fdc66600700 1 -- 172.21.15.16:0/2587304018 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc68010880 con 0x55e605b07630 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.488+0000 7fdc66600700 1 -- 172.21.15.16:0/2587304018 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68099670 con 0x55e605b07630 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.488+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2587304018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 msgr2=0x55e605b07a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.488+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2587304018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b07a10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdc680969b0 tx=0x7fdc68099bb0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.489+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2587304018 shutdown_connections 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.489+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2587304018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 0x55e605b0c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.489+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2587304018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b07a10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.489+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2587304018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b0bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.489+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2587304018 >> 172.21.15.16:0/2587304018 conn(0x55e605a771c0 msgr2=0x55e605afe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.489+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2587304018 shutdown_connections 2024-04-05T23:36:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.489+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2587304018 wait complete. 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.489+0000 7fdc8f87cb80 1 -- start start 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.489+0000 7fdc8f87cb80 1 Processor -- start 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07630 0x55e605b10e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b11380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b118c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605a53be0 con 0x55e605b07f50 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6059efd20 con 0x55e605b0c2d0 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6059ef9c0 con 0x55e605b07630 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc8f87cb80 10 client.0 _ll_register_callbacks cb 0x55e60585b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc79400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07630 0x55e605b10e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc79e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b118c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc79400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07630 0x55e605b10e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39910/0 (socket says 172.21.15.16:39910) 2024-04-05T23:36:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc79e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b118c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:46830/0 (socket says 172.21.15.16:46830) 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc79400700 1 -- 172.21.15.16:0/3872587795 learned_addr learned my addr 172.21.15.16:0/3872587795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.490+0000 7fdc78a00700 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b11380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc79400700 1 -- 172.21.15.16:0/3872587795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 msgr2=0x55e605b118c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc79400700 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b118c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc79400700 1 -- 172.21.15.16:0/3872587795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 msgr2=0x55e605b11380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc79400700 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b11380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc79400700 1 -- 172.21.15.16:0/3872587795 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc741a4210 con 0x55e605b07630 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc79e00700 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b118c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc79400700 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07630 0x55e605b10e40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc68099c40 tx=0x7fdc68010a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7919728f07265ed server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc5aa00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc5aa00700 1 -- 172.21.15.16:0/3872587795 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc680a23c0 con 0x55e605b07630 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc5aa00700 1 -- 172.21.15.16:0/3872587795 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fdc680a2140 con 0x55e605b07630 2024-04-05T23:36:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/3872587795 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e605af61e0 con 0x55e605b07630 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc5aa00700 1 -- 172.21.15.16:0/3872587795 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc680b8270 con 0x55e605b07630 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc8f87cb80 10 client.26161 fetch_fsmap learned FSMap version 39 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.491+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/3872587795 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55e605881b00 con 0x55e605b07630 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.492+0000 7fdc5aa00700 1 -- 172.21.15.16:0/3872587795 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e605af61e0 con 0x55e605b07630 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.492+0000 7fdc5aa00700 1 -- 172.21.15.16:0/3872587795 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 39) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fdc6807f070 con 0x55e605b07630 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.492+0000 7fdc8f87cb80 10 client.26161 fetch_fsmap finished waiting for FSMap version 39 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.492+0000 7fdc8f87cb80 10 client.26161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.492+0000 7fdc8f87cb80 20 client.26161 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.492+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e605b21200 0x55e605b23620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.492+0000 7fdc8f87cb80 4 client.26161 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.492+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/3872587795 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e605a4c6a0 con 0x55e605b21200 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.492+0000 7fdc78a00700 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e605b21200 0x55e605b23620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.493+0000 7fdc78a00700 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e605b21200 0x55e605b23620 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1991167ca2741b5 server_cookie=e6e6f4fbee2383d4 in_seq=0 out_seq=0 2024-04-05T23:36:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.493+0000 7fdc5aa00700 10 client.26161 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.493+0000 7fdc5aa00700 1 -- 172.21.15.16:0/3872587795 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e605a4c6a0 con 0x55e605b21200 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.493+0000 7fdc5aa00700 10 client.26161 handle_command_reply: tid=0 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.493+0000 7fdc8f87cb80 1 client.26161 shutdown 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/3872587795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07630 msgr2=0x55e605b10e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07630 0x55e605b10e40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc68099c40 tx=0x7fdc68010a60 comp rx=0 tx=0).stop 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/3872587795 shutdown_connections 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e605b21200 0x55e605b23620 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b118c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07630 0x55e605b10e40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc5aa00700 0 client.26161 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/3872587795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b11380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/3872587795 >> 172.21.15.16:0/3872587795 conn(0x55e605a771c0 msgr2=0x55e605afddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/3872587795 shutdown_connections 2024-04-05T23:36:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/3872587795 wait complete. 2024-04-05T23:36:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.494+0000 7fdc8f87cb80 20 client.26161 trim_cache size 0 max 16384 2024-04-05T23:36:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.510+0000 7fdc8f87cb80 1 Processor -- start 2024-04-05T23:36:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.510+0000 7fdc8f87cb80 1 -- start start 2024-04-05T23:36:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07630 0x55e605b09bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07f50 0x55e605b184f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b0d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6058d7640 con 0x55e605b07630 2024-04-05T23:36:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605a4c6a0 con 0x55e605b0c2d0 2024-04-05T23:36:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605affaf0 con 0x55e605b07f50 2024-04-05T23:36:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07630 0x55e605b09bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b0d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07630 0x55e605b09bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41824/0 (socket says 172.21.15.16:41824) 2024-04-05T23:36:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79400700 1 -- 172.21.15.16:0/2749404052 learned_addr learned my addr 172.21.15.16:0/2749404052 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79e00700 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b0d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:46844/0 (socket says 172.21.15.16:46844) 2024-04-05T23:36:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc78a00700 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07f50 0x55e605b184f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79e00700 1 -- 172.21.15.16:0/2749404052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07f50 msgr2=0x55e605b184f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79e00700 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07f50 0x55e605b184f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79e00700 1 -- 172.21.15.16:0/2749404052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07630 msgr2=0x55e605b09bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79e00700 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07630 0x55e605b09bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79e00700 1 -- 172.21.15.16:0/2749404052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e605b28210 con 0x55e605b0c2d0 2024-04-05T23:36:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.511+0000 7fdc79400700 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07630 0x55e605b09bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:36:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.512+0000 7fdc79e00700 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b0d6c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00f400 tx=0x7fdc6c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91bf3a507c206dd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.512+0000 7fdc67000700 1 -- 172.21.15.16:0/2749404052 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c00e040 con 0x55e605b0c2d0 2024-04-05T23:36:58.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.512+0000 7fdc67000700 1 -- 172.21.15.16:0/2749404052 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc6c00f040 con 0x55e605b0c2d0 2024-04-05T23:36:58.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.512+0000 7fdc67000700 1 -- 172.21.15.16:0/2749404052 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c008dc0 con 0x55e605b0c2d0 2024-04-05T23:36:58.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.512+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2749404052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 msgr2=0x55e605b0d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:58.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.512+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b0d6c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00f400 tx=0x7fdc6c014000 comp rx=0 tx=0).stop 2024-04-05T23:36:58.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.513+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2749404052 shutdown_connections 2024-04-05T23:36:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.513+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07630 0x55e605b09bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.513+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b0c2d0 0x55e605b0d6c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.513+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2749404052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b07f50 0x55e605b184f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.513+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2749404052 >> 172.21.15.16:0/2749404052 conn(0x55e605a771c0 msgr2=0x55e605b0a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.513+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2749404052 shutdown_connections 2024-04-05T23:36:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.513+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2749404052 wait complete. 2024-04-05T23:36:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.513+0000 7fdc8f87cb80 1 -- start start 2024-04-05T23:36:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.513+0000 7fdc8f87cb80 1 Processor -- start 2024-04-05T23:36:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b146e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b14c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc8f87cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 0x55e605b17a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605b17fd0 con 0x55e605b07f50 2024-04-05T23:36:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605b18140 con 0x55e605b07630 2024-04-05T23:36:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc8f87cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605b18300 con 0x55e605b0c2d0 2024-04-05T23:36:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc8f87cb80 10 client.0 _ll_register_callbacks cb 0x7fdc680998a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:36:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b146e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc78a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b14c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 0x55e605b17a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b146e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:46856/0 (socket says 172.21.15.16:46856) 2024-04-05T23:36:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79400700 1 -- 172.21.15.16:0/2243216852 learned_addr learned my addr 172.21.15.16:0/2243216852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:36:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79400700 1 -- 172.21.15.16:0/2243216852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 msgr2=0x55e605b17a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79400700 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 0x55e605b17a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79400700 1 -- 172.21.15.16:0/2243216852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 msgr2=0x55e605b14c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:36:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79400700 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b14c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79400700 1 -- 172.21.15.16:0/2243216852 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e605881b00 con 0x55e605b07630 2024-04-05T23:36:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.514+0000 7fdc79e00700 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 0x55e605b17a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:36:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc79400700 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b146e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc68099c40 tx=0x7fdc68099950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5670197059a7818 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:36:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc5aa00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:36:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc5aa00700 1 -- 172.21.15.16:0/2243216852 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68095f70 con 0x55e605b07630 2024-04-05T23:36:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc5aa00700 1 -- 172.21.15.16:0/2243216852 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7fdc68095d30 con 0x55e605b07630 2024-04-05T23:36:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2243216852 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc68010880 con 0x55e605b07630 2024-04-05T23:36:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc5aa00700 1 -- 172.21.15.16:0/2243216852 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc680a23c0 con 0x55e605b07630 2024-04-05T23:36:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc8f87cb80 10 client.26135 fetch_fsmap learned FSMap version 39 2024-04-05T23:36:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2243216852 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x7fdc6c01c040 con 0x55e605b07630 2024-04-05T23:36:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc5aa00700 1 -- 172.21.15.16:0/2243216852 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc68010880 con 0x55e605b07630 2024-04-05T23:36:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc5aa00700 1 -- 172.21.15.16:0/2243216852 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 39) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fdc6807f070 con 0x55e605b07630 2024-04-05T23:36:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc8f87cb80 10 client.26135 fetch_fsmap finished waiting for FSMap version 39 2024-04-05T23:36:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc8f87cb80 10 client.26135 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:36:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.515+0000 7fdc8f87cb80 20 client.26135 populate_metadata read hostname 'smithi016' 2024-04-05T23:36:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.516+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e605b75c20 0x55e605b78040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:36:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.516+0000 7fdc8f87cb80 4 client.26135 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:36:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.516+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2243216852 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e6058a2c90 con 0x55e605b75c20 2024-04-05T23:36:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.516+0000 7fdc78a00700 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e605b75c20 0x55e605b78040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:36:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.516+0000 7fdc5aa00700 10 client.26135 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.516+0000 7fdc78a00700 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e605b75c20 0x55e605b78040 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcc4f69adb2368a0 server_cookie=e0ba5ed8546bab16 in_seq=0 out_seq=0 2024-04-05T23:36:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.556+0000 7fdc5aa00700 1 -- 172.21.15.16:0/2243216852 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e6058a2c90 con 0x55e605b75c20 2024-04-05T23:36:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.556+0000 7fdc5aa00700 10 client.26135 handle_command_reply: tid=0 2024-04-05T23:36:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.556+0000 7fdc8f87cb80 1 client.26135 shutdown 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2243216852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 msgr2=0x55e605b146e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b146e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc68099c40 tx=0x7fdc68099950 comp rx=0 tx=0).stop 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2243216852 shutdown_connections 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e605b75c20 0x55e605b78040 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e605b0c2d0 0x55e605b17a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc5aa00700 0 client.26135 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e605b07630 0x55e605b146e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 --2- 172.21.15.16:0/2243216852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e605b07f50 0x55e605b14c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2243216852 >> 172.21.15.16:0/2243216852 conn(0x55e605a771c0 msgr2=0x55e605b0e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2243216852 shutdown_connections 2024-04-05T23:36:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 1 -- 172.21.15.16:0/2243216852 wait complete. 2024-04-05T23:36:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.557+0000 7fdc8f87cb80 20 client.26135 trim_cache size 0 max 16384 2024-04-05T23:36:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:36:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:36:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:36:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 -- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc5c0a1310 msgr2=0x7fdc5c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc5c0a1310 0x7fdc5c0a37d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00f590 tx=0x7fdc6c00b040 comp rx=0 tx=0).stop 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 -- 172.21.15.16:0/2381596955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc74100e90 msgr2=0x7fdc741971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc74100e90 0x7fdc741971a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800e9a0 tx=0x7fdc6800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 -- 172.21.15.16:0/2381596955 shutdown_connections 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc5c0a1310 0x7fdc5c0a37d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc7419ba80 0x7fdc7419bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc74100e90 0x7fdc741971a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 --2- 172.21.15.16:0/2381596955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc74105fb0 0x7fdc741976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 -- 172.21.15.16:0/2381596955 >> 172.21.15.16:0/2381596955 conn(0x7fdc740fc770 msgr2=0x7fdc740fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:36:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 -- 172.21.15.16:0/2381596955 shutdown_connections 2024-04-05T23:36:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:36:58.558+0000 7fdc7a800700 1 -- 172.21.15.16:0/2381596955 wait complete. 2024-04-05T23:36:58.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 152843 9.80 MB/sec execute 18 sec latency 72.776 ms 2024-04-05T23:36:59.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:36:59 smithi027 ceph-mon[78597]: pgmap v107: 97 pgs: 97 active+clean; 68 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 237 op/s 2024-04-05T23:36:59.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 154209 9.58 MB/sec execute 19 sec latency 19.646 ms 2024-04-05T23:37:00.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:36:59 smithi016 ceph-mon[38248]: pgmap v107: 97 pgs: 97 active+clean; 68 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 237 op/s 2024-04-05T23:37:00.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:36:59 smithi130 ceph-mon[79417]: pgmap v107: 97 pgs: 97 active+clean; 68 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 237 op/s 2024-04-05T23:37:00.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 156282 9.75 MB/sec execute 20 sec latency 21.101 ms 2024-04-05T23:37:01.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 157272 9.43 MB/sec execute 21 sec latency 23.596 ms 2024-04-05T23:37:02.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:01 smithi016 ceph-mon[38248]: pgmap v108: 97 pgs: 97 active+clean; 72 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 379 op/s 2024-04-05T23:37:02.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:01 smithi130 ceph-mon[79417]: pgmap v108: 97 pgs: 97 active+clean; 72 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 379 op/s 2024-04-05T23:37:02.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:01 smithi027 ceph-mon[78597]: pgmap v108: 97 pgs: 97 active+clean; 72 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 379 op/s 2024-04-05T23:37:02.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 159709 9.71 MB/sec execute 22 sec latency 38.390 ms 2024-04-05T23:37:03.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 160827 9.44 MB/sec execute 23 sec latency 11.430 ms 2024-04-05T23:37:04.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:03 smithi016 ceph-mon[38248]: pgmap v109: 97 pgs: 97 active+clean; 72 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 260 op/s 2024-04-05T23:37:04.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:03 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:04.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:03 smithi130 ceph-mon[79417]: pgmap v109: 97 pgs: 97 active+clean; 72 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 260 op/s 2024-04-05T23:37:04.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:03 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:04.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:03 smithi027 ceph-mon[78597]: pgmap v109: 97 pgs: 97 active+clean; 72 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 260 op/s 2024-04-05T23:37:04.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:03 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:04.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 163302 9.70 MB/sec execute 24 sec latency 43.129 ms 2024-04-05T23:37:05.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 164405 9.45 MB/sec execute 25 sec latency 14.814 ms 2024-04-05T23:37:06.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:05 smithi016 ceph-mon[38248]: pgmap v110: 97 pgs: 97 active+clean; 72 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 260 op/s 2024-04-05T23:37:06.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:05 smithi130 ceph-mon[79417]: pgmap v110: 97 pgs: 97 active+clean; 72 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 260 op/s 2024-04-05T23:37:06.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:05 smithi027 ceph-mon[78597]: pgmap v110: 97 pgs: 97 active+clean; 72 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 260 op/s 2024-04-05T23:37:06.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 166926 9.73 MB/sec execute 26 sec latency 11.596 ms 2024-04-05T23:37:07.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 168185 9.57 MB/sec execute 27 sec latency 18.550 ms 2024-04-05T23:37:08.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:07 smithi016 ceph-mon[38248]: pgmap v111: 97 pgs: 97 active+clean; 75 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 405 op/s 2024-04-05T23:37:08.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:07 smithi130 ceph-mon[79417]: pgmap v111: 97 pgs: 97 active+clean; 75 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 405 op/s 2024-04-05T23:37:08.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:07 smithi027 ceph-mon[78597]: pgmap v111: 97 pgs: 97 active+clean; 75 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 405 op/s 2024-04-05T23:37:08.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 170518 9.72 MB/sec execute 28 sec latency 12.698 ms 2024-04-05T23:37:09.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:08 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:09.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:08 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:09.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:08 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:09.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 171699 9.57 MB/sec execute 29 sec latency 40.008 ms 2024-04-05T23:37:10.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:09 smithi016 ceph-mon[38248]: pgmap v112: 97 pgs: 97 active+clean; 75 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 287 op/s 2024-04-05T23:37:10.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:09 smithi130 ceph-mon[79417]: pgmap v112: 97 pgs: 97 active+clean; 75 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 287 op/s 2024-04-05T23:37:10.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:09 smithi027 ceph-mon[78597]: pgmap v112: 97 pgs: 97 active+clean; 75 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 287 op/s 2024-04-05T23:37:10.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174060 9.71 MB/sec execute 30 sec latency 17.571 ms 2024-04-05T23:37:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.721+0000 7f27b5800700 1 -- 172.21.15.16:0/3274952160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 msgr2=0x7f27b010db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.721+0000 7f27b5800700 1 --2- 172.21.15.16:0/3274952160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 0x7f27b010db20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f27a800b0a0 tx=0x7f27a800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.721+0000 7f27b5800700 1 -- 172.21.15.16:0/3274952160 shutdown_connections 2024-04-05T23:37:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.721+0000 7f27b5800700 1 --2- 172.21.15.16:0/3274952160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 0x7f27b010db20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.721+0000 7f27b5800700 1 --2- 172.21.15.16:0/3274952160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b0100e80 0x7f27b0101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.721+0000 7f27b5800700 1 --2- 172.21.15.16:0/3274952160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0101830 0x7f27b0105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.722+0000 7f27b5800700 1 -- 172.21.15.16:0/3274952160 >> 172.21.15.16:0/3274952160 conn(0x7f27b00fc740 msgr2=0x7f27b00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.722+0000 7f27b5800700 1 -- 172.21.15.16:0/3274952160 shutdown_connections 2024-04-05T23:37:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.722+0000 7f27b5800700 1 -- 172.21.15.16:0/3274952160 wait complete. 2024-04-05T23:37:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.723+0000 7f27b5800700 1 Processor -- start 2024-04-05T23:37:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.723+0000 7f27b5800700 1 -- start start 2024-04-05T23:37:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.723+0000 7f27b5800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b0100e80 0x7f27b00731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.724+0000 7f27b5800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 0x7f27b0073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.724+0000 7f27b5800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0074ea0 0x7f27b0073ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.724+0000 7f27b5800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01a9cc0 con 0x7f27b0074ea0 2024-04-05T23:37:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.724+0000 7f27b5800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01a9e30 con 0x7f27b0105fa0 2024-04-05T23:37:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.724+0000 7f27b5800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01a9ff0 con 0x7f27b0100e80 2024-04-05T23:37:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.724+0000 7f27afe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b0100e80 0x7f27b00731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.724+0000 7f27af400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 0x7f27b0073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.724+0000 7f27af400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 0x7f27b0073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:40586/0 (socket says 172.21.15.16:40586) 2024-04-05T23:37:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.724+0000 7f27af400700 1 -- 172.21.15.16:0/1542884260 learned_addr learned my addr 172.21.15.16:0/1542884260 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.725+0000 7f27af400700 1 -- 172.21.15.16:0/1542884260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b0100e80 msgr2=0x7f27b00731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.725+0000 7f27af400700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b0100e80 0x7f27b00731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.725+0000 7f27af400700 1 -- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0074ea0 msgr2=0x7f27b0073ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T23:37:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.726+0000 7f27b4e00700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0074ea0 0x7f27b0073ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.726+0000 7f27af400700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0074ea0 0x7f27b0073ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.726+0000 7f27af400700 1 -- 172.21.15.16:0/1542884260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a8009d00 con 0x7f27b0105fa0 2024-04-05T23:37:11.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.726+0000 7f27b4e00700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0074ea0 0x7f27b0073ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:37:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.727+0000 7f27af400700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 0x7f27b0073720 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f279c00e970 tx=0x7f279c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:11.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.729+0000 7f27acc00700 1 -- 172.21.15.16:0/1542884260 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c009790 con 0x7f27b0105fa0 2024-04-05T23:37:11.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.729+0000 7f27acc00700 1 -- 172.21.15.16:0/1542884260 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f279c00eec0 con 0x7f27b0105fa0 2024-04-05T23:37:11.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.729+0000 7f27b5800700 1 -- 172.21.15.16:0/1542884260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b01aa2d0 con 0x7f27b0105fa0 2024-04-05T23:37:11.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.729+0000 7f27acc00700 1 -- 172.21.15.16:0/1542884260 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c005590 con 0x7f27b0105fa0 2024-04-05T23:37:11.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.729+0000 7f27b5800700 1 -- 172.21.15.16:0/1542884260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b01aa830 con 0x7f27b0105fa0 2024-04-05T23:37:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.733+0000 7f27acc00700 1 -- 172.21.15.16:0/1542884260 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f279c00cbd0 con 0x7f27b0105fa0 2024-04-05T23:37:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.736+0000 7f27acc00700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27940a1310 0x7f27940a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.736+0000 7f27afe00700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27940a1310 0x7f27940a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.736+0000 7f27ca913b80 1 Processor -- start 2024-04-05T23:37:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.737+0000 7f27acc00700 1 -- 172.21.15.16:0/1542884260 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f279c014070 con 0x7f27b0105fa0 2024-04-05T23:37:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.737+0000 7f27ca913b80 1 -- start start 2024-04-05T23:37:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.737+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616e3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.737+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616e7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.737+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 0x55c9616e8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.737+0000 7f27ca913b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c961649b80 con 0x55c9616e3b40 2024-04-05T23:37:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.737+0000 7f27ca913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c96163a930 con 0x55c9616e7ec0 2024-04-05T23:37:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.737+0000 7f27ca913b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9616382f0 con 0x55c9616e3220 2024-04-05T23:37:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.737+0000 7f27afe00700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27940a1310 0x7f27940a37d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009c60 tx=0x7f27a0009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27afe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616e3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27b4e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 0x55c9616e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27afe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616e3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55088/0 (socket says 172.21.15.16:55088) 2024-04-05T23:37:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27afe00700 1 -- 172.21.15.16:0/1691259593 learned_addr learned my addr 172.21.15.16:0/1691259593 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27af400700 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616e7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27b4e00700 1 -- 172.21.15.16:0/1691259593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 msgr2=0x55c9616e3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27b4e00700 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616e3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27b4e00700 1 -- 172.21.15.16:0/1691259593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 msgr2=0x55c9616e7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27b4e00700 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616e7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27b4e00700 1 -- 172.21.15.16:0/1691259593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9614b5350 con 0x55c9616e7ec0 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27afe00700 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616e3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27af400700 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616e7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.738+0000 7f27b4e00700 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 0x55c9616e8300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f27a800b780 tx=0x7f27a8009450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19740aa13cfc12f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27a5600700 1 -- 172.21.15.16:0/1691259593 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a800e040 con 0x55c9616e7ec0 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27a5600700 1 -- 172.21.15.16:0/1691259593 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27a8007b80 con 0x55c9616e7ec0 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27a5600700 1 -- 172.21.15.16:0/1691259593 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a8007740 con 0x55c9616e7ec0 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27ca913b80 1 -- 172.21.15.16:0/1691259593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 msgr2=0x55c9616e8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 0x55c9616e8300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f27a800b780 tx=0x7f27a8009450 comp rx=0 tx=0).stop 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27ca913b80 1 -- 172.21.15.16:0/1691259593 shutdown_connections 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 0x55c9616e8300 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616e3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1691259593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616e7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.739+0000 7f27ca913b80 1 -- 172.21.15.16:0/1691259593 >> 172.21.15.16:0/1691259593 conn(0x55c9616571c0 msgr2=0x55c9616de060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.740+0000 7f27ca913b80 1 -- 172.21.15.16:0/1691259593 shutdown_connections 2024-04-05T23:37:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.740+0000 7f27ca913b80 1 -- 172.21.15.16:0/1691259593 wait complete. 2024-04-05T23:37:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.740+0000 7f27ca913b80 1 -- start start 2024-04-05T23:37:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.741+0000 7f27ca913b80 1 Processor -- start 2024-04-05T23:37:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.741+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616ef730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.741+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616eca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.741+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 0x55c9616ed160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.741+0000 7f27ca913b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c961633be0 con 0x55c9616e3b40 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.741+0000 7f27ca913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9615cfd20 con 0x55c9616e7ec0 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.741+0000 7f27ca913b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9615cf9c0 con 0x55c9616e3220 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.741+0000 7f27ca913b80 10 client.0 _ll_register_callbacks cb 0x55c96143b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27af400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616eca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27af400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616eca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47628/0 (socket says 172.21.15.16:47628) 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27af400700 1 -- 172.21.15.16:0/1746179900 learned_addr learned my addr 172.21.15.16:0/1746179900 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27afe00700 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27af400700 1 -- 172.21.15.16:0/1746179900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 msgr2=0x55c9616ef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27af400700 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616ef730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27af400700 1 -- 172.21.15.16:0/1746179900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 msgr2=0x55c9616ed160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27af400700 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 0x55c9616ed160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27af400700 1 -- 172.21.15.16:0/1746179900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a0017070 con 0x55c9616e3b40 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.742+0000 7f27afe00700 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.743+0000 7f27af400700 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616eca20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f279c099f20 tx=0x7f279c09a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32ee629cb58cb21 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.743+0000 7f279a000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.743+0000 7f279a000700 1 -- 172.21.15.16:0/1746179900 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c09a3f0 con 0x55c9616e3b40 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.743+0000 7f279a000700 1 -- 172.21.15.16:0/1746179900 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f279c0b8300 con 0x55c9616e3b40 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.743+0000 7f27ca913b80 1 -- 172.21.15.16:0/1746179900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c9616d61e0 con 0x55c9616e3b40 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.743+0000 7f279a000700 1 -- 172.21.15.16:0/1746179900 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f279c099480 con 0x55c9616e3b40 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.743+0000 7f27ca913b80 10 client.16101 fetch_fsmap learned FSMap version 41 2024-04-05T23:37:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.743+0000 7f27ca913b80 1 -- 172.21.15.16:0/1746179900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x7f27b010e390 con 0x55c9616e3b40 2024-04-05T23:37:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.744+0000 7f279a000700 1 -- 172.21.15.16:0/1746179900 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c9616d61e0 con 0x55c9616e3b40 2024-04-05T23:37:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.744+0000 7f279a000700 1 -- 172.21.15.16:0/1746179900 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 41) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f279c094ba0 con 0x55c9616e3b40 2024-04-05T23:37:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.744+0000 7f27ca913b80 10 client.16101 fetch_fsmap finished waiting for FSMap version 41 2024-04-05T23:37:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.744+0000 7f27ca913b80 10 client.16101 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.744+0000 7f27ca913b80 20 client.16101 populate_metadata read hostname 'smithi016' 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.744+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c9616fd190 0x55c9616ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.744+0000 7f27ca913b80 4 client.16101 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.744+0000 7f27ca913b80 1 -- 172.21.15.16:0/1746179900 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c96162c6a0 con 0x55c9616fd190 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.744+0000 7f27afe00700 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c9616fd190 0x55c9616ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.745+0000 7f279a000700 10 client.16101 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.745+0000 7f27afe00700 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c9616fd190 0x55c9616ff5b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3742673e74078f8c server_cookie=3fdbfcd207f79127 in_seq=0 out_seq=0 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f279a000700 1 -- 172.21.15.16:0/1746179900 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c96162c6a0 con 0x55c9616fd190 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f279a000700 10 client.16101 handle_command_reply: tid=0 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 client.16101 shutdown 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 -- 172.21.15.16:0/1746179900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 msgr2=0x55c9616eca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616eca20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f279c099f20 tx=0x7f279c09a040 comp rx=0 tx=0).stop 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 -- 172.21.15.16:0/1746179900 shutdown_connections 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c9616fd190 0x55c9616ff5b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e7ec0 0x55c9616ed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 --2- 172.21.15.16:0/1746179900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e3b40 0x55c9616eca20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 -- 172.21.15.16:0/1746179900 >> 172.21.15.16:0/1746179900 conn(0x55c9616571c0 msgr2=0x55c9616dd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f279a000700 0 client.16101 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.746+0000 7f27ca913b80 1 -- 172.21.15.16:0/1746179900 shutdown_connections 2024-04-05T23:37:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.747+0000 7f27ca913b80 1 -- 172.21.15.16:0/1746179900 wait complete. 2024-04-05T23:37:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.747+0000 7f27ca913b80 20 client.16101 trim_cache size 0 max 16384 2024-04-05T23:37:11.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.766+0000 7f27ca913b80 1 Processor -- start 2024-04-05T23:37:11.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.766+0000 7f27ca913b80 1 -- start start 2024-04-05T23:37:11.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3b40 0x55c9616e92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 0x55c9616f8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27ca913b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9614b7640 con 0x55c9616e7ec0 2024-04-05T23:37:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27ca913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9616defc0 con 0x55c9616e3b40 2024-04-05T23:37:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27ca913b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9616e41d0 con 0x55c9616e3220 2024-04-05T23:37:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27af400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3b40 0x55c9616e92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27b4e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 0x55c9616f8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27af400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3b40 0x55c9616e92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:40614/0 (socket says 172.21.15.16:40614) 2024-04-05T23:37:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27af400700 1 -- 172.21.15.16:0/3614590658 learned_addr learned my addr 172.21.15.16:0/3614590658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27afe00700 1 --2- 172.21.15.16:0/3614590658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27af400700 1 -- 172.21.15.16:0/3614590658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 msgr2=0x55c9616f40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.767+0000 7f27af400700 1 --2- 172.21.15.16:0/3614590658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616f40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.768+0000 7f27af400700 1 -- 172.21.15.16:0/3614590658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 msgr2=0x55c9616f8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.768+0000 7f27af400700 1 --2- 172.21.15.16:0/3614590658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 0x55c9616f8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.768+0000 7f27af400700 1 -- 172.21.15.16:0/3614590658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c961461b00 con 0x55c9616e3b40 2024-04-05T23:37:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.768+0000 7f27b4e00700 1 --2- 172.21.15.16:0/3614590658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 0x55c9616f8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:37:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.768+0000 7f27af400700 1 --2- 172.21.15.16:0/3614590658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3b40 0x55c9616e92b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f279c099950 tx=0x7f279c0a2340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0f6daba994bffd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.768+0000 7f27a6000700 1 -- 172.21.15.16:0/3614590658 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c09de40 con 0x55c9616e3b40 2024-04-05T23:37:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.768+0000 7f27a6000700 1 -- 172.21.15.16:0/3614590658 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f279c095e50 con 0x55c9616e3b40 2024-04-05T23:37:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.768+0000 7f27a6000700 1 -- 172.21.15.16:0/3614590658 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c099480 con 0x55c9616e3b40 2024-04-05T23:37:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.769+0000 7f27ca913b80 1 -- 172.21.15.16:0/3614590658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3b40 msgr2=0x55c9616e92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.769+0000 7f27ca913b80 1 --2- 172.21.15.16:0/3614590658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3b40 0x55c9616e92b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f279c099950 tx=0x7f279c0a2340 comp rx=0 tx=0).stop 2024-04-05T23:37:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.769+0000 7f27ca913b80 1 -- 172.21.15.16:0/3614590658 shutdown_connections 2024-04-05T23:37:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.769+0000 7f27ca913b80 1 --2- 172.21.15.16:0/3614590658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 0x55c9616f8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.769+0000 7f27ca913b80 1 --2- 172.21.15.16:0/3614590658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3220 0x55c9616f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.769+0000 7f27ca913b80 1 --2- 172.21.15.16:0/3614590658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3b40 0x55c9616e92b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.769+0000 7f27ca913b80 1 -- 172.21.15.16:0/3614590658 >> 172.21.15.16:0/3614590658 conn(0x55c9616571c0 msgr2=0x55c9616df580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.769+0000 7f27ca913b80 1 -- 172.21.15.16:0/3614590658 shutdown_connections 2024-04-05T23:37:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.769+0000 7f27ca913b80 1 -- 172.21.15.16:0/3614590658 wait complete. 2024-04-05T23:37:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.770+0000 7f27ca913b80 1 -- start start 2024-04-05T23:37:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.770+0000 7f27ca913b80 1 Processor -- start 2024-04-05T23:37:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3220 0x55c9616f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3b40 0x55c9616e8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27ca913b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 0x55c9616e9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27ca913b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9616ee250 con 0x55c9616e7ec0 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27ca913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9616ee3c0 con 0x55c9616e3220 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27ca913b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9616ee580 con 0x55c9616e3b40 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27afe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3220 0x55c9616f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27ca913b80 10 client.0 _ll_register_callbacks cb 0x7f279c004f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27afe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3220 0x55c9616f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:40620/0 (socket says 172.21.15.16:40620) 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27afe00700 1 -- 172.21.15.16:0/922534524 learned_addr learned my addr 172.21.15.16:0/922534524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27af400700 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3b40 0x55c9616e8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.771+0000 7f27b4e00700 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 0x55c9616e9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f27afe00700 1 -- 172.21.15.16:0/922534524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3b40 msgr2=0x55c9616e8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f27afe00700 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3b40 0x55c9616e8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f27afe00700 1 -- 172.21.15.16:0/922534524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 msgr2=0x55c9616e9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f27afe00700 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 0x55c9616e9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f27afe00700 1 -- 172.21.15.16:0/922534524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9614b5350 con 0x55c9616e3220 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f27afe00700 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3220 0x55c9616f42a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f27a0013010 tx=0x7f27a00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18e12e83ebe55954 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f279a000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f279a000700 1 -- 172.21.15.16:0/922534524 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0023040 con 0x55c9616e3220 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f27ca913b80 1 -- 172.21.15.16:0/922534524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f27a8007b80 con 0x55c9616e3220 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f279a000700 1 -- 172.21.15.16:0/922534524 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f27a0004830 con 0x55c9616e3220 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f279a000700 1 -- 172.21.15.16:0/922534524 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27a002a5b0 con 0x55c9616e3220 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f27ca913b80 10 client.26150 fetch_fsmap learned FSMap version 41 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.772+0000 7f27ca913b80 1 -- 172.21.15.16:0/922534524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x7f27b010e390 con 0x55c9616e3220 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.773+0000 7f279a000700 1 -- 172.21.15.16:0/922534524 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27a8007b80 con 0x55c9616e3220 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.773+0000 7f279a000700 1 -- 172.21.15.16:0/922534524 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 41) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f27a003a070 con 0x55c9616e3220 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.773+0000 7f27ca913b80 10 client.26150 fetch_fsmap finished waiting for FSMap version 41 2024-04-05T23:37:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.773+0000 7f27ca913b80 10 client.26150 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:37:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.773+0000 7f27ca913b80 20 client.26150 populate_metadata read hostname 'smithi016' 2024-04-05T23:37:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.773+0000 7f27ca913b80 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c961751930 0x55c961753d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.773+0000 7f27ca913b80 4 client.26150 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:37:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.773+0000 7f27ca913b80 1 -- 172.21.15.16:0/922534524 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c961482c90 con 0x55c961751930 2024-04-05T23:37:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.773+0000 7f27af400700 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c961751930 0x55c961753d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.774+0000 7f27af400700 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c961751930 0x55c961753d50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73df4dd6c3634440 server_cookie=6ac889a27c67a640 in_seq=0 out_seq=0 2024-04-05T23:37:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.774+0000 7f279a000700 10 client.26150 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:11.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.814+0000 7f279a000700 1 -- 172.21.15.16:0/922534524 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c961482c90 con 0x55c961751930 2024-04-05T23:37:11.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.814+0000 7f279a000700 10 client.26150 handle_command_reply: tid=0 2024-04-05T23:37:11.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.814+0000 7f27ca913b80 1 client.26150 shutdown 2024-04-05T23:37:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 -- 172.21.15.16:0/922534524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3220 msgr2=0x55c9616f42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3220 0x55c9616f42a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f27a0013010 tx=0x7f27a00047c0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 -- 172.21.15.16:0/922534524 shutdown_connections 2024-04-05T23:37:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c961751930 0x55c961753d50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9616e7ec0 0x55c9616e9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c9616e3220 0x55c9616f42a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 --2- 172.21.15.16:0/922534524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9616e3b40 0x55c9616e8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f279a000700 0 client.26150 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 -- 172.21.15.16:0/922534524 >> 172.21.15.16:0/922534524 conn(0x55c9616571c0 msgr2=0x55c9616f8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 -- 172.21.15.16:0/922534524 shutdown_connections 2024-04-05T23:37:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.815+0000 7f27ca913b80 1 -- 172.21.15.16:0/922534524 wait complete. 2024-04-05T23:37:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.816+0000 7f27ca913b80 20 client.26150 trim_cache size 0 max 16384 2024-04-05T23:37:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:37:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:37:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:37:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:37:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.816+0000 7f27b5800700 1 -- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27940a1310 msgr2=0x7f27940a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27940a1310 0x7f27940a37d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009c60 tx=0x7f27a0009260 comp rx=0 tx=0).stop 2024-04-05T23:37:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 -- 172.21.15.16:0/1542884260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 msgr2=0x7f27b0073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 0x7f27b0073720 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f279c00e970 tx=0x7f279c00ec80 comp rx=0 tx=0).stop 2024-04-05T23:37:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 -- 172.21.15.16:0/1542884260 shutdown_connections 2024-04-05T23:37:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27940a1310 0x7f27940a37d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b0074ea0 0x7f27b0073ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b0100e80 0x7f27b00731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 --2- 172.21.15.16:0/1542884260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b0105fa0 0x7f27b0073720 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 -- 172.21.15.16:0/1542884260 >> 172.21.15.16:0/1542884260 conn(0x7f27b00fc740 msgr2=0x7f27b00feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 -- 172.21.15.16:0/1542884260 shutdown_connections 2024-04-05T23:37:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:11.817+0000 7f27b5800700 1 -- 172.21.15.16:0/1542884260 wait complete. 2024-04-05T23:37:11.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 175351 9.57 MB/sec execute 31 sec latency 36.347 ms 2024-04-05T23:37:12.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:11 smithi016 ceph-mon[38248]: pgmap v113: 97 pgs: 97 active+clean; 82 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 407 op/s 2024-04-05T23:37:12.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:11 smithi130 ceph-mon[79417]: pgmap v113: 97 pgs: 97 active+clean; 82 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 407 op/s 2024-04-05T23:37:12.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:11 smithi027 ceph-mon[78597]: pgmap v113: 97 pgs: 97 active+clean; 82 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 407 op/s 2024-04-05T23:37:12.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 177608 9.70 MB/sec execute 32 sec latency 55.264 ms 2024-04-05T23:37:13.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 179070 9.57 MB/sec execute 33 sec latency 11.962 ms 2024-04-05T23:37:14.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:13 smithi016 ceph-mon[38248]: pgmap v114: 97 pgs: 97 active+clean; 82 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-04-05T23:37:14.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:13 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:14.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:13 smithi130 ceph-mon[79417]: pgmap v114: 97 pgs: 97 active+clean; 82 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-04-05T23:37:14.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:13 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:14.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:13 smithi027 ceph-mon[78597]: pgmap v114: 97 pgs: 97 active+clean; 82 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-04-05T23:37:14.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:13 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:14.991 INFO:tasks.workunit.client.0.smithi016.stdout: 1 181181 9.69 MB/sec execute 34 sec latency 56.398 ms 2024-04-05T23:37:15.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 182598 9.57 MB/sec execute 35 sec latency 14.474 ms 2024-04-05T23:37:16.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:15 smithi016 ceph-mon[38248]: pgmap v115: 97 pgs: 97 active+clean; 82 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-04-05T23:37:16.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:15 smithi130 ceph-mon[79417]: pgmap v115: 97 pgs: 97 active+clean; 82 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-04-05T23:37:16.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:15 smithi027 ceph-mon[78597]: pgmap v115: 97 pgs: 97 active+clean; 82 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-04-05T23:37:16.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184751 9.68 MB/sec execute 36 sec latency 14.100 ms 2024-04-05T23:37:17.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 186302 9.57 MB/sec execute 37 sec latency 63.055 ms 2024-04-05T23:37:18.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:17 smithi016 ceph-mon[38248]: pgmap v116: 97 pgs: 97 active+clean; 81 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 404 op/s 2024-04-05T23:37:18.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:17 smithi130 ceph-mon[79417]: pgmap v116: 97 pgs: 97 active+clean; 81 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 404 op/s 2024-04-05T23:37:18.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:17 smithi027 ceph-mon[78597]: pgmap v116: 97 pgs: 97 active+clean; 81 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 404 op/s 2024-04-05T23:37:18.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 188365 9.68 MB/sec execute 38 sec latency 11.860 ms 2024-04-05T23:37:19.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:18 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:19.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:18 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:19.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:18 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:19.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 190335 9.61 MB/sec execute 39 sec latency 29.913 ms 2024-04-05T23:37:20.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:19 smithi016 ceph-mon[38248]: pgmap v117: 97 pgs: 97 active+clean; 81 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 259 op/s 2024-04-05T23:37:20.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:19 smithi130 ceph-mon[79417]: pgmap v117: 97 pgs: 97 active+clean; 81 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 259 op/s 2024-04-05T23:37:20.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:19 smithi027 ceph-mon[78597]: pgmap v117: 97 pgs: 97 active+clean; 81 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 259 op/s 2024-04-05T23:37:20.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 191906 9.68 MB/sec execute 40 sec latency 14.834 ms 2024-04-05T23:37:21.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 193876 9.61 MB/sec execute 41 sec latency 15.985 ms 2024-04-05T23:37:22.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:21 smithi016 ceph-mon[38248]: pgmap v118: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 376 op/s 2024-04-05T23:37:22.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:21 smithi130 ceph-mon[79417]: pgmap v118: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 376 op/s 2024-04-05T23:37:22.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:21 smithi027 ceph-mon[78597]: pgmap v118: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 376 op/s 2024-04-05T23:37:22.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 195469 9.67 MB/sec execute 42 sec latency 10.818 ms 2024-04-05T23:37:23.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 197392 9.61 MB/sec execute 43 sec latency 14.020 ms 2024-04-05T23:37:24.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:23 smithi016 ceph-mon[38248]: pgmap v119: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 256 op/s 2024-04-05T23:37:24.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:23 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:24.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:23 smithi130 ceph-mon[79417]: pgmap v119: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 256 op/s 2024-04-05T23:37:24.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:23 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:24.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:23 smithi027 ceph-mon[78597]: pgmap v119: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 256 op/s 2024-04-05T23:37:24.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:23 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:37:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.979+0000 7f4db1200700 1 -- 172.21.15.16:0/1414129209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac073bd0 msgr2=0x7f4dac10d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.979+0000 7f4db1200700 1 --2- 172.21.15.16:0/1414129209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac073bd0 0x7f4dac10d110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009990 tx=0x7f4d98009ca0 comp rx=0 tx=0).stop 2024-04-05T23:37:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.979+0000 7f4db1200700 1 -- 172.21.15.16:0/1414129209 shutdown_connections 2024-04-05T23:37:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.979+0000 7f4db1200700 1 --2- 172.21.15.16:0/1414129209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac0732b0 0x7f4dac073690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.979+0000 7f4db1200700 1 --2- 172.21.15.16:0/1414129209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4dac10d650 0x7f4dac10fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.979+0000 7f4db1200700 1 --2- 172.21.15.16:0/1414129209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac073bd0 0x7f4dac10d110 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.979+0000 7f4db1200700 1 -- 172.21.15.16:0/1414129209 >> 172.21.15.16:0/1414129209 conn(0x7f4dac0fc880 msgr2=0x7f4dac0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.979+0000 7f4db1200700 1 -- 172.21.15.16:0/1414129209 shutdown_connections 2024-04-05T23:37:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.979+0000 7f4db1200700 1 -- 172.21.15.16:0/1414129209 wait complete. 2024-04-05T23:37:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.980+0000 7f4db1200700 1 Processor -- start 2024-04-05T23:37:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.980+0000 7f4db1200700 1 -- start start 2024-04-05T23:37:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4db1200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4dac0732b0 0x7f4dac1994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4db1200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac073bd0 0x7f4dac199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4db1200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac10d650 0x7f4dac19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4db1200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac19e400 con 0x7f4dac073bd0 2024-04-05T23:37:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4db1200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac19e570 con 0x7f4dac10d650 2024-04-05T23:37:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4db1200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac19e730 con 0x7f4dac0732b0 2024-04-05T23:37:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4da9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac073bd0 0x7f4dac199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4da9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4dac0732b0 0x7f4dac1994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4da9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac073bd0 0x7f4dac199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36202/0 (socket says 172.21.15.16:36202) 2024-04-05T23:37:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4da9400700 1 -- 172.21.15.16:0/2797344912 learned_addr learned my addr 172.21.15.16:0/2797344912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.981+0000 7f4daa800700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac10d650 0x7f4dac19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da9e00700 1 -- 172.21.15.16:0/2797344912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac10d650 msgr2=0x7f4dac19ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da9e00700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac10d650 0x7f4dac19ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da9e00700 1 -- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac073bd0 msgr2=0x7f4dac199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da9e00700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac073bd0 0x7f4dac199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da9e00700 1 -- 172.21.15.16:0/2797344912 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c009520 con 0x7f4dac0732b0 2024-04-05T23:37:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4daa800700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac10d650 0x7f4dac19ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:37:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da9400700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac073bd0 0x7f4dac199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:37:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da9e00700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4dac0732b0 0x7f4dac1994e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00e9f0 tx=0x7f4d9c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da2a00700 1 -- 172.21.15.16:0/2797344912 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c00cb70 con 0x7f4dac0732b0 2024-04-05T23:37:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4db1200700 1 -- 172.21.15.16:0/2797344912 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d980095f0 con 0x7f4dac0732b0 2024-04-05T23:37:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da2a00700 1 -- 172.21.15.16:0/2797344912 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d9c00ee90 con 0x7f4dac0732b0 2024-04-05T23:37:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4da2a00700 1 -- 172.21.15.16:0/2797344912 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c003cf0 con 0x7f4dac0732b0 2024-04-05T23:37:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.982+0000 7f4db1200700 1 -- 172.21.15.16:0/2797344912 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac1a6070 con 0x7f4dac0732b0 2024-04-05T23:37:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.988+0000 7f4da2a00700 1 -- 172.21.15.16:0/2797344912 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f4d9c00ccd0 con 0x7f4dac0732b0 2024-04-05T23:37:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.993+0000 7f4da2a00700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4d900a1290 0x7f4d900a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:24.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.993+0000 7f4da2a00700 1 -- 172.21.15.16:0/2797344912 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f4d9c014070 con 0x7f4dac0732b0 2024-04-05T23:37:24.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.994+0000 7f4da9400700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4d900a1290 0x7f4d900a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:24.998+0000 7f4da9400700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4d900a1290 0x7f4d900a3750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4dac19aa30 tx=0x7f4d9800b280 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:24.992 INFO:tasks.workunit.client.0.smithi016.stdout: 1 199001 9.67 MB/sec execute 44 sec latency 18.797 ms 2024-04-05T23:37:24.997 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.005+0000 7f4dc4e08b80 1 Processor -- start 2024-04-05T23:37:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.006+0000 7f4dc4e08b80 1 -- start start 2024-04-05T23:37:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.007+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb8b80 0x55c8ffeb8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.007+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb94a0 0x55c8ffeb3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.007+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.007+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffe19b80 con 0x55c8ffeb8b80 2024-04-05T23:37:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.007+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffe0a930 con 0x55c8ffeb94a0 2024-04-05T23:37:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.007+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffe082f0 con 0x55c8ffeb35c0 2024-04-05T23:37:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.008+0000 7f4daa800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.008+0000 7f4daa800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59642/0 (socket says 172.21.15.16:59642) 2024-04-05T23:37:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.008+0000 7f4daa800700 1 -- 172.21.15.16:0/1724947105 learned_addr learned my addr 172.21.15.16:0/1724947105 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.008+0000 7f4daa800700 1 -- 172.21.15.16:0/1724947105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb94a0 msgr2=0x55c8ffeb3030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:37:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.008+0000 7f4daa800700 1 --2- 172.21.15.16:0/1724947105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb94a0 0x55c8ffeb3030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.008+0000 7f4daa800700 1 -- 172.21.15.16:0/1724947105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb8b80 msgr2=0x55c8ffeb8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.008+0000 7f4daa800700 1 --2- 172.21.15.16:0/1724947105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb8b80 0x55c8ffeb8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.008+0000 7f4daa800700 1 -- 172.21.15.16:0/1724947105 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8ffc85350 con 0x55c8ffeb35c0 2024-04-05T23:37:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.010+0000 7f4daa800700 1 --2- 172.21.15.16:0/1724947105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4da4003e40 tx=0x7f4da4009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98354000bf276af6 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.012+0000 7f4d97400700 1 -- 172.21.15.16:0/1724947105 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da40092d0 con 0x55c8ffeb35c0 2024-04-05T23:37:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.012+0000 7f4d97400700 1 -- 172.21.15.16:0/1724947105 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4da400f040 con 0x55c8ffeb35c0 2024-04-05T23:37:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.012+0000 7f4d97400700 1 -- 172.21.15.16:0/1724947105 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da400e870 con 0x55c8ffeb35c0 2024-04-05T23:37:25.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.014+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1724947105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 msgr2=0x55c8ffeb3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:25.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.014+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1724947105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4da4003e40 tx=0x7f4da4009920 comp rx=0 tx=0).stop 2024-04-05T23:37:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.014+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1724947105 shutdown_connections 2024-04-05T23:37:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.014+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1724947105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb8b80 0x55c8ffeb8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.014+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1724947105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3a20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.014+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1724947105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb94a0 0x55c8ffeb3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.014+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1724947105 >> 172.21.15.16:0/1724947105 conn(0x55c8ffe271c0 msgr2=0x55c8ffeadbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.015+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1724947105 shutdown_connections 2024-04-05T23:37:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.015+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1724947105 wait complete. 2024-04-05T23:37:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.016+0000 7f4dc4e08b80 1 -- start start 2024-04-05T23:37:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.016+0000 7f4dc4e08b80 1 Processor -- start 2024-04-05T23:37:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffebf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb94a0 0x55c8ffebd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffe03be0 con 0x55c8ffeb94a0 2024-04-05T23:37:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffd9fd20 con 0x55c8ffeb8b80 2024-04-05T23:37:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffd9f9c0 con 0x55c8ffeb35c0 2024-04-05T23:37:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4dc4e08b80 10 client.0 _ll_register_callbacks cb 0x55c8ffc0b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:37:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4da9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffebf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4da9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffebf560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59646/0 (socket says 172.21.15.16:59646) 2024-04-05T23:37:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4da9e00700 1 -- 172.21.15.16:0/1045527305 learned_addr learned my addr 172.21.15.16:0/1045527305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.017+0000 7f4da9400700 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.018+0000 7f4da9e00700 1 -- 172.21.15.16:0/1045527305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 msgr2=0x55c8ffebc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.018+0000 7f4da9e00700 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.018+0000 7f4da9e00700 1 -- 172.21.15.16:0/1045527305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb94a0 msgr2=0x55c8ffebd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.018+0000 7f4da9e00700 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb94a0 0x55c8ffebd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.018+0000 7f4da9e00700 1 -- 172.21.15.16:0/1045527305 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da4003b20 con 0x55c8ffeb35c0 2024-04-05T23:37:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.018+0000 7f4da9e00700 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffebf560 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009230 tx=0x7f4d9c09a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9132f7c18d8602ea server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.018+0000 7f4d8be00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:37:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.019+0000 7f4d8be00700 1 -- 172.21.15.16:0/1045527305 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c09a4e0 con 0x55c8ffeb35c0 2024-04-05T23:37:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.019+0000 7f4d8be00700 1 -- 172.21.15.16:0/1045527305 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f4d9c0a2500 con 0x55c8ffeb35c0 2024-04-05T23:37:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.019+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1045527305 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8ffea61e0 con 0x55c8ffeb35c0 2024-04-05T23:37:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.020+0000 7f4d8be00700 1 -- 172.21.15.16:0/1045527305 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d9c0b82a0 con 0x55c8ffeb35c0 2024-04-05T23:37:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.020+0000 7f4dc4e08b80 10 client.26194 fetch_fsmap learned FSMap version 44 2024-04-05T23:37:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.020+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1045527305 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x55c8ffc31b00 con 0x55c8ffeb35c0 2024-04-05T23:37:25.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.021+0000 7f4d8be00700 1 -- 172.21.15.16:0/1045527305 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c8ffea61e0 con 0x55c8ffeb35c0 2024-04-05T23:37:25.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.021+0000 7f4d8be00700 1 -- 172.21.15.16:0/1045527305 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f4d9c0a2170 con 0x55c8ffeb35c0 2024-04-05T23:37:25.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.022+0000 7f4dc4e08b80 10 client.26194 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:37:25.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.022+0000 7f4dc4e08b80 10 client.26194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:37:25.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.022+0000 7f4dc4e08b80 20 client.26194 populate_metadata read hostname 'smithi016' 2024-04-05T23:37:25.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.022+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c8ffec3ff0 0x55c8ffec6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.022+0000 7f4dc4e08b80 4 client.26194 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:37:25.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.022+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1045527305 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8ffdfc6a0 con 0x55c8ffec3ff0 2024-04-05T23:37:25.015 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.023+0000 7f4da9400700 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c8ffec3ff0 0x55c8ffec6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.024+0000 7f4da9400700 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c8ffec3ff0 0x55c8ffec6410 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=306f7d928d7cf8d4 server_cookie=961a2afbc791c65b in_seq=0 out_seq=0 2024-04-05T23:37:25.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.024+0000 7f4d8be00700 10 client.26194 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:25.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.025+0000 7f4d8be00700 1 -- 172.21.15.16:0/1045527305 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c8ffdfc6a0 con 0x55c8ffec3ff0 2024-04-05T23:37:25.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.025+0000 7f4d8be00700 10 client.26194 handle_command_reply: tid=0 2024-04-05T23:37:25.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.025+0000 7f4dc4e08b80 1 client.26194 shutdown 2024-04-05T23:37:25.019 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.027+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1045527305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 msgr2=0x55c8ffebf560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:25.019 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.027+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffebf560 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009230 tx=0x7f4d9c09a780 comp rx=0 tx=0).stop 2024-04-05T23:37:25.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.027+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1045527305 shutdown_connections 2024-04-05T23:37:25.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.027+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c8ffec3ff0 0x55c8ffec6410 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.027+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb94a0 0x55c8ffebd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.027+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb35c0 0x55c8ffebf560 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.027+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/1045527305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.028+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1045527305 >> 172.21.15.16:0/1045527305 conn(0x55c8ffe271c0 msgr2=0x55c8ffead280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:25.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.028+0000 7f4d8be00700 0 client.26194 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:25.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.031+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1045527305 shutdown_connections 2024-04-05T23:37:25.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.031+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/1045527305 wait complete. 2024-04-05T23:37:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.032+0000 7f4dc4e08b80 20 client.26194 trim_cache size 0 max 16384 2024-04-05T23:37:25.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.057+0000 7f4dc4e08b80 1 Processor -- start 2024-04-05T23:37:25.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.057+0000 7f4dc4e08b80 1 -- start start 2024-04-05T23:37:25.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.057+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.057+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.057+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb94a0 0x55c8ffeada00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.057+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffc87640 con 0x55c8ffeb35c0 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.057+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffeaaad0 con 0x55c8ffeb8b80 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.057+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffed1770 con 0x55c8ffeb94a0 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38306/0 (socket says 172.21.15.16:38306) 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9400700 1 -- 172.21.15.16:0/2705838572 learned_addr learned my addr 172.21.15.16:0/2705838572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4daa800700 1 --2- 172.21.15.16:0/2705838572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb94a0 0x55c8ffeada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9400700 1 -- 172.21.15.16:0/2705838572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb94a0 msgr2=0x55c8ffeada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9400700 1 --2- 172.21.15.16:0/2705838572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb94a0 0x55c8ffeada00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9400700 1 -- 172.21.15.16:0/2705838572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 msgr2=0x55c8ffeb3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9400700 1 --2- 172.21.15.16:0/2705838572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9400700 1 -- 172.21.15.16:0/2705838572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8ffdc9e40 con 0x55c8ffeb8b80 2024-04-05T23:37:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4daa800700 1 --2- 172.21.15.16:0/2705838572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb94a0 0x55c8ffeada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:37:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4da9400700 1 --2- 172.21.15.16:0/2705838572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4d98006010 tx=0x7f4d98020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a619b1b2a95e6caa server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.058+0000 7f4d97e00700 1 -- 172.21.15.16:0/2705838572 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98005600 con 0x55c8ffeb8b80 2024-04-05T23:37:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.059+0000 7f4d97e00700 1 -- 172.21.15.16:0/2705838572 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d9802c040 con 0x55c8ffeb8b80 2024-04-05T23:37:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.059+0000 7f4d97e00700 1 -- 172.21.15.16:0/2705838572 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98004d20 con 0x55c8ffeb8b80 2024-04-05T23:37:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.059+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/2705838572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 msgr2=0x55c8ffebc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.059+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/2705838572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4d98006010 tx=0x7f4d98020000 comp rx=0 tx=0).stop 2024-04-05T23:37:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.059+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/2705838572 shutdown_connections 2024-04-05T23:37:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.059+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/2705838572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeb3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.059+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/2705838572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb94a0 0x55c8ffeada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.059+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/2705838572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb8b80 0x55c8ffebc230 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.059+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/2705838572 >> 172.21.15.16:0/2705838572 conn(0x55c8ffe271c0 msgr2=0x55c8ffeb39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.060+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/2705838572 shutdown_connections 2024-04-05T23:37:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.060+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/2705838572 wait complete. 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.060+0000 7f4dc4e08b80 1 -- start start 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.060+0000 7f4dc4e08b80 1 Processor -- start 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeca960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb8b80 0x55c8ffecaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4dc4e08b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb94a0 0x55c8ffec8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffec9640 con 0x55c8ffeb35c0 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffec97b0 con 0x55c8ffeb94a0 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4dc4e08b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8ffec9970 con 0x55c8ffeb8b80 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4dc4e08b80 10 client.0 _ll_register_callbacks cb 0x7f4d9c099bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4da9400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb8b80 0x55c8ffecaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4da9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4da9400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb8b80 0x55c8ffecaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59668/0 (socket says 172.21.15.16:59668) 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4da9400700 1 -- 172.21.15.16:0/3913132817 learned_addr learned my addr 172.21.15.16:0/3913132817 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.061+0000 7f4daa800700 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb94a0 0x55c8ffec8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4da9e00700 1 -- 172.21.15.16:0/3913132817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb8b80 msgr2=0x55c8ffecaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4da9e00700 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb8b80 0x55c8ffecaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4da9e00700 1 -- 172.21.15.16:0/3913132817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb94a0 msgr2=0x55c8ffec8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4da9e00700 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb94a0 0x55c8ffec8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4da9e00700 1 -- 172.21.15.16:0/3913132817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac110790 con 0x55c8ffeb35c0 2024-04-05T23:37:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4da9400700 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb8b80 0x55c8ffecaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:37:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4da9e00700 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeca960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c096af0 tx=0x7f4d9c0b88a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70fd39a9bdb66549 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4d8be00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:37:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4d8be00700 1 -- 172.21.15.16:0/3913132817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c099370 con 0x55c8ffeb35c0 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/3913132817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4da400f040 con 0x55c8ffeb35c0 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.062+0000 7f4d8be00700 1 -- 172.21.15.16:0/3913132817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f4d9c0a2500 con 0x55c8ffeb35c0 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4d8be00700 1 -- 172.21.15.16:0/3913132817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d9c09a2e0 con 0x55c8ffeb35c0 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4dc4e08b80 10 client.16119 fetch_fsmap learned FSMap version 44 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/3913132817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x55c8ffc89c70 con 0x55c8ffeb35c0 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4d8be00700 1 -- 172.21.15.16:0/3913132817 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d9c0a6080 con 0x55c8ffeb35c0 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4d8be00700 1 -- 172.21.15.16:0/3913132817 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f4d9c083020 con 0x55c8ffeb35c0 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4dc4e08b80 10 client.16119 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4dc4e08b80 10 client.16119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4dc4e08b80 20 client.16119 populate_metadata read hostname 'smithi016' 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c8fff32810 0x55c8fff34c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4dc4e08b80 4 client.16119 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:37:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/3913132817 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c8ffc52c90 con 0x55c8fff32810 2024-04-05T23:37:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.063+0000 7f4da9400700 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c8fff32810 0x55c8fff34c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.064+0000 7f4d8be00700 10 client.16119 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.064+0000 7f4da9400700 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c8fff32810 0x55c8fff34c30 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a62c0798909ff339 server_cookie=d6e62e46e83b2272 in_seq=0 out_seq=0 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4d8be00700 1 -- 172.21.15.16:0/3913132817 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c8ffc52c90 con 0x55c8fff32810 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4d8be00700 10 client.16119 handle_command_reply: tid=0 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 client.16119 shutdown 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/3913132817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 msgr2=0x55c8ffeca960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeca960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c096af0 tx=0x7f4d9c0b88a0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/3913132817 shutdown_connections 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c8fff32810 0x55c8fff34c30 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ffeb35c0 0x55c8ffeca960 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c8ffeb94a0 0x55c8ffec8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 --2- 172.21.15.16:0/3913132817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8ffeb8b80 0x55c8ffecaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4d8be00700 0 client.16119 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/3913132817 >> 172.21.15.16:0/3913132817 conn(0x55c8ffe271c0 msgr2=0x55c8ffeb7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/3913132817 shutdown_connections 2024-04-05T23:37:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 1 -- 172.21.15.16:0/3913132817 wait complete. 2024-04-05T23:37:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.119+0000 7f4dc4e08b80 20 client.16119 trim_cache size 0 max 16384 2024-04-05T23:37:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:37:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:37:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:37:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:37:25.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 -- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4d900a1290 msgr2=0x7f4d900a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:25.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4d900a1290 0x7f4d900a3750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4dac19aa30 tx=0x7f4d9800b280 comp rx=0 tx=0).stop 2024-04-05T23:37:25.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 -- 172.21.15.16:0/2797344912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4dac0732b0 msgr2=0x7f4dac1994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:25.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4dac0732b0 0x7f4dac1994e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00e9f0 tx=0x7f4d9c00ed00 comp rx=0 tx=0).stop 2024-04-05T23:37:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 -- 172.21.15.16:0/2797344912 shutdown_connections 2024-04-05T23:37:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f4d900a1290 0x7f4d900a3750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac10d650 0x7f4dac19ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4dac0732b0 0x7f4dac1994e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 --2- 172.21.15.16:0/2797344912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac073bd0 0x7f4dac199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 -- 172.21.15.16:0/2797344912 >> 172.21.15.16:0/2797344912 conn(0x7f4dac0fc880 msgr2=0x7f4dac0fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 -- 172.21.15.16:0/2797344912 shutdown_connections 2024-04-05T23:37:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:25.120+0000 7f4db1200700 1 -- 172.21.15.16:0/2797344912 wait complete. 2024-04-05T23:37:25.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 200228 9.57 MB/sec execute 45 sec latency 56.159 ms 2024-04-05T23:37:26.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:25 smithi016 ceph-mon[38248]: pgmap v120: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 256 op/s 2024-04-05T23:37:26.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:25 smithi130 ceph-mon[79417]: pgmap v120: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 256 op/s 2024-04-05T23:37:26.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:25 smithi027 ceph-mon[78597]: pgmap v120: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 256 op/s 2024-04-05T23:37:26.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 202447 9.66 MB/sec execute 46 sec latency 13.351 ms 2024-04-05T23:37:27.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 203962 9.57 MB/sec execute 47 sec latency 11.715 ms 2024-04-05T23:37:28.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:27 smithi016 ceph-mon[38248]: pgmap v121: 97 pgs: 97 active+clean; 88 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 401 op/s 2024-04-05T23:37:28.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:27 smithi130 ceph-mon[79417]: pgmap v121: 97 pgs: 97 active+clean; 88 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 401 op/s 2024-04-05T23:37:28.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:27 smithi027 ceph-mon[78597]: pgmap v121: 97 pgs: 97 active+clean; 88 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 401 op/s 2024-04-05T23:37:28.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 206053 9.65 MB/sec execute 48 sec latency 10.001 ms 2024-04-05T23:37:29.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 208118 9.60 MB/sec execute 49 sec latency 15.044 ms 2024-04-05T23:37:30.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:29 smithi016 ceph-mon[38248]: pgmap v122: 97 pgs: 97 active+clean; 88 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-05T23:37:30.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:29 smithi130 ceph-mon[79417]: pgmap v122: 97 pgs: 97 active+clean; 88 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-05T23:37:30.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:29 smithi027 ceph-mon[78597]: pgmap v122: 97 pgs: 97 active+clean; 88 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-05T23:37:30.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 209650 9.65 MB/sec execute 50 sec latency 33.229 ms 2024-04-05T23:37:31.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 211656 9.60 MB/sec execute 51 sec latency 12.517 ms 2024-04-05T23:37:32.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:31 smithi016 ceph-mon[38248]: pgmap v123: 97 pgs: 97 active+clean; 93 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 382 op/s 2024-04-05T23:37:32.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:31 smithi130 ceph-mon[79417]: pgmap v123: 97 pgs: 97 active+clean; 93 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 382 op/s 2024-04-05T23:37:32.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:31 smithi027 ceph-mon[78597]: pgmap v123: 97 pgs: 97 active+clean; 93 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 382 op/s 2024-04-05T23:37:32.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213210 9.65 MB/sec execute 52 sec latency 15.720 ms 2024-04-05T23:37:33.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 215282 9.60 MB/sec execute 53 sec latency 11.195 ms 2024-04-05T23:37:34.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:33 smithi016 ceph-mon[38248]: pgmap v124: 97 pgs: 97 active+clean; 93 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-05T23:37:34.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:33 smithi130 ceph-mon[79417]: pgmap v124: 97 pgs: 97 active+clean; 93 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-05T23:37:34.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:33 smithi027 ceph-mon[78597]: pgmap v124: 97 pgs: 97 active+clean; 93 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-05T23:37:34.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 216846 9.65 MB/sec execute 54 sec latency 10.966 ms 2024-04-05T23:37:35.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 218852 9.60 MB/sec execute 55 sec latency 47.400 ms 2024-04-05T23:37:36.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:35 smithi016 ceph-mon[38248]: pgmap v125: 97 pgs: 97 active+clean; 93 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-05T23:37:36.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:35 smithi130 ceph-mon[79417]: pgmap v125: 97 pgs: 97 active+clean; 93 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-05T23:37:36.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:35 smithi027 ceph-mon[78597]: pgmap v125: 97 pgs: 97 active+clean; 93 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-05T23:37:36.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220410 9.64 MB/sec execute 56 sec latency 10.838 ms 2024-04-05T23:37:37.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 222414 9.60 MB/sec execute 57 sec latency 11.462 ms 2024-04-05T23:37:38.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:37 smithi016 ceph-mon[38248]: pgmap v126: 97 pgs: 97 active+clean; 100 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 380 op/s 2024-04-05T23:37:38.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:37 smithi130 ceph-mon[79417]: pgmap v126: 97 pgs: 97 active+clean; 100 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 380 op/s 2024-04-05T23:37:38.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:37 smithi027 ceph-mon[78597]: pgmap v126: 97 pgs: 97 active+clean; 100 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 380 op/s 2024-04-05T23:37:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.276+0000 7ff713200700 1 -- 172.21.15.16:0/3228467983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c1019f0 msgr2=0x7ff70c105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.276+0000 7ff713200700 1 --2- 172.21.15.16:0/3228467983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c1019f0 0x7ff70c105a30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8009990 tx=0x7ff6f8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.276+0000 7ff713200700 1 -- 172.21.15.16:0/3228467983 shutdown_connections 2024-04-05T23:37:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.276+0000 7ff713200700 1 --2- 172.21.15.16:0/3228467983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff70c101040 0x7ff70c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.276+0000 7ff713200700 1 --2- 172.21.15.16:0/3228467983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff70c106160 0x7ff70c10dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.276+0000 7ff713200700 1 --2- 172.21.15.16:0/3228467983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c1019f0 0x7ff70c105a30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.276+0000 7ff713200700 1 -- 172.21.15.16:0/3228467983 >> 172.21.15.16:0/3228467983 conn(0x7ff70c0fc8e0 msgr2=0x7ff70c0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.277+0000 7ff713200700 1 -- 172.21.15.16:0/3228467983 shutdown_connections 2024-04-05T23:37:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.277+0000 7ff713200700 1 -- 172.21.15.16:0/3228467983 wait complete. 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.278+0000 7ff713200700 1 Processor -- start 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.278+0000 7ff713200700 1 -- start start 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.278+0000 7ff713200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff70c101040 0x7ff70c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.278+0000 7ff713200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c106160 0x7ff70c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.278+0000 7ff713200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff70c073c60 0x7ff70c1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.278+0000 7ff713200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c1a5ad0 con 0x7ff70c101040 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.278+0000 7ff713200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c1a5c40 con 0x7ff70c106160 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.278+0000 7ff713200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c1a5e00 con 0x7ff70c073c60 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c106160 0x7ff70c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff70c101040 0x7ff70c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff711200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff70c073c60 0x7ff70c1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c106160 0x7ff70c073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42418/0 (socket says 172.21.15.16:42418) 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70b400700 1 -- 172.21.15.16:0/337121065 learned_addr learned my addr 172.21.15.16:0/337121065 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70b400700 1 -- 172.21.15.16:0/337121065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff70c073c60 msgr2=0x7ff70c1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70b400700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff70c073c60 0x7ff70c1a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70b400700 1 -- 172.21.15.16:0/337121065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff70c101040 msgr2=0x7ff70c0731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70b400700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff70c101040 0x7ff70c0731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70b400700 1 -- 172.21.15.16:0/337121065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f80095f0 con 0x7ff70c106160 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff70b400700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c106160 0x7ff70c073720 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000c00 tx=0x7ff6f8009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff708c00700 1 -- 172.21.15.16:0/337121065 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f801d070 con 0x7ff70c106160 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff708c00700 1 -- 172.21.15.16:0/337121065 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6f8005030 con 0x7ff70c106160 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff708c00700 1 -- 172.21.15.16:0/337121065 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8004a40 con 0x7ff70c106160 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.279+0000 7ff713200700 1 -- 172.21.15.16:0/337121065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70c1a6080 con 0x7ff70c106160 2024-04-05T23:37:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.280+0000 7ff713200700 1 -- 172.21.15.16:0/337121065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70c1a64a0 con 0x7ff70c106160 2024-04-05T23:37:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.283+0000 7ff708c00700 1 -- 172.21.15.16:0/337121065 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7ff6f8005460 con 0x7ff70c106160 2024-04-05T23:37:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.284+0000 7ff726c9fb80 1 Processor -- start 2024-04-05T23:37:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff708c00700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6f00a1310 0x7ff6f00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff726c9fb80 1 -- start start 2024-04-05T23:37:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff708c00700 1 -- 172.21.15.16:0/337121065 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff6f80cdce0 con 0x7ff70c106160 2024-04-05T23:37:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e955f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 0x55c12e950030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff711200700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6f00a1310 0x7ff6f00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e950a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e8b6b80 con 0x55c12e9505c0 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e8a7930 con 0x55c12e955b80 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e8a52f0 con 0x55c12e9564a0 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff70be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e955f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff70be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e955f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42434/0 (socket says 172.21.15.16:42434) 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff70be00700 1 -- 172.21.15.16:0/1562816733 learned_addr learned my addr 172.21.15.16:0/1562816733 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff70b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 0x55c12e950030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.285+0000 7ff711200700 1 --2- 172.21.15.16:0/1562816733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e950a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff70be00700 1 -- 172.21.15.16:0/1562816733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 msgr2=0x55c12e950030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff70be00700 1 --2- 172.21.15.16:0/1562816733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 0x55c12e950030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff70be00700 1 -- 172.21.15.16:0/1562816733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 msgr2=0x55c12e950a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff70be00700 1 --2- 172.21.15.16:0/1562816733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e950a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff70be00700 1 -- 172.21.15.16:0/1562816733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c12e722350 con 0x55c12e955b80 2024-04-05T23:37:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff70be00700 1 --2- 172.21.15.16:0/1562816733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e955f60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc007cf0 tx=0x7ff6fc0103d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d6052f7484d01d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff711200700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6f00a1310 0x7ff6f00a37d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff70400f870 tx=0x7ff70400fc30 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff701600700 1 -- 172.21.15.16:0/1562816733 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc010d10 con 0x55c12e955b80 2024-04-05T23:37:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff701600700 1 -- 172.21.15.16:0/1562816733 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6fc015070 con 0x55c12e955b80 2024-04-05T23:37:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.286+0000 7ff701600700 1 -- 172.21.15.16:0/1562816733 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc019630 con 0x55c12e955b80 2024-04-05T23:37:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.287+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1562816733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 msgr2=0x55c12e955f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.287+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/1562816733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e955f60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc007cf0 tx=0x7ff6fc0103d0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.287+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1562816733 shutdown_connections 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.287+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/1562816733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e950a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.287+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/1562816733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e955f60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.287+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/1562816733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 0x55c12e950030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.287+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1562816733 >> 172.21.15.16:0/1562816733 conn(0x55c12e8c41c0 msgr2=0x55c12e94abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.287+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1562816733 shutdown_connections 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.287+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1562816733 wait complete. 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.288+0000 7ff726c9fb80 1 -- start start 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.288+0000 7ff726c9fb80 1 Processor -- start 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.288+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9505c0 0x55c12e95c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e959840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9564a0 0x55c12e959f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e8a0be0 con 0x55c12e9564a0 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e83cd20 con 0x55c12e9505c0 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e83c9c0 con 0x55c12e955b80 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff726c9fb80 10 client.0 _ll_register_callbacks cb 0x55c12e6a8340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff70be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9505c0 0x55c12e95c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff70be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9505c0 0x55c12e95c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42442/0 (socket says 172.21.15.16:42442) 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff70be00700 1 -- 172.21.15.16:0/483867269 learned_addr learned my addr 172.21.15.16:0/483867269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff70b400700 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e959840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff711200700 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9564a0 0x55c12e959f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff70be00700 1 -- 172.21.15.16:0/483867269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 msgr2=0x55c12e959840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff70be00700 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e959840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff70be00700 1 -- 172.21.15.16:0/483867269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9564a0 msgr2=0x55c12e959f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff70be00700 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9564a0 0x55c12e959f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.289+0000 7ff70be00700 1 -- 172.21.15.16:0/483867269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6fc007920 con 0x55c12e9505c0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.290+0000 7ff70be00700 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9505c0 0x55c12e95c4e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc000c00 tx=0x7ff6fc010b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5226a69d86701ce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.290+0000 7ff6f6000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.290+0000 7ff6f6000700 1 -- 172.21.15.16:0/483867269 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc011440 con 0x55c12e9505c0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.290+0000 7ff6f6000700 1 -- 172.21.15.16:0/483867269 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff6fc0115a0 con 0x55c12e9505c0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.290+0000 7ff726c9fb80 1 -- 172.21.15.16:0/483867269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c12e9431e0 con 0x55c12e9505c0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.290+0000 7ff6f6000700 1 -- 172.21.15.16:0/483867269 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6fc011760 con 0x55c12e9505c0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.290+0000 7ff726c9fb80 10 client.26177 fetch_fsmap learned FSMap version 44 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.290+0000 7ff726c9fb80 1 -- 172.21.15.16:0/483867269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x55c12e6ceb00 con 0x55c12e9505c0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.290+0000 7ff6f6000700 1 -- 172.21.15.16:0/483867269 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c12e9431e0 con 0x55c12e9505c0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff6f6000700 1 -- 172.21.15.16:0/483867269 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff6fc004990 con 0x55c12e9505c0 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff726c9fb80 10 client.26177 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff726c9fb80 10 client.26177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff726c9fb80 20 client.26177 populate_metadata read hostname 'smithi016' 2024-04-05T23:37:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c12e969e50 0x55c12e96c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff726c9fb80 4 client.26177 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:37:38.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff726c9fb80 1 -- 172.21.15.16:0/483867269 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c12e8996a0 con 0x55c12e969e50 2024-04-05T23:37:38.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff70b400700 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c12e969e50 0x55c12e96c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff70b400700 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c12e969e50 0x55c12e96c240 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a135aa100f8c0b2d server_cookie=aed506d98c19503 in_seq=0 out_seq=0 2024-04-05T23:37:38.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.291+0000 7ff6f6000700 10 client.26177 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff6f6000700 1 -- 172.21.15.16:0/483867269 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c12e8996a0 con 0x55c12e969e50 2024-04-05T23:37:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff6f6000700 10 client.26177 handle_command_reply: tid=0 2024-04-05T23:37:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 client.26177 shutdown 2024-04-05T23:37:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 -- 172.21.15.16:0/483867269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9505c0 msgr2=0x55c12e95c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9505c0 0x55c12e95c4e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc000c00 tx=0x7ff6fc010b10 comp rx=0 tx=0).stop 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 -- 172.21.15.16:0/483867269 shutdown_connections 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c12e969e50 0x55c12e96c240 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9564a0 0x55c12e959f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9505c0 0x55c12e95c4e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/483867269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e959840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 -- 172.21.15.16:0/483867269 >> 172.21.15.16:0/483867269 conn(0x55c12e8c41c0 msgr2=0x55c12e94a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff6f6000700 0 client.26177 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 -- 172.21.15.16:0/483867269 shutdown_connections 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.292+0000 7ff726c9fb80 1 -- 172.21.15.16:0/483867269 wait complete. 2024-04-05T23:37:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.293+0000 7ff726c9fb80 20 client.26177 trim_cache size 0 max 16384 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.309+0000 7ff726c9fb80 1 Processor -- start 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.309+0000 7ff726c9fb80 1 -- start start 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.309+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e94aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.309+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e9539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.309+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 0x55c12e953f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.309+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e724640 con 0x55c12e9505c0 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.309+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e94c3d0 con 0x55c12e955b80 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.309+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e94bc10 con 0x55c12e9564a0 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e9539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e94aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff711200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 0x55c12e953f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e9539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42446/0 (socket says 172.21.15.16:42446) 2024-04-05T23:37:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70b400700 1 -- 172.21.15.16:0/1915400622 learned_addr learned my addr 172.21.15.16:0/1915400622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70b400700 1 -- 172.21.15.16:0/1915400622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 msgr2=0x55c12e953f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70b400700 1 --2- 172.21.15.16:0/1915400622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 0x55c12e953f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70b400700 1 -- 172.21.15.16:0/1915400622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 msgr2=0x55c12e94aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70b400700 1 --2- 172.21.15.16:0/1915400622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e94aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70b400700 1 -- 172.21.15.16:0/1915400622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c12e866e40 con 0x55c12e955b80 2024-04-05T23:37:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.310+0000 7ff70b400700 1 --2- 172.21.15.16:0/1915400622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e9539e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff6f809ab50 tx=0x7ff6f8023040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f48646dc65171e32 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff702000700 1 -- 172.21.15.16:0/1915400622 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f809ec80 con 0x55c12e955b80 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff702000700 1 -- 172.21.15.16:0/1915400622 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6f80a3240 con 0x55c12e955b80 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff702000700 1 -- 172.21.15.16:0/1915400622 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f809a340 con 0x55c12e955b80 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1915400622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 msgr2=0x55c12e9539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/1915400622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e9539e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff6f809ab50 tx=0x7ff6f8023040 comp rx=0 tx=0).stop 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1915400622 shutdown_connections 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/1915400622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e94aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/1915400622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e9564a0 0x55c12e953f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/1915400622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e955b80 0x55c12e9539e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1915400622 >> 172.21.15.16:0/1915400622 conn(0x55c12e8c41c0 msgr2=0x55c12e95c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1915400622 shutdown_connections 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.311+0000 7ff726c9fb80 1 -- 172.21.15.16:0/1915400622 wait complete. 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.312+0000 7ff726c9fb80 1 -- start start 2024-04-05T23:37:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.312+0000 7ff726c9fb80 1 Processor -- start 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e952760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e952ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff726c9fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9564a0 0x55c12e9516e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e9531e0 con 0x55c12e9505c0 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e953350 con 0x55c12e9564a0 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff726c9fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e953510 con 0x55c12e955b80 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff726c9fb80 10 client.0 _ll_register_callbacks cb 0x7ff6f8097190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff70b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e952ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff70b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e952ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:46602/0 (socket says 172.21.15.16:46602) 2024-04-05T23:37:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff70b400700 1 -- 172.21.15.16:0/991599013 learned_addr learned my addr 172.21.15.16:0/991599013 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff70be00700 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e952760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.313+0000 7ff711200700 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9564a0 0x55c12e9516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff70b400700 1 -- 172.21.15.16:0/991599013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9564a0 msgr2=0x55c12e9516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff70b400700 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9564a0 0x55c12e9516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff70b400700 1 -- 172.21.15.16:0/991599013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 msgr2=0x55c12e952760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff70b400700 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e952760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff70b400700 1 -- 172.21.15.16:0/991599013 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70c1a64a0 con 0x55c12e955b80 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff70b400700 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e952ca0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80a2d10 tx=0x7ff6f80cb420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=381a4c4e2ca6f70a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff6f6000700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff6f6000700 1 -- 172.21.15.16:0/991599013 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f809b0b0 con 0x55c12e955b80 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff726c9fb80 1 -- 172.21.15.16:0/991599013 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6fc015070 con 0x55c12e955b80 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff6f6000700 1 -- 172.21.15.16:0/991599013 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff6f809b210 con 0x55c12e955b80 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff6f6000700 1 -- 172.21.15.16:0/991599013 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6f809a340 con 0x55c12e955b80 2024-04-05T23:37:38.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff726c9fb80 10 client.26218 fetch_fsmap learned FSMap version 44 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.314+0000 7ff726c9fb80 1 -- 172.21.15.16:0/991599013 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x55c12e726c70 con 0x55c12e955b80 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff6f6000700 1 -- 172.21.15.16:0/991599013 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6fc015070 con 0x55c12e955b80 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff6f6000700 1 -- 172.21.15.16:0/991599013 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff6f809fba0 con 0x55c12e955b80 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff726c9fb80 10 client.26218 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff726c9fb80 10 client.26218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff726c9fb80 20 client.26218 populate_metadata read hostname 'smithi016' 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c12e9be300 0x55c12e9c0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff726c9fb80 4 client.26218 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff726c9fb80 1 -- 172.21.15.16:0/991599013 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c12e6efc90 con 0x55c12e9be300 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff70be00700 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c12e9be300 0x55c12e9c0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff70be00700 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c12e9be300 0x55c12e9c0720 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c0f942f5556ebbb server_cookie=fc16bf93829fafda in_seq=0 out_seq=0 2024-04-05T23:37:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.315+0000 7ff6f6000700 10 client.26218 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff6f6000700 1 -- 172.21.15.16:0/991599013 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c12e6efc90 con 0x55c12e9be300 2024-04-05T23:37:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff6f6000700 10 client.26218 handle_command_reply: tid=0 2024-04-05T23:37:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 client.26218 shutdown 2024-04-05T23:37:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 -- 172.21.15.16:0/991599013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 msgr2=0x55c12e952ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e952ca0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80a2d10 tx=0x7ff6f80cb420 comp rx=0 tx=0).stop 2024-04-05T23:37:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 -- 172.21.15.16:0/991599013 shutdown_connections 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c12e9be300 0x55c12e9c0720 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c12e9505c0 0x55c12e952760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c12e9564a0 0x55c12e9516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 --2- 172.21.15.16:0/991599013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c12e955b80 0x55c12e952ca0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff6f6000700 0 client.26218 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 -- 172.21.15.16:0/991599013 >> 172.21.15.16:0/991599013 conn(0x55c12e8c41c0 msgr2=0x55c12e958110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 -- 172.21.15.16:0/991599013 shutdown_connections 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.363+0000 7ff726c9fb80 1 -- 172.21.15.16:0/991599013 wait complete. 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff726c9fb80 20 client.26218 trim_cache size 0 max 16384 2024-04-05T23:37:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:37:38.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:37:38.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:37:38.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:37:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 -- 172.21.15.16:0/337121065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6f00a1310 msgr2=0x7ff6f00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6f00a1310 0x7ff6f00a37d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff70400f870 tx=0x7ff70400fc30 comp rx=0 tx=0).stop 2024-04-05T23:37:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 -- 172.21.15.16:0/337121065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c106160 msgr2=0x7ff70c073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c106160 0x7ff70c073720 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000c00 tx=0x7ff6f8009580 comp rx=0 tx=0).stop 2024-04-05T23:37:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 -- 172.21.15.16:0/337121065 shutdown_connections 2024-04-05T23:37:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6f00a1310 0x7ff6f00a37d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff70c101040 0x7ff70c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff70c073c60 0x7ff70c1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 --2- 172.21.15.16:0/337121065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff70c106160 0x7ff70c073720 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:38.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 -- 172.21.15.16:0/337121065 >> 172.21.15.16:0/337121065 conn(0x7ff70c0fc8e0 msgr2=0x7ff70c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:38.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 -- 172.21.15.16:0/337121065 shutdown_connections 2024-04-05T23:37:38.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:38.364+0000 7ff713200700 1 -- 172.21.15.16:0/337121065 wait complete. 2024-04-05T23:37:38.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 223931 9.64 MB/sec execute 58 sec latency 22.401 ms 2024-04-05T23:37:39.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 225969 9.60 MB/sec execute 59 sec latency 19.308 ms 2024-04-05T23:37:40.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:40 smithi016 ceph-mon[38248]: pgmap v127: 97 pgs: 97 active+clean; 100 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 235 op/s 2024-04-05T23:37:40.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:40 smithi130 ceph-mon[79417]: pgmap v127: 97 pgs: 97 active+clean; 100 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 235 op/s 2024-04-05T23:37:40.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:40 smithi027 ceph-mon[78597]: pgmap v127: 97 pgs: 97 active+clean; 100 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 235 op/s 2024-04-05T23:37:40.993 INFO:tasks.workunit.client.0.smithi016.stdout: 1 227462 9.64 MB/sec execute 60 sec latency 31.306 ms 2024-04-05T23:37:41.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 229473 9.60 MB/sec execute 61 sec latency 16.661 ms 2024-04-05T23:37:42.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:42 smithi016 ceph-mon[38248]: pgmap v128: 97 pgs: 97 active+clean; 95 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 381 op/s 2024-04-05T23:37:42.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:42 smithi130 ceph-mon[79417]: pgmap v128: 97 pgs: 97 active+clean; 95 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 381 op/s 2024-04-05T23:37:42.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:42 smithi027 ceph-mon[78597]: pgmap v128: 97 pgs: 97 active+clean; 95 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 381 op/s 2024-04-05T23:37:42.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 231007 9.64 MB/sec execute 62 sec latency 10.665 ms 2024-04-05T23:37:43.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 233039 9.60 MB/sec execute 63 sec latency 12.995 ms 2024-04-05T23:37:44.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:44 smithi016 ceph-mon[38248]: pgmap v129: 97 pgs: 97 active+clean; 95 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 261 op/s 2024-04-05T23:37:44.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:37:44.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:44 smithi130 ceph-mon[79417]: pgmap v129: 97 pgs: 97 active+clean; 95 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 261 op/s 2024-04-05T23:37:44.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:37:44.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:44 smithi027 ceph-mon[78597]: pgmap v129: 97 pgs: 97 active+clean; 95 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 261 op/s 2024-04-05T23:37:44.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:37:44.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234549 9.63 MB/sec execute 64 sec latency 10.980 ms 2024-04-05T23:37:45.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:45 smithi027 ceph-mon[78597]: pgmap v130: 97 pgs: 97 active+clean; 95 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 261 op/s 2024-04-05T23:37:45.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:45 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:45.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:45 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:45.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:45 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:45.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:45 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:45.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:45 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:37:45.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:45 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:37:45.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:45 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:45.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 236238 9.57 MB/sec execute 65 sec latency 49.419 ms 2024-04-05T23:37:46.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:45 smithi016 ceph-mon[38248]: pgmap v130: 97 pgs: 97 active+clean; 95 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 261 op/s 2024-04-05T23:37:46.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:45 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:45 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:45 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:45 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:45 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:37:46.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:45 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:37:46.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:45 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:45 smithi130 ceph-mon[79417]: pgmap v130: 97 pgs: 97 active+clean; 95 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 261 op/s 2024-04-05T23:37:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:45 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:45 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:45 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:45 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:45 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:37:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:45 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:37:46.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:45 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:37:46.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 238034 9.63 MB/sec execute 66 sec latency 13.666 ms 2024-04-05T23:37:47.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:47 smithi027 ceph-mon[78597]: pgmap v131: 97 pgs: 97 active+clean; 104 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 375 op/s 2024-04-05T23:37:47.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 239896 9.57 MB/sec execute 67 sec latency 14.471 ms 2024-04-05T23:37:48.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:47 smithi016 ceph-mon[38248]: pgmap v131: 97 pgs: 97 active+clean; 104 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 375 op/s 2024-04-05T23:37:48.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:47 smithi130 ceph-mon[79417]: pgmap v131: 97 pgs: 97 active+clean; 104 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 375 op/s 2024-04-05T23:37:48.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 241565 9.63 MB/sec execute 68 sec latency 13.446 ms 2024-04-05T23:37:49.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:49 smithi027 ceph-mon[78597]: pgmap v132: 97 pgs: 97 active+clean; 104 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 259 op/s 2024-04-05T23:37:49.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 243381 9.57 MB/sec execute 69 sec latency 13.799 ms 2024-04-05T23:37:50.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:49 smithi016 ceph-mon[38248]: pgmap v132: 97 pgs: 97 active+clean; 104 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 259 op/s 2024-04-05T23:37:50.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:49 smithi130 ceph-mon[79417]: pgmap v132: 97 pgs: 97 active+clean; 104 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 259 op/s 2024-04-05T23:37:50.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 245063 9.62 MB/sec execute 70 sec latency 25.843 ms 2024-04-05T23:37:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.526+0000 7ff6da400700 1 -- 172.21.15.16:0/760110803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6d4073280 msgr2=0x7ff6d4073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.526+0000 7ff6da400700 1 --2- 172.21.15.16:0/760110803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6d4073280 0x7ff6d4073660 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009990 tx=0x7ff6c4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.526+0000 7ff6da400700 1 -- 172.21.15.16:0/760110803 shutdown_connections 2024-04-05T23:37:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.526+0000 7ff6da400700 1 --2- 172.21.15.16:0/760110803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d4111de0 0x7ff6d41141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.526+0000 7ff6da400700 1 --2- 172.21.15.16:0/760110803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6d4073280 0x7ff6d4073660 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.526+0000 7ff6da400700 1 --2- 172.21.15.16:0/760110803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d4073ba0 0x7ff6d41118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.526+0000 7ff6da400700 1 -- 172.21.15.16:0/760110803 >> 172.21.15.16:0/760110803 conn(0x7ff6d40fc8e0 msgr2=0x7ff6d40fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.527+0000 7ff6da400700 1 -- 172.21.15.16:0/760110803 shutdown_connections 2024-04-05T23:37:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.527+0000 7ff6da400700 1 -- 172.21.15.16:0/760110803 wait complete. 2024-04-05T23:37:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.528+0000 7ff6da400700 1 Processor -- start 2024-04-05T23:37:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.528+0000 7ff6da400700 1 -- start start 2024-04-05T23:37:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6da400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d4073280 0x7ff6d41a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6da400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6d4073ba0 0x7ff6d41a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6da400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d4111de0 0x7ff6d419d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6da400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d419dda0 con 0x7ff6d4111de0 2024-04-05T23:37:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6da400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d419df10 con 0x7ff6d4073280 2024-04-05T23:37:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6da400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d419e0d0 con 0x7ff6d4073ba0 2024-04-05T23:37:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6d3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d4073280 0x7ff6d41a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6d2a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6d4073ba0 0x7ff6d41a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6d3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d4073280 0x7ff6d41a3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42154/0 (socket says 172.21.15.16:42154) 2024-04-05T23:37:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6d3400700 1 -- 172.21.15.16:0/964510310 learned_addr learned my addr 172.21.15.16:0/964510310 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.529+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d4111de0 0x7ff6d419d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6d3400700 1 -- 172.21.15.16:0/964510310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6d4073ba0 msgr2=0x7ff6d41a3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6d3400700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6d4073ba0 0x7ff6d41a3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6d3400700 1 -- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d4111de0 msgr2=0x7ff6d419d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6d3400700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d4111de0 0x7ff6d419d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6d3400700 1 -- 172.21.15.16:0/964510310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c40095f0 con 0x7ff6d4073280 2024-04-05T23:37:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6d4073ba0 0x7ff6d41a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:37:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6d3400700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d4073280 0x7ff6d41a3610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4000c00 tx=0x7ff6c4005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6cbe00700 1 -- 172.21.15.16:0/964510310 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c401d070 con 0x7ff6d4073280 2024-04-05T23:37:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6da400700 1 -- 172.21.15.16:0/964510310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d419e350 con 0x7ff6d4073280 2024-04-05T23:37:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6cbe00700 1 -- 172.21.15.16:0/964510310 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6c4022470 con 0x7ff6d4073280 2024-04-05T23:37:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6cbe00700 1 -- 172.21.15.16:0/964510310 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4017650 con 0x7ff6d4073280 2024-04-05T23:37:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.530+0000 7ff6da400700 1 -- 172.21.15.16:0/964510310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d41aa600 con 0x7ff6d4073280 2024-04-05T23:37:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.535+0000 7ff6cbe00700 1 -- 172.21.15.16:0/964510310 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7ff6c40225e0 con 0x7ff6d4073280 2024-04-05T23:37:51.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.536+0000 7ff6ede49b80 1 Processor -- start 2024-04-05T23:37:51.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.536+0000 7ff6ede49b80 1 -- start start 2024-04-05T23:37:51.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc40f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc3b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc3ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bba1b80 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bb92930 con 0x55ad2bc40b80 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bb902f0 con 0x55ad2bc414a0 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6d3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc40f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6d3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc40f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42156/0 (socket says 172.21.15.16:42156) 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6d3e00700 1 -- 172.21.15.16:0/3009529188 learned_addr learned my addr 172.21.15.16:0/3009529188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.537+0000 7ff6cbe00700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6b80a1310 0x7ff6b80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d3400700 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc3b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc3ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d3e00700 1 -- 172.21.15.16:0/3009529188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 msgr2=0x55ad2bc3b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc3b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d3e00700 1 -- 172.21.15.16:0/3009529188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 msgr2=0x55ad2bc3ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc3ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d3e00700 1 -- 172.21.15.16:0/3009529188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad2ba0d350 con 0x55ad2bc40b80 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6cbe00700 1 -- 172.21.15.16:0/964510310 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff6c40cdfb0 con 0x7ff6d4073280 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc3ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:37:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6b80a1310 0x7ff6b80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d3400700 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc3b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:37:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.538+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc40f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0116d0 tx=0x7ff6cc0119e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94d942bcb3f665dc server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.539+0000 7ff6c8c00700 1 -- 172.21.15.16:0/3009529188 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc00e040 con 0x55ad2bc40b80 2024-04-05T23:37:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.539+0000 7ff6c8c00700 1 -- 172.21.15.16:0/3009529188 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6cc00f040 con 0x55ad2bc40b80 2024-04-05T23:37:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.539+0000 7ff6c8c00700 1 -- 172.21.15.16:0/3009529188 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc020690 con 0x55ad2bc40b80 2024-04-05T23:37:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6ede49b80 1 -- 172.21.15.16:0/3009529188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 msgr2=0x55ad2bc40f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6b80a1310 0x7ff6b80a37d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc022000 tx=0x7ff6cc023000 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc40f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0116d0 tx=0x7ff6cc0119e0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6ede49b80 1 -- 172.21.15.16:0/3009529188 shutdown_connections 2024-04-05T23:37:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc3ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc40f60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/3009529188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc3b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6ede49b80 1 -- 172.21.15.16:0/3009529188 >> 172.21.15.16:0/3009529188 conn(0x55ad2bbaf1c0 msgr2=0x55ad2bc35be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6ede49b80 1 -- 172.21.15.16:0/3009529188 shutdown_connections 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.540+0000 7ff6ede49b80 1 -- 172.21.15.16:0/3009529188 wait complete. 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.541+0000 7ff6ede49b80 1 -- start start 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.541+0000 7ff6ede49b80 1 Processor -- start 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.541+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc47440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.541+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc40b80 0x55ad2bc447d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.541+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc44de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.541+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bb8bbe0 con 0x55ad2bc40b80 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.541+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bb27d20 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.541+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bb279c0 con 0x55ad2bc414a0 2024-04-05T23:37:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.541+0000 7ff6ede49b80 10 client.0 _ll_register_callbacks cb 0x55ad2b993340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc47440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc47440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42160/0 (socket says 172.21.15.16:42160) 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3400700 1 -- 172.21.15.16:0/950128762 learned_addr learned my addr 172.21.15.16:0/950128762 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc44de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc40b80 0x55ad2bc447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3400700 1 -- 172.21.15.16:0/950128762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 msgr2=0x55ad2bc44de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3400700 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc44de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3400700 1 -- 172.21.15.16:0/950128762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc40b80 msgr2=0x55ad2bc447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3400700 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc40b80 0x55ad2bc447d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3400700 1 -- 172.21.15.16:0/950128762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6cc011300 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc44de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6d3400700 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc47440 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400f740 tx=0x7ff6c406b320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d95c738423d9a140 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6bd600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:37:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6bd600700 1 -- 172.21.15.16:0/950128762 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c409a970 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6bd600700 1 -- 172.21.15.16:0/950128762 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff6c409b980 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.542+0000 7ff6ede49b80 1 -- 172.21.15.16:0/950128762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad2bc2e1e0 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6bd600700 1 -- 172.21.15.16:0/950128762 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6c409abe0 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6ede49b80 10 client.26195 fetch_fsmap learned FSMap version 44 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6ede49b80 1 -- 172.21.15.16:0/950128762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x55ad2b9b9b00 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6bd600700 1 -- 172.21.15.16:0/950128762 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ad2bc2e1e0 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6bd600700 1 -- 172.21.15.16:0/950128762 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff6c40a31e0 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6ede49b80 10 client.26195 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6ede49b80 10 client.26195 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6ede49b80 20 client.26195 populate_metadata read hostname 'smithi016' 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ad2bc54bb0 0x55ad2bc56fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6ede49b80 4 client.26195 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6ede49b80 1 -- 172.21.15.16:0/950128762 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad2bb846a0 con 0x55ad2bc54bb0 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.543+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ad2bc54bb0 0x55ad2bc56fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.544+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ad2bc54bb0 0x55ad2bc56fd0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54607c3343739e89 server_cookie=4cdd01524f90eeca in_seq=0 out_seq=0 2024-04-05T23:37:51.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.544+0000 7ff6bd600700 10 client.26195 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:51.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.544+0000 7ff6bd600700 1 -- 172.21.15.16:0/950128762 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ad2bb846a0 con 0x55ad2bc54bb0 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.544+0000 7ff6bd600700 10 client.26195 handle_command_reply: tid=0 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 client.26195 shutdown 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 -- 172.21.15.16:0/950128762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc3b5c0 msgr2=0x55ad2bc47440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc47440 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400f740 tx=0x7ff6c406b320 comp rx=0 tx=0).stop 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 -- 172.21.15.16:0/950128762 shutdown_connections 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ad2bc54bb0 0x55ad2bc56fd0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc44de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc47440 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6bd600700 0 client.26195 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/950128762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc40b80 0x55ad2bc447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 -- 172.21.15.16:0/950128762 >> 172.21.15.16:0/950128762 conn(0x55ad2bbaf1c0 msgr2=0x55ad2bc35280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 -- 172.21.15.16:0/950128762 shutdown_connections 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 1 -- 172.21.15.16:0/950128762 wait complete. 2024-04-05T23:37:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.545+0000 7ff6ede49b80 20 client.26195 trim_cache size 0 max 16384 2024-04-05T23:37:51.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.562+0000 7ff6ede49b80 1 Processor -- start 2024-04-05T23:37:51.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.562+0000 7ff6ede49b80 1 -- start start 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc50df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc35a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc440d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2ba0f640 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bc51360 con 0x55ad2bc40b80 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bc36e00 con 0x55ad2bc414a0 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6d2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc35a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6d2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc35a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42164/0 (socket says 172.21.15.16:42164) 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6d2a00700 1 -- 172.21.15.16:0/1775039977 learned_addr learned my addr 172.21.15.16:0/1775039977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6d3400700 1 --2- 172.21.15.16:0/1775039977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc50df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/1775039977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6d2a00700 1 -- 172.21.15.16:0/1775039977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 msgr2=0x55ad2bc440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/1775039977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6d2a00700 1 -- 172.21.15.16:0/1775039977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 msgr2=0x55ad2bc50df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.563+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/1775039977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc50df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.564+0000 7ff6d2a00700 1 -- 172.21.15.16:0/1775039977 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad2bb51e40 con 0x55ad2bc40b80 2024-04-05T23:37:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.564+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/1775039977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc35a00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0004790 tx=0x7ff6c0004640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea3c6d36a15e37a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.564+0000 7ff6c9600700 1 -- 172.21.15.16:0/1775039977 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0017040 con 0x55ad2bc40b80 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.564+0000 7ff6c9600700 1 -- 172.21.15.16:0/1775039977 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6c0019040 con 0x55ad2bc40b80 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.564+0000 7ff6c9600700 1 -- 172.21.15.16:0/1775039977 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c00128c0 con 0x55ad2bc40b80 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 -- 172.21.15.16:0/1775039977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 msgr2=0x55ad2bc35a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/1775039977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc35a00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0004790 tx=0x7ff6c0004640 comp rx=0 tx=0).stop 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 -- 172.21.15.16:0/1775039977 shutdown_connections 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/1775039977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc50df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/1775039977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc414a0 0x55ad2bc440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/1775039977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc40b80 0x55ad2bc35a00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 -- 172.21.15.16:0/1775039977 >> 172.21.15.16:0/1775039977 conn(0x55ad2bbaf1c0 msgr2=0x55ad2bc506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 -- 172.21.15.16:0/1775039977 shutdown_connections 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 -- 172.21.15.16:0/1775039977 wait complete. 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.565+0000 7ff6ede49b80 1 -- start start 2024-04-05T23:37:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6ede49b80 1 Processor -- start 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc4b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc40b80 0x55ad2bc4beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6ede49b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc414a0 0x55ad2bc49fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bc4a540 con 0x55ad2bc3b5c0 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bc4a6b0 con 0x55ad2bc414a0 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6ede49b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2bc4a870 con 0x55ad2bc40b80 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6ede49b80 10 client.0 _ll_register_callbacks cb 0x7ff6c400bf90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6d2a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc40b80 0x55ad2bc4beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6d2a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc40b80 0x55ad2bc4beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44354/0 (socket says 172.21.15.16:44354) 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6d2a00700 1 -- 172.21.15.16:0/2569501638 learned_addr learned my addr 172.21.15.16:0/2569501638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:37:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.566+0000 7ff6d3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc4b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6d3e00700 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc414a0 0x55ad2bc49fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6d2a00700 1 -- 172.21.15.16:0/2569501638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc414a0 msgr2=0x55ad2bc49fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc414a0 0x55ad2bc49fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6d2a00700 1 -- 172.21.15.16:0/2569501638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 msgr2=0x55ad2bc4b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc4b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6d2a00700 1 -- 172.21.15.16:0/2569501638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c0004040 con 0x55ad2bc40b80 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6d2a00700 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc40b80 0x55ad2bc4beb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0000c00 tx=0x7ff6c0005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f421de15e697696 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6bd600700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6bd600700 1 -- 172.21.15.16:0/2569501638 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0017050 con 0x55ad2bc40b80 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6ede49b80 1 -- 172.21.15.16:0/2569501638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6cc00f040 con 0x55ad2bc40b80 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.567+0000 7ff6bd600700 1 -- 172.21.15.16:0/2569501638 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7ff6c001e910 con 0x55ad2bc40b80 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6bd600700 1 -- 172.21.15.16:0/2569501638 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6c001ead0 con 0x55ad2bc40b80 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6ede49b80 10 client.26236 fetch_fsmap learned FSMap version 44 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6ede49b80 1 -- 172.21.15.16:0/2569501638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x55ad2ba11c70 con 0x55ad2bc40b80 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6bd600700 1 -- 172.21.15.16:0/2569501638 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6cc00f040 con 0x55ad2bc40b80 2024-04-05T23:37:51.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6bd600700 1 -- 172.21.15.16:0/2569501638 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff6c0037070 con 0x55ad2bc40b80 2024-04-05T23:37:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6ede49b80 10 client.26236 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:37:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6ede49b80 10 client.26236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:37:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6ede49b80 20 client.26236 populate_metadata read hostname 'smithi016' 2024-04-05T23:37:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ad2bc54bb0 0x55ad2bcabc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:37:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6ede49b80 4 client.26236 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:37:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.568+0000 7ff6ede49b80 1 -- 172.21.15.16:0/2569501638 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ad2b9dac90 con 0x55ad2bc54bb0 2024-04-05T23:37:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.569+0000 7ff6d3400700 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ad2bc54bb0 0x55ad2bcabc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:37:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.569+0000 7ff6bd600700 10 client.26236 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:51.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.569+0000 7ff6d3400700 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ad2bc54bb0 0x55ad2bcabc60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c940ee2666a8387 server_cookie=81b6735f5bc9470e in_seq=0 out_seq=0 2024-04-05T23:37:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.612+0000 7ff6bd600700 1 -- 172.21.15.16:0/2569501638 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ad2b9dac90 con 0x55ad2bc54bb0 2024-04-05T23:37:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.612+0000 7ff6bd600700 10 client.26236 handle_command_reply: tid=0 2024-04-05T23:37:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.612+0000 7ff6ede49b80 1 client.26236 shutdown 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 -- 172.21.15.16:0/2569501638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc40b80 msgr2=0x55ad2bc4beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc40b80 0x55ad2bc4beb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0000c00 tx=0x7ff6c0005e50 comp rx=0 tx=0).stop 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 -- 172.21.15.16:0/2569501638 shutdown_connections 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ad2bc54bb0 0x55ad2bcabc60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad2bc3b5c0 0x55ad2bc4b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ad2bc414a0 0x55ad2bc49fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6bd600700 0 client.26236 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 --2- 172.21.15.16:0/2569501638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ad2bc40b80 0x55ad2bc4beb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 -- 172.21.15.16:0/2569501638 >> 172.21.15.16:0/2569501638 conn(0x55ad2bbaf1c0 msgr2=0x55ad2bc45fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 -- 172.21.15.16:0/2569501638 shutdown_connections 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 1 -- 172.21.15.16:0/2569501638 wait complete. 2024-04-05T23:37:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.613+0000 7ff6ede49b80 20 client.26236 trim_cache size 0 max 16384 2024-04-05T23:37:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:37:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:37:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:37:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 -- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6b80a1310 msgr2=0x7ff6b80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6b80a1310 0x7ff6b80a37d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc022000 tx=0x7ff6cc023000 comp rx=0 tx=0).stop 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 -- 172.21.15.16:0/964510310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d4073280 msgr2=0x7ff6d41a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d4073280 0x7ff6d41a3610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4000c00 tx=0x7ff6c4005740 comp rx=0 tx=0).stop 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 -- 172.21.15.16:0/964510310 shutdown_connections 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff6b80a1310 0x7ff6b80a37d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d4111de0 0x7ff6d419d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d4073280 0x7ff6d41a3610 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 --2- 172.21.15.16:0/964510310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6d4073ba0 0x7ff6d41a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 -- 172.21.15.16:0/964510310 >> 172.21.15.16:0/964510310 conn(0x7ff6d40fc8e0 msgr2=0x7ff6d40fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.614+0000 7ff6da400700 1 -- 172.21.15.16:0/964510310 shutdown_connections 2024-04-05T23:37:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:37:51.615+0000 7ff6da400700 1 -- 172.21.15.16:0/964510310 wait complete. 2024-04-05T23:37:51.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:51 smithi027 ceph-mon[78597]: pgmap v133: 97 pgs: 97 active+clean; 109 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 375 op/s 2024-04-05T23:37:51.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 246292 9.56 MB/sec execute 71 sec latency 41.639 ms 2024-04-05T23:37:52.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:51 smithi016 ceph-mon[38248]: pgmap v133: 97 pgs: 97 active+clean; 109 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 375 op/s 2024-04-05T23:37:52.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:51 smithi130 ceph-mon[79417]: pgmap v133: 97 pgs: 97 active+clean; 109 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 375 op/s 2024-04-05T23:37:52.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 248558 9.62 MB/sec execute 72 sec latency 15.014 ms 2024-04-05T23:37:53.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:53 smithi027 ceph-mon[78597]: pgmap v134: 97 pgs: 97 active+clean; 109 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 230 op/s 2024-04-05T23:37:53.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 249618 9.52 MB/sec execute 73 sec latency 15.268 ms 2024-04-05T23:37:54.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:53 smithi016 ceph-mon[38248]: pgmap v134: 97 pgs: 97 active+clean; 109 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 230 op/s 2024-04-05T23:37:54.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:53 smithi130 ceph-mon[79417]: pgmap v134: 97 pgs: 97 active+clean; 109 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 230 op/s 2024-04-05T23:37:54.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 252091 9.62 MB/sec execute 74 sec latency 11.720 ms 2024-04-05T23:37:55.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:55 smithi027 ceph-mon[78597]: pgmap v135: 97 pgs: 97 active+clean; 109 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 230 op/s 2024-04-05T23:37:55.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 253058 9.52 MB/sec execute 75 sec latency 40.835 ms 2024-04-05T23:37:56.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:55 smithi016 ceph-mon[38248]: pgmap v135: 97 pgs: 97 active+clean; 109 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 230 op/s 2024-04-05T23:37:56.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:55 smithi130 ceph-mon[79417]: pgmap v135: 97 pgs: 97 active+clean; 109 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 230 op/s 2024-04-05T23:37:56.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 255505 9.60 MB/sec execute 76 sec latency 16.227 ms 2024-04-05T23:37:57.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:57 smithi027 ceph-mon[78597]: pgmap v136: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 357 op/s 2024-04-05T23:37:57.994 INFO:tasks.workunit.client.0.smithi016.stdout: 1 256584 9.52 MB/sec execute 77 sec latency 11.554 ms 2024-04-05T23:37:58.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:57 smithi016 ceph-mon[38248]: pgmap v136: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 357 op/s 2024-04-05T23:37:58.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:57 smithi130 ceph-mon[79417]: pgmap v136: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 357 op/s 2024-04-05T23:37:58.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 259029 9.60 MB/sec execute 78 sec latency 13.648 ms 2024-04-05T23:37:59.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:37:59 smithi027 ceph-mon[78597]: pgmap v137: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 243 op/s 2024-04-05T23:37:59.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 260063 9.52 MB/sec execute 79 sec latency 14.242 ms 2024-04-05T23:38:00.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:37:59 smithi016 ceph-mon[38248]: pgmap v137: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 243 op/s 2024-04-05T23:38:00.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:37:59 smithi130 ceph-mon[79417]: pgmap v137: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 243 op/s 2024-04-05T23:38:00.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 262212 9.55 MB/sec execute 80 sec latency 83.991 ms 2024-04-05T23:38:01.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:01 smithi027 ceph-mon[78597]: pgmap v138: 97 pgs: 97 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 356 op/s 2024-04-05T23:38:01.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 263047 9.50 MB/sec execute 81 sec latency 29.741 ms 2024-04-05T23:38:02.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:01 smithi016 ceph-mon[38248]: pgmap v138: 97 pgs: 97 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 356 op/s 2024-04-05T23:38:02.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:01 smithi130 ceph-mon[79417]: pgmap v138: 97 pgs: 97 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 356 op/s 2024-04-05T23:38:02.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 265114 9.48 MB/sec execute 82 sec latency 13.665 ms 2024-04-05T23:38:03.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:03 smithi027 ceph-mon[78597]: pgmap v139: 97 pgs: 97 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-04-05T23:38:03.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266522 9.50 MB/sec execute 83 sec latency 11.754 ms 2024-04-05T23:38:04.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:03 smithi016 ceph-mon[38248]: pgmap v139: 97 pgs: 97 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-04-05T23:38:04.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:03 smithi130 ceph-mon[79417]: pgmap v139: 97 pgs: 97 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-04-05T23:38:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.791+0000 7faaebe00700 1 -- 172.21.15.16:0/3770402130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae4101030 msgr2=0x7faae4101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.791+0000 7faaebe00700 1 --2- 172.21.15.16:0/3770402130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae4101030 0x7faae4101410 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7faad8009990 tx=0x7faad8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.791+0000 7faaebe00700 1 -- 172.21.15.16:0/3770402130 shutdown_connections 2024-04-05T23:38:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.791+0000 7faaebe00700 1 --2- 172.21.15.16:0/3770402130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae4101030 0x7faae4101410 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.791+0000 7faaebe00700 1 --2- 172.21.15.16:0/3770402130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae4106150 0x7faae410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.791+0000 7faaebe00700 1 --2- 172.21.15.16:0/3770402130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faae41019e0 0x7faae4105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.791+0000 7faaebe00700 1 -- 172.21.15.16:0/3770402130 >> 172.21.15.16:0/3770402130 conn(0x7faae40fc8b0 msgr2=0x7faae40fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.791+0000 7faaebe00700 1 -- 172.21.15.16:0/3770402130 shutdown_connections 2024-04-05T23:38:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.792+0000 7faaebe00700 1 -- 172.21.15.16:0/3770402130 wait complete. 2024-04-05T23:38:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.792+0000 7faaebe00700 1 Processor -- start 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faaebe00700 1 -- start start 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faaebe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae4101030 0x7faae4199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faaebe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae4106150 0x7faae4199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faaebe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faae419de30 0x7faae419e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faaebe00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae419e8b0 con 0x7faae4106150 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faaebe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae419ea20 con 0x7faae4101030 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faaebe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae419ebe0 con 0x7faae419de30 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faae9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae4101030 0x7faae4199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faae9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae4101030 0x7faae4199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:34768/0 (socket says 172.21.15.16:34768) 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faae8a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae4106150 0x7faae4199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faae9400700 1 -- 172.21.15.16:0/1428108944 learned_addr learned my addr 172.21.15.16:0/1428108944 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.793+0000 7faae9e00700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faae419de30 0x7faae419e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faae9400700 1 -- 172.21.15.16:0/1428108944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faae419de30 msgr2=0x7faae419e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faae9400700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faae419de30 0x7faae419e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faae9400700 1 -- 172.21.15.16:0/1428108944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae4106150 msgr2=0x7faae4199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faae9400700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae4106150 0x7faae4199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faae9400700 1 -- 172.21.15.16:0/1428108944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad80095f0 con 0x7faae4101030 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faae9400700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae4101030 0x7faae4199550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faad8000c00 tx=0x7faad8005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faae2000700 1 -- 172.21.15.16:0/1428108944 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad801d070 con 0x7faae4101030 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faae2000700 1 -- 172.21.15.16:0/1428108944 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faad8022470 con 0x7faae4101030 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faaebe00700 1 -- 172.21.15.16:0/1428108944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae41a60e0 con 0x7faae4101030 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faae2000700 1 -- 172.21.15.16:0/1428108944 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8017650 con 0x7faae4101030 2024-04-05T23:38:04.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.794+0000 7faaebe00700 1 -- 172.21.15.16:0/1428108944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae41a6560 con 0x7faae4101030 2024-04-05T23:38:04.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.797+0000 7faae2000700 1 -- 172.21.15.16:0/1428108944 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7faad80225e0 con 0x7faae4101030 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faae2000700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7faacc0a1310 0x7faacc0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faaff9aab80 1 Processor -- start 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faae2000700 1 -- 172.21.15.16:0/1428108944 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7faad80cdfa0 con 0x7faae4101030 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faaff9aab80 1 -- start start 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faae8a00700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7faacc0a1310 0x7faacc0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba70b80 0x55915ba70f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba6b5c0 0x55915ba6ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faaff9aab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915b9d1b80 con 0x55915ba70b80 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faaff9aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915b9c2930 con 0x55915ba714a0 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.799+0000 7faaff9aab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915b9c02f0 con 0x55915ba6b5c0 2024-04-05T23:38:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba70b80 0x55915ba70f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba6b5c0 0x55915ba6ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba70b80 0x55915ba70f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32976/0 (socket says 172.21.15.16:32976) 2024-04-05T23:38:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9400700 1 -- 172.21.15.16:0/2191055686 learned_addr learned my addr 172.21.15.16:0/2191055686 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae8a00700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7faacc0a1310 0x7faacc0a37d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faae419aaa0 tx=0x7faad0009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9e00700 1 -- 172.21.15.16:0/2191055686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 msgr2=0x55915ba6b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9e00700 1 --2- 172.21.15.16:0/2191055686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9e00700 1 -- 172.21.15.16:0/2191055686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba70b80 msgr2=0x55915ba70f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9e00700 1 --2- 172.21.15.16:0/2191055686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba70b80 0x55915ba70f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9e00700 1 -- 172.21.15.16:0/2191055686 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55915b83d350 con 0x55915ba6b5c0 2024-04-05T23:38:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9400700 1 --2- 172.21.15.16:0/2191055686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba70b80 0x55915ba70f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.800+0000 7faae9e00700 1 --2- 172.21.15.16:0/2191055686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba6b5c0 0x55915ba6ba20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faadc011580 tx=0x7faadc009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b37d3974769bfe7a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.801+0000 7faad6600700 1 -- 172.21.15.16:0/2191055686 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc00e040 con 0x55915ba6b5c0 2024-04-05T23:38:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.801+0000 7faad6600700 1 -- 172.21.15.16:0/2191055686 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faadc00f040 con 0x55915ba6b5c0 2024-04-05T23:38:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.801+0000 7faad6600700 1 -- 172.21.15.16:0/2191055686 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc0044f0 con 0x55915ba6b5c0 2024-04-05T23:38:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.801+0000 7faaff9aab80 1 -- 172.21.15.16:0/2191055686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba6b5c0 msgr2=0x55915ba6ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.801+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2191055686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba6b5c0 0x55915ba6ba20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faadc011580 tx=0x7faadc009920 comp rx=0 tx=0).stop 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.802+0000 7faaff9aab80 1 -- 172.21.15.16:0/2191055686 shutdown_connections 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.802+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2191055686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba70b80 0x55915ba70f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.802+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2191055686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba6b5c0 0x55915ba6ba20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.802+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2191055686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.802+0000 7faaff9aab80 1 -- 172.21.15.16:0/2191055686 >> 172.21.15.16:0/2191055686 conn(0x55915b9df1c0 msgr2=0x55915ba65be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.802+0000 7faaff9aab80 1 -- 172.21.15.16:0/2191055686 shutdown_connections 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.802+0000 7faaff9aab80 1 -- 172.21.15.16:0/2191055686 wait complete. 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.802+0000 7faaff9aab80 1 -- start start 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.802+0000 7faaff9aab80 1 Processor -- start 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba6b5c0 0x55915ba77480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba74770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba714a0 0x55915ba74df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faaff9aab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915b9bbbe0 con 0x55915ba714a0 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faaff9aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915b957d20 con 0x55915ba6b5c0 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faaff9aab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915b9579c0 con 0x55915ba70b80 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faaff9aab80 10 client.0 _ll_register_callbacks cb 0x55915b7c3340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faae9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba6b5c0 0x55915ba77480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faae9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba714a0 0x55915ba74df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faae9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba6b5c0 0x55915ba77480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:34784/0 (socket says 172.21.15.16:34784) 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faae9400700 1 -- 172.21.15.16:0/3317579337 learned_addr learned my addr 172.21.15.16:0/3317579337 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.803+0000 7faae8a00700 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba74770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faae9400700 1 -- 172.21.15.16:0/3317579337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 msgr2=0x55915ba74770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faae9400700 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba74770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faae9400700 1 -- 172.21.15.16:0/3317579337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba714a0 msgr2=0x55915ba74df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faae9400700 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba714a0 0x55915ba74df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faae9400700 1 -- 172.21.15.16:0/3317579337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae41a6560 con 0x55915ba6b5c0 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faae9e00700 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba714a0 0x55915ba74df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faae9400700 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba6b5c0 0x55915ba77480 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faad8005f50 tx=0x7faad809b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d3ee563c391f190 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faacaa00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faacaa00700 1 -- 172.21.15.16:0/3317579337 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad809a920 con 0x55915ba6b5c0 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faacaa00700 1 -- 172.21.15.16:0/3317579337 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7faad80b2030 con 0x55915ba6b5c0 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.804+0000 7faaff9aab80 1 -- 172.21.15.16:0/3317579337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55915ba5e1e0 con 0x55915ba6b5c0 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faacaa00700 1 -- 172.21.15.16:0/3317579337 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad809ab90 con 0x55915ba6b5c0 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faaff9aab80 10 client.26213 fetch_fsmap learned FSMap version 44 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faaff9aab80 1 -- 172.21.15.16:0/3317579337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x7faadc011230 con 0x55915ba6b5c0 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faacaa00700 1 -- 172.21.15.16:0/3317579337 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55915ba5e1e0 con 0x55915ba6b5c0 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faacaa00700 1 -- 172.21.15.16:0/3317579337 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7faad80a3190 con 0x55915ba6b5c0 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faaff9aab80 10 client.26213 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faaff9aab80 10 client.26213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faaff9aab80 20 client.26213 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faaff9aab80 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55915ba84bb0 0x55915ba86fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faaff9aab80 4 client.26213 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:38:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faaff9aab80 1 -- 172.21.15.16:0/3317579337 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55915b9b46a0 con 0x55915ba84bb0 2024-04-05T23:38:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.805+0000 7faae8a00700 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55915ba84bb0 0x55915ba86fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.806+0000 7faae8a00700 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55915ba84bb0 0x55915ba86fd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8e14aaf9a5a1fa6 server_cookie=969a7506928a8d67 in_seq=0 out_seq=0 2024-04-05T23:38:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.806+0000 7faacaa00700 10 client.26213 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.807+0000 7faacaa00700 1 -- 172.21.15.16:0/3317579337 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55915b9b46a0 con 0x55915ba84bb0 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.807+0000 7faacaa00700 10 client.26213 handle_command_reply: tid=0 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.807+0000 7faaff9aab80 1 client.26213 shutdown 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 -- 172.21.15.16:0/3317579337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba6b5c0 msgr2=0x55915ba77480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba6b5c0 0x55915ba77480 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faad8005f50 tx=0x7faad809b000 comp rx=0 tx=0).stop 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 -- 172.21.15.16:0/3317579337 shutdown_connections 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55915ba84bb0 0x55915ba86fd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba714a0 0x55915ba74df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba6b5c0 0x55915ba77480 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 --2- 172.21.15.16:0/3317579337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba74770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faacaa00700 0 client.26213 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 -- 172.21.15.16:0/3317579337 >> 172.21.15.16:0/3317579337 conn(0x55915b9df1c0 msgr2=0x55915ba65280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 -- 172.21.15.16:0/3317579337 shutdown_connections 2024-04-05T23:38:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 1 -- 172.21.15.16:0/3317579337 wait complete. 2024-04-05T23:38:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.808+0000 7faaff9aab80 20 client.26213 trim_cache size 0 max 16384 2024-04-05T23:38:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.824+0000 7faaff9aab80 1 Processor -- start 2024-04-05T23:38:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.824+0000 7faaff9aab80 1 -- start start 2024-04-05T23:38:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 0x55915ba6b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba7bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba65a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faaff9aab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915b83f640 con 0x55915ba6b5c0 2024-04-05T23:38:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faaff9aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915ba67400 con 0x55915ba714a0 2024-04-05T23:38:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faaff9aab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915ba62af0 con 0x55915ba70b80 2024-04-05T23:38:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faae9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 0x55915ba6b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faae9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba65a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faae9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba65a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:34792/0 (socket says 172.21.15.16:34792) 2024-04-05T23:38:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faae9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 0x55915ba6b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32990/0 (socket says 172.21.15.16:32990) 2024-04-05T23:38:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faae9e00700 1 -- 172.21.15.16:0/709225966 learned_addr learned my addr 172.21.15.16:0/709225966 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.825+0000 7faae8a00700 1 --2- 172.21.15.16:0/709225966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba7bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faae9e00700 1 -- 172.21.15.16:0/709225966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 msgr2=0x55915ba7bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faae9e00700 1 --2- 172.21.15.16:0/709225966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba7bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faae9e00700 1 -- 172.21.15.16:0/709225966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 msgr2=0x55915ba6b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faae9e00700 1 --2- 172.21.15.16:0/709225966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 0x55915ba6b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faae9e00700 1 -- 172.21.15.16:0/709225966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55915ba621c0 con 0x55915ba714a0 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faae8a00700 1 --2- 172.21.15.16:0/709225966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba7bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faae9e00700 1 --2- 172.21.15.16:0/709225966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba65a00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faadc011580 tx=0x7faadc011e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9a1cea2f585efb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faad7000700 1 -- 172.21.15.16:0/709225966 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc00e040 con 0x55915ba714a0 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faad7000700 1 -- 172.21.15.16:0/709225966 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faadc003ec0 con 0x55915ba714a0 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.826+0000 7faad7000700 1 -- 172.21.15.16:0/709225966 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc007500 con 0x55915ba714a0 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.827+0000 7faaff9aab80 1 -- 172.21.15.16:0/709225966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 msgr2=0x55915ba65a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.827+0000 7faaff9aab80 1 --2- 172.21.15.16:0/709225966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba65a00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faadc011580 tx=0x7faadc011e80 comp rx=0 tx=0).stop 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.827+0000 7faaff9aab80 1 -- 172.21.15.16:0/709225966 shutdown_connections 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.827+0000 7faaff9aab80 1 --2- 172.21.15.16:0/709225966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 0x55915ba6b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.827+0000 7faaff9aab80 1 --2- 172.21.15.16:0/709225966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba65a00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.827+0000 7faaff9aab80 1 --2- 172.21.15.16:0/709225966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba7bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.827+0000 7faaff9aab80 1 -- 172.21.15.16:0/709225966 >> 172.21.15.16:0/709225966 conn(0x55915b9df1c0 msgr2=0x55915ba73040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.827+0000 7faaff9aab80 1 -- 172.21.15.16:0/709225966 shutdown_connections 2024-04-05T23:38:04.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.827+0000 7faaff9aab80 1 -- 172.21.15.16:0/709225966 wait complete. 2024-04-05T23:38:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.828+0000 7faaff9aab80 1 -- start start 2024-04-05T23:38:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.828+0000 7faaff9aab80 1 Processor -- start 2024-04-05T23:38:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.828+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 0x55915ba6d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.828+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba6df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faaff9aab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faaff9aab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915ba6cbb0 con 0x55915ba6b5c0 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faaff9aab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915ba6cd20 con 0x55915ba714a0 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faaff9aab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915ba6cee0 con 0x55915ba70b80 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faaff9aab80 10 client.0 _ll_register_callbacks cb 0x7faad80057b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 0x55915ba6d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:34802/0 (socket says 172.21.15.16:34802) 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae9e00700 1 -- 172.21.15.16:0/2024993059 learned_addr learned my addr 172.21.15.16:0/2024993059 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae8a00700 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba6df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae9e00700 1 -- 172.21.15.16:0/2024993059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 msgr2=0x55915ba6df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae9e00700 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba6df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae9e00700 1 -- 172.21.15.16:0/2024993059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 msgr2=0x55915ba6d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae9e00700 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 0x55915ba6d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.829+0000 7faae9e00700 1 -- 172.21.15.16:0/2024993059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad0021040 con 0x55915ba714a0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.830+0000 7faae9e00700 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6c600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faadc00c000 tx=0x7faadc003c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9236c97fccf52086 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.830+0000 7faacaa00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.830+0000 7faacaa00700 1 -- 172.21.15.16:0/2024993059 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc004230 con 0x55915ba714a0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.830+0000 7faacaa00700 1 -- 172.21.15.16:0/2024993059 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7faadc004390 con 0x55915ba714a0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.830+0000 7faaff9aab80 1 -- 172.21.15.16:0/2024993059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faadc00f040 con 0x55915ba714a0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.830+0000 7faacaa00700 1 -- 172.21.15.16:0/2024993059 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7faadc007500 con 0x55915ba714a0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.830+0000 7faaff9aab80 10 client.26219 fetch_fsmap learned FSMap version 44 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.830+0000 7faaff9aab80 1 -- 172.21.15.16:0/2024993059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x55915b83d350 con 0x55915ba714a0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.830+0000 7faacaa00700 1 -- 172.21.15.16:0/2024993059 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faadc00f040 con 0x55915ba714a0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faacaa00700 1 -- 172.21.15.16:0/2024993059 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7faadc01b020 con 0x55915ba714a0 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faaff9aab80 10 client.26219 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faaff9aab80 10 client.26219 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faaff9aab80 20 client.26219 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55915bad9550 0x55915badb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faaff9aab80 4 client.26219 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:38:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faaff9aab80 1 -- 172.21.15.16:0/2024993059 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55915b80ac90 con 0x55915bad9550 2024-04-05T23:38:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faae9400700 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55915bad9550 0x55915badb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faacaa00700 10 client.26219 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.831+0000 7faae9400700 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55915bad9550 0x55915badb940 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b19b938b737bcc32 server_cookie=4b81e17249e9d500 in_seq=0 out_seq=0 2024-04-05T23:38:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faacaa00700 1 -- 172.21.15.16:0/2024993059 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55915b80ac90 con 0x55915bad9550 2024-04-05T23:38:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faacaa00700 10 client.26219 handle_command_reply: tid=0 2024-04-05T23:38:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faaff9aab80 1 client.26219 shutdown 2024-04-05T23:38:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faaff9aab80 1 -- 172.21.15.16:0/2024993059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 msgr2=0x55915ba6c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6c600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faadc00c000 tx=0x7faadc003c30 comp rx=0 tx=0).stop 2024-04-05T23:38:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faaff9aab80 1 -- 172.21.15.16:0/2024993059 shutdown_connections 2024-04-05T23:38:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55915bad9550 0x55915badb940 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55915ba6b5c0 0x55915ba6d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55915ba714a0 0x55915ba6c600 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faaff9aab80 1 --2- 172.21.15.16:0/2024993059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55915ba70b80 0x55915ba6df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faaff9aab80 1 -- 172.21.15.16:0/2024993059 >> 172.21.15.16:0/2024993059 conn(0x55915b9df1c0 msgr2=0x55915ba80710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.881+0000 7faacaa00700 0 client.26219 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.882+0000 7faaff9aab80 1 -- 172.21.15.16:0/2024993059 shutdown_connections 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.882+0000 7faaff9aab80 1 -- 172.21.15.16:0/2024993059 wait complete. 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.882+0000 7faaff9aab80 20 client.26219 trim_cache size 0 max 16384 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:38:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:38:04.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:38:04.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 -- 172.21.15.16:0/1428108944 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7faacc0a1310 msgr2=0x7faacc0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7faacc0a1310 0x7faacc0a37d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faae419aaa0 tx=0x7faad0009260 comp rx=0 tx=0).stop 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 -- 172.21.15.16:0/1428108944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae4101030 msgr2=0x7faae4199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae4101030 0x7faae4199550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faad8000c00 tx=0x7faad8005740 comp rx=0 tx=0).stop 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 -- 172.21.15.16:0/1428108944 shutdown_connections 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7faacc0a1310 0x7faacc0a37d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faae419de30 0x7faae419e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae4101030 0x7faae4199550 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 --2- 172.21.15.16:0/1428108944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae4106150 0x7faae4199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 -- 172.21.15.16:0/1428108944 >> 172.21.15.16:0/1428108944 conn(0x7faae40fc8b0 msgr2=0x7faae40fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 -- 172.21.15.16:0/1428108944 shutdown_connections 2024-04-05T23:38:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:04.883+0000 7faaebe00700 1 -- 172.21.15.16:0/1428108944 wait complete. 2024-04-05T23:38:04.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 268528 9.48 MB/sec execute 84 sec latency 39.857 ms 2024-04-05T23:38:05.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:05 smithi027 ceph-mon[78597]: pgmap v140: 97 pgs: 97 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-04-05T23:38:05.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270013 9.50 MB/sec execute 85 sec latency 28.642 ms 2024-04-05T23:38:06.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:05 smithi016 ceph-mon[38248]: pgmap v140: 97 pgs: 97 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-04-05T23:38:06.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:05 smithi130 ceph-mon[79417]: pgmap v140: 97 pgs: 97 active+clean; 102 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-04-05T23:38:06.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 272081 9.48 MB/sec execute 86 sec latency 16.468 ms 2024-04-05T23:38:07.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:07 smithi027 ceph-mon[78597]: pgmap v141: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 411 op/s 2024-04-05T23:38:07.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273609 9.50 MB/sec execute 87 sec latency 13.577 ms 2024-04-05T23:38:08.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:07 smithi016 ceph-mon[38248]: pgmap v141: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 411 op/s 2024-04-05T23:38:08.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:07 smithi130 ceph-mon[79417]: pgmap v141: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 411 op/s 2024-04-05T23:38:08.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273745 9.40 MB/sec execute 88 sec latency 882.529 ms 2024-04-05T23:38:09.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:09 smithi027 ceph-mon[78597]: pgmap v142: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 283 op/s 2024-04-05T23:38:09.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273745 9.29 MB/sec execute 89 sec latency 1882.696 ms 2024-04-05T23:38:10.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:09 smithi016 ceph-mon[38248]: pgmap v142: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 283 op/s 2024-04-05T23:38:10.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:09 smithi130 ceph-mon[79417]: pgmap v142: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 283 op/s 2024-04-05T23:38:10.995 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273745 9.19 MB/sec execute 90 sec latency 2882.834 ms 2024-04-05T23:38:11.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273745 9.09 MB/sec execute 91 sec latency 3882.980 ms 2024-04-05T23:38:12.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:11 smithi016 ceph-mon[38248]: pgmap v143: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 358 op/s 2024-04-05T23:38:12.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:11 smithi130 ceph-mon[79417]: pgmap v143: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 358 op/s 2024-04-05T23:38:12.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:11 smithi027 ceph-mon[78597]: pgmap v143: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 358 op/s 2024-04-05T23:38:12.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273745 8.99 MB/sec execute 92 sec latency 4883.139 ms 2024-04-05T23:38:13.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273745 8.89 MB/sec execute 93 sec latency 5883.215 ms 2024-04-05T23:38:14.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:13 smithi016 ceph-mon[38248]: pgmap v144: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 2.3 MiB/s wr, 244 op/s 2024-04-05T23:38:14.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:13 smithi130 ceph-mon[79417]: pgmap v144: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 2.3 MiB/s wr, 244 op/s 2024-04-05T23:38:14.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:13 smithi027 ceph-mon[78597]: pgmap v144: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 2.3 MiB/s wr, 244 op/s 2024-04-05T23:38:14.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273745 8.80 MB/sec execute 94 sec latency 6883.269 ms 2024-04-05T23:38:15.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 275518 8.78 MB/sec execute 95 sec latency 7003.509 ms 2024-04-05T23:38:16.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:15 smithi016 ceph-mon[38248]: pgmap v145: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 2.3 MiB/s wr, 244 op/s 2024-04-05T23:38:16.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:15 smithi130 ceph-mon[79417]: pgmap v145: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 2.3 MiB/s wr, 244 op/s 2024-04-05T23:38:16.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:15 smithi027 ceph-mon[78597]: pgmap v145: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 2.3 MiB/s wr, 244 op/s 2024-04-05T23:38:16.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277032 8.81 MB/sec execute 96 sec latency 48.376 ms 2024-04-05T23:38:17.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 278352 8.78 MB/sec execute 97 sec latency 17.995 ms 2024-04-05T23:38:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.042+0000 7f43b7200700 1 -- 172.21.15.16:0/834817590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b0103a70 msgr2=0x7f43b0107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.042+0000 7f43b7200700 1 --2- 172.21.15.16:0/834817590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b0103a70 0x7f43b0107ab0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009990 tx=0x7f43a0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.042+0000 7f43b7200700 1 -- 172.21.15.16:0/834817590 shutdown_connections 2024-04-05T23:38:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.042+0000 7f43b7200700 1 --2- 172.21.15.16:0/834817590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b01081e0 0x7f43b010fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.042+0000 7f43b7200700 1 --2- 172.21.15.16:0/834817590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43b01030c0 0x7f43b01034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.042+0000 7f43b7200700 1 --2- 172.21.15.16:0/834817590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b0103a70 0x7f43b0107ab0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.042+0000 7f43b7200700 1 -- 172.21.15.16:0/834817590 >> 172.21.15.16:0/834817590 conn(0x7f43b00fe8e0 msgr2=0x7f43b0100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.043+0000 7f43b7200700 1 -- 172.21.15.16:0/834817590 shutdown_connections 2024-04-05T23:38:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.043+0000 7f43b7200700 1 -- 172.21.15.16:0/834817590 wait complete. 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.044+0000 7f43b7200700 1 Processor -- start 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.044+0000 7f43b7200700 1 -- start start 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.044+0000 7f43b7200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b01030c0 0x7f43b0199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.044+0000 7f43b7200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b01081e0 0x7f43b01998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.044+0000 7f43b7200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43b019dc10 0x7f43b019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.044+0000 7f43b7200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b019e690 con 0x7f43b01081e0 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.044+0000 7f43b7200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b019e800 con 0x7f43b01030c0 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.044+0000 7f43b7200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b019e9c0 con 0x7f43b019dc10 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.044+0000 7f43b5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b01030c0 0x7f43b0199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b01030c0 0x7f43b0199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56774/0 (socket says 172.21.15.16:56774) 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b5e00700 1 -- 172.21.15.16:0/3338951610 learned_addr learned my addr 172.21.15.16:0/3338951610 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b5400700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b01081e0 0x7f43b01998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b6800700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43b019dc10 0x7f43b019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b5e00700 1 -- 172.21.15.16:0/3338951610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43b019dc10 msgr2=0x7f43b019e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b5e00700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43b019dc10 0x7f43b019e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b5e00700 1 -- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b01081e0 msgr2=0x7f43b01998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b5e00700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b01081e0 0x7f43b01998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b5e00700 1 -- 172.21.15.16:0/3338951610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4009520 con 0x7f43b01030c0 2024-04-05T23:38:18.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43b5e00700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b01030c0 0x7f43b0199380 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f43a400eb30 tx=0x7f43a400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:18.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.045+0000 7f43aaa00700 1 -- 172.21.15.16:0/3338951610 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4009880 con 0x7f43b01030c0 2024-04-05T23:38:18.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.046+0000 7f43aaa00700 1 -- 172.21.15.16:0/3338951610 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43a40044e0 con 0x7f43b01030c0 2024-04-05T23:38:18.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.046+0000 7f43aaa00700 1 -- 172.21.15.16:0/3338951610 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4003d80 con 0x7f43b01030c0 2024-04-05T23:38:18.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.046+0000 7f43b7200700 1 -- 172.21.15.16:0/3338951610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ac009c40 con 0x7f43b01030c0 2024-04-05T23:38:18.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.046+0000 7f43b7200700 1 -- 172.21.15.16:0/3338951610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a00095f0 con 0x7f43b01030c0 2024-04-05T23:38:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.049+0000 7f43aaa00700 1 -- 172.21.15.16:0/3338951610 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f43a4003ee0 con 0x7f43b01030c0 2024-04-05T23:38:18.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.051+0000 7f43cc29fb80 1 Processor -- start 2024-04-05T23:38:18.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.051+0000 7f43cc29fb80 1 -- start start 2024-04-05T23:38:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.051+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 0x55de1f4f7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7b40 0x55de1f4fb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f45db80 con 0x55de1f4f7220 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f44e930 con 0x55de1f4fbec0 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f44c2f0 con 0x55de1f4f7b40 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43aaa00700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43980a1310 0x7f43980a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7b40 0x55de1f4fb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56790/0 (socket says 172.21.15.16:56790) 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7b40 0x55de1f4fb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44414/0 (socket says 172.21.15.16:44414) 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5400700 1 -- 172.21.15.16:0/3611856568 learned_addr learned my addr 172.21.15.16:0/3611856568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b6800700 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 0x55de1f4f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b6800700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43980a1310 0x7f43980a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43aaa00700 1 -- 172.21.15.16:0/3338951610 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f43a4014070 con 0x7f43b01030c0 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5400700 1 -- 172.21.15.16:0/3611856568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7b40 msgr2=0x55de1f4fb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5400700 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7b40 0x55de1f4fb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5400700 1 -- 172.21.15.16:0/3611856568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 msgr2=0x55de1f4f7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5400700 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 0x55de1f4f7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5400700 1 -- 172.21.15.16:0/3611856568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de1f2c9350 con 0x55de1f4fbec0 2024-04-05T23:38:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b6800700 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 0x55de1f4f7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:38:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.052+0000 7f43b5e00700 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7b40 0x55de1f4fb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.053+0000 7f43b5400700 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009990 tx=0x7f43a0005ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d12de68ff443bcae server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.053+0000 7f43b6800700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43980a1310 0x7f43980a37d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f43ac009fd0 tx=0x7f43ac00b040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.053+0000 7f439f000700 1 -- 172.21.15.16:0/3611856568 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a001d070 con 0x55de1f4fbec0 2024-04-05T23:38:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.053+0000 7f439f000700 1 -- 172.21.15.16:0/3611856568 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43a001e040 con 0x55de1f4fbec0 2024-04-05T23:38:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.053+0000 7f439f000700 1 -- 172.21.15.16:0/3611856568 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0004880 con 0x55de1f4fbec0 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3611856568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4fbec0 msgr2=0x55de1f4fc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009990 tx=0x7f43a0005ec0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3611856568 shutdown_connections 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 0x55de1f4f7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc300 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3611856568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7b40 0x55de1f4fb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3611856568 >> 172.21.15.16:0/3611856568 conn(0x55de1f46b1c0 msgr2=0x55de1f4f2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3611856568 shutdown_connections 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3611856568 wait complete. 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.054+0000 7f43cc29fb80 1 -- start start 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.055+0000 7f43cc29fb80 1 Processor -- start 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.055+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 0x55de1f503680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.055+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f500970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.055+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 0x55de1f5010b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.055+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f447be0 con 0x55de1f4f7220 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.055+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f3e3d20 con 0x55de1f4f7b40 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.055+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f3e39c0 con 0x55de1f4fbec0 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.055+0000 7f43cc29fb80 10 client.0 _ll_register_callbacks cb 0x55de1f24f340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f500970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f500970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56792/0 (socket says 172.21.15.16:56792) 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b6800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 0x55de1f5010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5400700 1 -- 172.21.15.16:0/3690714951 learned_addr learned my addr 172.21.15.16:0/3690714951 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5e00700 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 0x55de1f503680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5400700 1 -- 172.21.15.16:0/3690714951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 msgr2=0x55de1f5010b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5400700 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 0x55de1f5010b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5400700 1 -- 172.21.15.16:0/3690714951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 msgr2=0x55de1f503680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5400700 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 0x55de1f503680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5400700 1 -- 172.21.15.16:0/3690714951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a00095f0 con 0x55de1f4f7b40 2024-04-05T23:38:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f43b5400700 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f500970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009350 tx=0x7f43a0005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0616500b2da2bf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.056+0000 7f4393400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:38:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.057+0000 7f4393400700 1 -- 172.21.15.16:0/3690714951 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a001d030 con 0x55de1f4f7b40 2024-04-05T23:38:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.057+0000 7f4393400700 1 -- 172.21.15.16:0/3690714951 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f43a0004500 con 0x55de1f4f7b40 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.057+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3690714951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de1f4ea1e0 con 0x55de1f4f7b40 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.057+0000 7f4393400700 1 -- 172.21.15.16:0/3690714951 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43a00046c0 con 0x55de1f4f7b40 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.057+0000 7f43cc29fb80 10 client.26231 fetch_fsmap learned FSMap version 44 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.057+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3690714951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x7f43b01105d0 con 0x55de1f4f7b40 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.057+0000 7f4393400700 1 -- 172.21.15.16:0/3690714951 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55de1f4ea1e0 con 0x55de1f4f7b40 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.057+0000 7f4393400700 1 -- 172.21.15.16:0/3690714951 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f43a000f910 con 0x55de1f4f7b40 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.058+0000 7f43cc29fb80 10 client.26231 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.058+0000 7f43cc29fb80 10 client.26231 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.058+0000 7f43cc29fb80 20 client.26231 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.058+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1f510c60 0x55de1f513050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.058+0000 7f43cc29fb80 4 client.26231 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.058+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3690714951 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de1f4406a0 con 0x55de1f510c60 2024-04-05T23:38:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.058+0000 7f43b5e00700 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1f510c60 0x55de1f513050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.058+0000 7f43b5e00700 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1f510c60 0x55de1f513050 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ca0e80a7d41c7bc server_cookie=278bce10f8dcefc7 in_seq=0 out_seq=0 2024-04-05T23:38:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.058+0000 7f4393400700 10 client.26231 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.059+0000 7f4393400700 1 -- 172.21.15.16:0/3690714951 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55de1f4406a0 con 0x55de1f510c60 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.059+0000 7f4393400700 10 client.26231 handle_command_reply: tid=0 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.059+0000 7f43cc29fb80 1 client.26231 shutdown 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3690714951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 msgr2=0x55de1f500970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f500970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009350 tx=0x7f43a0005bc0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3690714951 shutdown_connections 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1f510c60 0x55de1f513050 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7220 0x55de1f503680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 0x55de1f5010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f4393400700 0 client.26231 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/3690714951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f500970 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3690714951 >> 172.21.15.16:0/3690714951 conn(0x55de1f46b1c0 msgr2=0x55de1f4f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:18.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3690714951 shutdown_connections 2024-04-05T23:38:18.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 1 -- 172.21.15.16:0/3690714951 wait complete. 2024-04-05T23:38:18.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.060+0000 7f43cc29fb80 20 client.26231 trim_cache size 0 max 16384 2024-04-05T23:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.075+0000 7f43cc29fb80 1 Processor -- start 2024-04-05T23:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43cc29fb80 1 -- start start 2024-04-05T23:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7220 0x55de1f501030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7b40 0x55de1f5080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f2cb640 con 0x55de1f4f7b40 2024-04-05T23:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f4f37d0 con 0x55de1f4f7220 2024-04-05T23:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f4f81d0 con 0x55de1f4fbec0 2024-04-05T23:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43b5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7220 0x55de1f501030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43b5400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7b40 0x55de1f5080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43b5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7220 0x55de1f501030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56800/0 (socket says 172.21.15.16:56800) 2024-04-05T23:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43b5e00700 1 -- 172.21.15.16:0/553882415 learned_addr learned my addr 172.21.15.16:0/553882415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43b6800700 1 --2- 172.21.15.16:0/553882415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43b5e00700 1 -- 172.21.15.16:0/553882415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 msgr2=0x55de1f4fc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43b5e00700 1 --2- 172.21.15.16:0/553882415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.076+0000 7f43b5e00700 1 -- 172.21.15.16:0/553882415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7b40 msgr2=0x55de1f5080b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.077+0000 7f43b5e00700 1 --2- 172.21.15.16:0/553882415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7b40 0x55de1f5080b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.077+0000 7f43b5e00700 1 -- 172.21.15.16:0/553882415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de1f275b00 con 0x55de1f4f7220 2024-04-05T23:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.077+0000 7f43b5400700 1 --2- 172.21.15.16:0/553882415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7b40 0x55de1f5080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.077+0000 7f43b5e00700 1 --2- 172.21.15.16:0/553882415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7220 0x55de1f501030 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f43a4096bf0 tx=0x7f43a40964c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42623c09751f404a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.077+0000 7f439fa00700 1 -- 172.21.15.16:0/553882415 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40b1040 con 0x55de1f4f7220 2024-04-05T23:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.077+0000 7f439fa00700 1 -- 172.21.15.16:0/553882415 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43a409a880 con 0x55de1f4f7220 2024-04-05T23:38:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.077+0000 7f439fa00700 1 -- 172.21.15.16:0/553882415 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a400cdd0 con 0x55de1f4f7220 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 -- 172.21.15.16:0/553882415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7220 msgr2=0x55de1f501030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/553882415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7220 0x55de1f501030 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f43a4096bf0 tx=0x7f43a40964c0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 -- 172.21.15.16:0/553882415 shutdown_connections 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/553882415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4fbec0 0x55de1f4fc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/553882415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7220 0x55de1f501030 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/553882415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4f7b40 0x55de1f5080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 -- 172.21.15.16:0/553882415 >> 172.21.15.16:0/553882415 conn(0x55de1f46b1c0 msgr2=0x55de1f4edef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 -- 172.21.15.16:0/553882415 shutdown_connections 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 -- 172.21.15.16:0/553882415 wait complete. 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.078+0000 7f43cc29fb80 1 -- start start 2024-04-05T23:38:18.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.079+0000 7f43cc29fb80 1 Processor -- start 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.079+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7220 0x55de1f4f1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f4f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43cc29fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4fbec0 0x55de1f563a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f563fa0 con 0x55de1f4fbec0 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f5640e0 con 0x55de1f4f7b40 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43cc29fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f564250 con 0x55de1f4f7220 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43cc29fb80 10 client.0 _ll_register_callbacks cb 0x55de1f24f340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f4f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7220 0x55de1f4f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f4f1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56802/0 (socket says 172.21.15.16:56802) 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5400700 1 -- 172.21.15.16:0/1763876331 learned_addr learned my addr 172.21.15.16:0/1763876331 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b6800700 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4fbec0 0x55de1f563a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5400700 1 -- 172.21.15.16:0/1763876331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7220 msgr2=0x55de1f4f1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5400700 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7220 0x55de1f4f1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5400700 1 -- 172.21.15.16:0/1763876331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4fbec0 msgr2=0x55de1f563a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5400700 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4fbec0 0x55de1f563a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5400700 1 -- 172.21.15.16:0/1763876331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b01105d0 con 0x55de1f4f7b40 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b6800700 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4fbec0 0x55de1f563a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5e00700 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7220 0x55de1f4f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f43b5400700 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f4f1bc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f43a0001eb0 tx=0x7f43a0010fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37ff7a376b647748 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.080+0000 7f4393400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.081+0000 7f4393400700 1 -- 172.21.15.16:0/1763876331 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a001d040 con 0x55de1f4f7b40 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.081+0000 7f43cc29fb80 1 -- 172.21.15.16:0/1763876331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43a001e040 con 0x55de1f4f7b40 2024-04-05T23:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.081+0000 7f4393400700 1 -- 172.21.15.16:0/1763876331 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f43a000bc30 con 0x55de1f4f7b40 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.081+0000 7f4393400700 1 -- 172.21.15.16:0/1763876331 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43a000f660 con 0x55de1f4f7b40 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.081+0000 7f43cc29fb80 10 client.26237 fetch_fsmap learned FSMap version 44 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.081+0000 7f43cc29fb80 1 -- 172.21.15.16:0/1763876331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x7f43ac009c40 con 0x55de1f4f7b40 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.081+0000 7f4393400700 1 -- 172.21.15.16:0/1763876331 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43a004e080 con 0x55de1f4f7b40 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.081+0000 7f4393400700 1 -- 172.21.15.16:0/1763876331 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f43a0004500 con 0x55de1f4f7b40 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.082+0000 7f43cc29fb80 10 client.26237 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.082+0000 7f43cc29fb80 10 client.26237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.082+0000 7f43cc29fb80 20 client.26237 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.082+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1f565510 0x55de1f567930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.082+0000 7f43cc29fb80 4 client.26237 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.082+0000 7f43cc29fb80 1 -- 172.21.15.16:0/1763876331 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55de1f296c90 con 0x55de1f565510 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.082+0000 7f43b5e00700 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1f565510 0x55de1f567930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.082+0000 7f43b5e00700 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1f565510 0x55de1f567930 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e67bbf28729ab9c3 server_cookie=60b55f1444166339 in_seq=0 out_seq=0 2024-04-05T23:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.082+0000 7f4393400700 10 client.26237 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:18.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:17 smithi016 ceph-mon[38248]: pgmap v146: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 3.2 MiB/s wr, 290 op/s 2024-04-05T23:38:18.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:17 smithi130 ceph-mon[79417]: pgmap v146: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 3.2 MiB/s wr, 290 op/s 2024-04-05T23:38:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f4393400700 1 -- 172.21.15.16:0/1763876331 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55de1f296c90 con 0x55de1f565510 2024-04-05T23:38:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f4393400700 10 client.26237 handle_command_reply: tid=0 2024-04-05T23:38:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f43cc29fb80 1 client.26237 shutdown 2024-04-05T23:38:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f43cc29fb80 1 -- 172.21.15.16:0/1763876331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 msgr2=0x55de1f4f1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f4f1bc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f43a0001eb0 tx=0x7f43a0010fc0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f43cc29fb80 1 -- 172.21.15.16:0/1763876331 shutdown_connections 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55de1f565510 0x55de1f567930 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de1f4fbec0 0x55de1f563a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de1f4f7220 0x55de1f4f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f43cc29fb80 1 --2- 172.21.15.16:0/1763876331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55de1f4f7b40 0x55de1f4f1bc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.121+0000 7f43cc29fb80 1 -- 172.21.15.16:0/1763876331 >> 172.21.15.16:0/1763876331 conn(0x55de1f46b1c0 msgr2=0x55de1f4fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.122+0000 7f4393400700 0 client.26237 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.122+0000 7f43cc29fb80 1 -- 172.21.15.16:0/1763876331 shutdown_connections 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.122+0000 7f43cc29fb80 1 -- 172.21.15.16:0/1763876331 wait complete. 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.122+0000 7f43cc29fb80 20 client.26237 trim_cache size 0 max 16384 2024-04-05T23:38:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:38:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:38:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:38:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:38:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.123+0000 7f43b7200700 1 -- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43980a1310 msgr2=0x7f43980a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.123+0000 7f43b7200700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43980a1310 0x7f43980a37d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f43ac009fd0 tx=0x7f43ac00b040 comp rx=0 tx=0).stop 2024-04-05T23:38:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.123+0000 7f43b7200700 1 -- 172.21.15.16:0/3338951610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b01030c0 msgr2=0x7f43b0199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.123+0000 7f43b7200700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b01030c0 0x7f43b0199380 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f43a400eb30 tx=0x7f43a400ee40 comp rx=0 tx=0).stop 2024-04-05T23:38:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.124+0000 7f43b7200700 1 -- 172.21.15.16:0/3338951610 shutdown_connections 2024-04-05T23:38:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.124+0000 7f43b7200700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43980a1310 0x7f43980a37d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.124+0000 7f43b7200700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43b019dc10 0x7f43b019e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.124+0000 7f43b7200700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b01030c0 0x7f43b0199380 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.124+0000 7f43b7200700 1 --2- 172.21.15.16:0/3338951610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b01081e0 0x7f43b01998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:18.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.124+0000 7f43b7200700 1 -- 172.21.15.16:0/3338951610 >> 172.21.15.16:0/3338951610 conn(0x7f43b00fe8e0 msgr2=0x7f43b00ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:18.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.124+0000 7f43b7200700 1 -- 172.21.15.16:0/3338951610 shutdown_connections 2024-04-05T23:38:18.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:18.124+0000 7f43b7200700 1 -- 172.21.15.16:0/3338951610 wait complete. 2024-04-05T23:38:18.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:17 smithi027 ceph-mon[78597]: pgmap v146: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 3.2 MiB/s wr, 290 op/s 2024-04-05T23:38:18.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 280467 8.82 MB/sec execute 98 sec latency 33.652 ms 2024-04-05T23:38:19.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 281748 8.79 MB/sec execute 99 sec latency 13.171 ms 2024-04-05T23:38:20.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:19 smithi016 ceph-mon[38248]: pgmap v147: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 1.8 MiB/s wr, 119 op/s 2024-04-05T23:38:20.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:19 smithi130 ceph-mon[79417]: pgmap v147: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 1.8 MiB/s wr, 119 op/s 2024-04-05T23:38:20.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:19 smithi027 ceph-mon[78597]: pgmap v147: 97 pgs: 97 active+clean; 103 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 1.8 MiB/s wr, 119 op/s 2024-04-05T23:38:20.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 283892 8.83 MB/sec execute 100 sec latency 49.305 ms 2024-04-05T23:38:21.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 284968 8.78 MB/sec execute 101 sec latency 45.022 ms 2024-04-05T23:38:22.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:21 smithi016 ceph-mon[38248]: pgmap v148: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 247 op/s 2024-04-05T23:38:22.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:21 smithi130 ceph-mon[79417]: pgmap v148: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 247 op/s 2024-04-05T23:38:22.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:21 smithi027 ceph-mon[78597]: pgmap v148: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 247 op/s 2024-04-05T23:38:22.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287079 8.81 MB/sec execute 102 sec latency 17.715 ms 2024-04-05T23:38:23.996 INFO:tasks.workunit.client.0.smithi016.stdout: 1 288276 8.78 MB/sec execute 103 sec latency 12.376 ms 2024-04-05T23:38:24.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:23 smithi016 ceph-mon[38248]: pgmap v149: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-04-05T23:38:24.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:23 smithi130 ceph-mon[79417]: pgmap v149: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-04-05T23:38:24.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:23 smithi027 ceph-mon[78597]: pgmap v149: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-04-05T23:38:24.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 290264 8.81 MB/sec execute 104 sec latency 19.002 ms 2024-04-05T23:38:25.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 291487 8.79 MB/sec execute 105 sec latency 11.436 ms 2024-04-05T23:38:26.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:25 smithi016 ceph-mon[38248]: pgmap v150: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-04-05T23:38:26.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:25 smithi130 ceph-mon[79417]: pgmap v150: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-04-05T23:38:26.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:25 smithi027 ceph-mon[78597]: pgmap v150: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-04-05T23:38:26.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 293312 8.77 MB/sec execute 106 sec latency 54.742 ms 2024-04-05T23:38:27.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 294845 8.80 MB/sec execute 107 sec latency 17.671 ms 2024-04-05T23:38:28.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:27 smithi016 ceph-mon[38248]: pgmap v151: 97 pgs: 97 active+clean; 110 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 280 op/s 2024-04-05T23:38:28.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:27 smithi130 ceph-mon[79417]: pgmap v151: 97 pgs: 97 active+clean; 110 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 280 op/s 2024-04-05T23:38:28.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:27 smithi027 ceph-mon[78597]: pgmap v151: 97 pgs: 97 active+clean; 110 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 280 op/s 2024-04-05T23:38:28.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 296046 8.77 MB/sec execute 108 sec latency 23.820 ms 2024-04-05T23:38:29.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 297824 8.77 MB/sec execute 109 sec latency 13.574 ms 2024-04-05T23:38:30.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:29 smithi016 ceph-mon[38248]: pgmap v152: 97 pgs: 97 active+clean; 110 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-04-05T23:38:30.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:29 smithi130 ceph-mon[79417]: pgmap v152: 97 pgs: 97 active+clean; 110 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-04-05T23:38:30.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:29 smithi027 ceph-mon[78597]: pgmap v152: 97 pgs: 97 active+clean; 110 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-04-05T23:38:30.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 299020 8.75 MB/sec execute 110 sec latency 11.539 ms 2024-04-05T23:38:31.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.286+0000 7f6209c00700 1 -- 172.21.15.16:0/2576441060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6204111dd0 msgr2=0x7f62041141c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:31.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.286+0000 7f6209c00700 1 --2- 172.21.15.16:0/2576441060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6204111dd0 0x7f62041141c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00b0a0 tx=0x7f61fc00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.287+0000 7f6209c00700 1 -- 172.21.15.16:0/2576441060 shutdown_connections 2024-04-05T23:38:31.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.287+0000 7f6209c00700 1 --2- 172.21.15.16:0/2576441060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6204111dd0 0x7f62041141c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.287+0000 7f6209c00700 1 --2- 172.21.15.16:0/2576441060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6204073280 0x7f6204073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.287+0000 7f6209c00700 1 --2- 172.21.15.16:0/2576441060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6204073ba0 0x7f6204111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.287+0000 7f6209c00700 1 -- 172.21.15.16:0/2576441060 >> 172.21.15.16:0/2576441060 conn(0x7f62040fc8b0 msgr2=0x7f62040fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.287+0000 7f6209c00700 1 -- 172.21.15.16:0/2576441060 shutdown_connections 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.287+0000 7f6209c00700 1 -- 172.21.15.16:0/2576441060 wait complete. 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6209c00700 1 Processor -- start 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6209c00700 1 -- start start 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6209c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6204073280 0x7f62041a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6209c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6204073ba0 0x7f62041a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6209c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6204111dd0 0x7f620419d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6209c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620419deb0 con 0x7f6204073ba0 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6209c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620419e020 con 0x7f6204111dd0 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6209c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620419e1e0 con 0x7f6204073280 2024-04-05T23:38:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6203400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6204111dd0 0x7f620419d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6202000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6204073ba0 0x7f62041a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.288+0000 7f6202a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6204073280 0x7f62041a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6203400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6204111dd0 0x7f620419d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52514/0 (socket says 172.21.15.16:52514) 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6203400700 1 -- 172.21.15.16:0/2080000447 learned_addr learned my addr 172.21.15.16:0/2080000447 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6203400700 1 -- 172.21.15.16:0/2080000447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6204073280 msgr2=0x7f62041a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6203400700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6204073280 0x7f62041a3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6203400700 1 -- 172.21.15.16:0/2080000447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6204073ba0 msgr2=0x7f62041a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6203400700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6204073ba0 0x7f62041a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6203400700 1 -- 172.21.15.16:0/2080000447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc009d00 con 0x7f6204111dd0 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6202a00700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6204073280 0x7f62041a3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6203400700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6204111dd0 0x7f620419d840 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f61fc006070 tx=0x7f61fc0078c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f61fb400700 1 -- 172.21.15.16:0/2080000447 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc00e040 con 0x7f6204111dd0 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f61fb400700 1 -- 172.21.15.16:0/2080000447 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61fc01d470 con 0x7f6204111dd0 2024-04-05T23:38:31.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f6209c00700 1 -- 172.21.15.16:0/2080000447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620419e460 con 0x7f6204111dd0 2024-04-05T23:38:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.289+0000 7f61fb400700 1 -- 172.21.15.16:0/2080000447 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc01b6c0 con 0x7f6204111dd0 2024-04-05T23:38:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.290+0000 7f6209c00700 1 -- 172.21.15.16:0/2080000447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62041aa700 con 0x7f6204111dd0 2024-04-05T23:38:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.293+0000 7f61fb400700 1 -- 172.21.15.16:0/2080000447 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f61fc01d5e0 con 0x7f6204111dd0 2024-04-05T23:38:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.294+0000 7f621d6aeb80 1 Processor -- start 2024-04-05T23:38:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.294+0000 7f621d6aeb80 1 -- start start 2024-04-05T23:38:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.294+0000 7f61fb400700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e80a1290 0x7f61e80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f61fb400700 1 -- 172.21.15.16:0/2080000447 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f61fc0cdd10 con 0x7f6204111dd0 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 0x557db4aa6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6202000700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e80a1290 0x7f61e80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa74a0 0x557db4aa1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4aa1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db4a07b80 con 0x557db4aa74a0 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db49f8930 con 0x557db4aa15c0 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db49f62f0 con 0x557db4aa6b80 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6203400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4aa1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6203400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4aa1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52526/0 (socket says 172.21.15.16:52526) 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6203400700 1 -- 172.21.15.16:0/1996252070 learned_addr learned my addr 172.21.15.16:0/1996252070 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6202000700 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa74a0 0x557db4aa1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6202a00700 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 0x557db4aa6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6203400700 1 -- 172.21.15.16:0/1996252070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 msgr2=0x557db4aa6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6203400700 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 0x557db4aa6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6203400700 1 -- 172.21.15.16:0/1996252070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa74a0 msgr2=0x557db4aa1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6203400700 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa74a0 0x557db4aa1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6203400700 1 -- 172.21.15.16:0/1996252070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557db4873350 con 0x557db4aa15c0 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.295+0000 7f6202a00700 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 0x557db4aa6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.296+0000 7f6203400700 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4aa1a20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0a2d40 tx=0x7f61fc00b740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d1a160322d04225 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.296+0000 7f61efe00700 1 -- 172.21.15.16:0/1996252070 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc011e70 con 0x557db4aa15c0 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.296+0000 7f6202000700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e80a1290 0x7f61e80a3750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f620419ec60 tx=0x7f61f0009ac0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.296+0000 7f61efe00700 1 -- 172.21.15.16:0/1996252070 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61fc09be90 con 0x557db4aa15c0 2024-04-05T23:38:31.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.296+0000 7f61efe00700 1 -- 172.21.15.16:0/1996252070 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc01d8c0 con 0x557db4aa15c0 2024-04-05T23:38:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1996252070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 msgr2=0x557db4aa1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4aa1a20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0a2d40 tx=0x7f61fc00b740 comp rx=0 tx=0).stop 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1996252070 shutdown_connections 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4aa1a20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 0x557db4aa6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1996252070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa74a0 0x557db4aa1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1996252070 >> 172.21.15.16:0/1996252070 conn(0x557db4a151c0 msgr2=0x557db4a9bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1996252070 shutdown_connections 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1996252070 wait complete. 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 -- start start 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.297+0000 7f621d6aeb80 1 Processor -- start 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa15c0 0x557db4aad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 0x557db4aaa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa74a0 0x557db4aaaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db49f1be0 con 0x557db4aa15c0 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db498dd20 con 0x557db4aa74a0 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db498d9c0 con 0x557db4aa6b80 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f621d6aeb80 10 client.0 _ll_register_callbacks cb 0x557db47f9340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f6203400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa74a0 0x557db4aaaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f6202a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa15c0 0x557db4aad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f6203400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa74a0 0x557db4aaaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52540/0 (socket says 172.21.15.16:52540) 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f6203400700 1 -- 172.21.15.16:0/1681165287 learned_addr learned my addr 172.21.15.16:0/1681165287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.298+0000 7f6202000700 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 0x557db4aaa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f6203400700 1 -- 172.21.15.16:0/1681165287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 msgr2=0x557db4aaa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f6203400700 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 0x557db4aaa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f6203400700 1 -- 172.21.15.16:0/1681165287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa15c0 msgr2=0x557db4aad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f6203400700 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa15c0 0x557db4aad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f6203400700 1 -- 172.21.15.16:0/1681165287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62041aa700 con 0x557db4aa74a0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f6203400700 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa74a0 0x557db4aaaff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f61fc09ab80 tx=0x7f61fc0cb420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2f014b13400ed4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f61ecc00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f61ecc00700 1 -- 172.21.15.16:0/1681165287 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc09b2e0 con 0x557db4aa74a0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1681165287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557db4a941e0 con 0x557db4aa74a0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.299+0000 7f61ecc00700 1 -- 172.21.15.16:0/1681165287 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f61fc09b580 con 0x557db4aa74a0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f61ecc00700 1 -- 172.21.15.16:0/1681165287 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61fc01dad0 con 0x557db4aa74a0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f621d6aeb80 10 client.26249 fetch_fsmap learned FSMap version 44 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1681165287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x557db481fb00 con 0x557db4aa74a0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f61ecc00700 1 -- 172.21.15.16:0/1681165287 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61fc064020 con 0x557db4aa74a0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f61ecc00700 1 -- 172.21.15.16:0/1681165287 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f61fc09fbd0 con 0x557db4aa74a0 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f621d6aeb80 10 client.26249 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f621d6aeb80 10 client.26249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f621d6aeb80 20 client.26249 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557db4aba8f0 0x557db4abcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f621d6aeb80 4 client.26249 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:38:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.300+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1681165287 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557db49ea6a0 con 0x557db4aba8f0 2024-04-05T23:38:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.301+0000 7f6202a00700 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557db4aba8f0 0x557db4abcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.301+0000 7f6202a00700 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557db4aba8f0 0x557db4abcce0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9edbfcf67e0bb13f server_cookie=47c905471dd02bc1 in_seq=0 out_seq=0 2024-04-05T23:38:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.301+0000 7f61ecc00700 10 client.26249 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.302+0000 7f61ecc00700 1 -- 172.21.15.16:0/1681165287 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557db49ea6a0 con 0x557db4aba8f0 2024-04-05T23:38:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.302+0000 7f61ecc00700 10 client.26249 handle_command_reply: tid=0 2024-04-05T23:38:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.302+0000 7f621d6aeb80 1 client.26249 shutdown 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1681165287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa74a0 msgr2=0x557db4aaaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa74a0 0x557db4aaaff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f61fc09ab80 tx=0x7f61fc0cb420 comp rx=0 tx=0).stop 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1681165287 shutdown_connections 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557db4aba8f0 0x557db4abcce0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa15c0 0x557db4aad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa74a0 0x557db4aaaff0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f61ecc00700 0 client.26249 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/1681165287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa6b80 0x557db4aaa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1681165287 >> 172.21.15.16:0/1681165287 conn(0x557db4a151c0 msgr2=0x557db4a9b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1681165287 shutdown_connections 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 1 -- 172.21.15.16:0/1681165287 wait complete. 2024-04-05T23:38:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.303+0000 7f621d6aeb80 20 client.26249 trim_cache size 0 max 16384 2024-04-05T23:38:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.319+0000 7f621d6aeb80 1 Processor -- start 2024-04-05T23:38:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.319+0000 7f621d6aeb80 1 -- start start 2024-04-05T23:38:31.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4a151c0 0x557db4aa1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4ab1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa6b80 0x557db4a9ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db4875640 con 0x557db4aa6b80 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db49ea6a0 con 0x557db4aa15c0 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db4ab7920 con 0x557db4a151c0 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f6202a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4a151c0 0x557db4aa1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f6202000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4ab1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f6203400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa6b80 0x557db4a9ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f6202a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4a151c0 0x557db4aa1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:50992/0 (socket says 172.21.15.16:50992) 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f6202a00700 1 -- 172.21.15.16:0/3320638086 learned_addr learned my addr 172.21.15.16:0/3320638086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:31.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f6202000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4ab1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52548/0 (socket says 172.21.15.16:52548) 2024-04-05T23:38:31.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f6202a00700 1 -- 172.21.15.16:0/3320638086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 msgr2=0x557db4ab1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:31.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.320+0000 7f6202a00700 1 --2- 172.21.15.16:0/3320638086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4ab1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.321+0000 7f6202a00700 1 -- 172.21.15.16:0/3320638086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa6b80 msgr2=0x557db4a9ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:31.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.321+0000 7f6202a00700 1 --2- 172.21.15.16:0/3320638086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa6b80 0x557db4a9ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.321+0000 7f6202a00700 1 -- 172.21.15.16:0/3320638086 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557db49b7e40 con 0x557db4a151c0 2024-04-05T23:38:31.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.321+0000 7f6202a00700 1 --2- 172.21.15.16:0/3320638086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4a151c0 0x557db4aa1030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x557db4abd5c0 tx=0x7f61f4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59ef287ae0da0767 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:31.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.321+0000 7f61f8c00700 1 -- 172.21.15.16:0/3320638086 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4017070 con 0x557db4a151c0 2024-04-05T23:38:31.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.321+0000 7f61f8c00700 1 -- 172.21.15.16:0/3320638086 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61f4018040 con 0x557db4a151c0 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.321+0000 7f61f8c00700 1 -- 172.21.15.16:0/3320638086 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f401f360 con 0x557db4a151c0 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 -- 172.21.15.16:0/3320638086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4a151c0 msgr2=0x557db4aa1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/3320638086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4a151c0 0x557db4aa1030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x557db4abd5c0 tx=0x7f61f4009260 comp rx=0 tx=0).stop 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 -- 172.21.15.16:0/3320638086 shutdown_connections 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/3320638086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4aa6b80 0x557db4a9ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/3320638086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4a151c0 0x557db4aa1030 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/3320638086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa15c0 0x557db4ab1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 -- 172.21.15.16:0/3320638086 >> 172.21.15.16:0/3320638086 conn(0x557db4aa1cb0 msgr2=0x557db4aa4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 -- 172.21.15.16:0/3320638086 shutdown_connections 2024-04-05T23:38:31.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 -- 172.21.15.16:0/3320638086 wait complete. 2024-04-05T23:38:31.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.322+0000 7f621d6aeb80 1 -- start start 2024-04-05T23:38:31.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.323+0000 7f621d6aeb80 1 Processor -- start 2024-04-05T23:38:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.323+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4a151c0 0x557db4ab1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.323+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa15c0 0x557db4ab0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.323+0000 7f621d6aeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa6b80 0x557db4ab08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.323+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db4ab1000 con 0x557db4a151c0 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.323+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db4ab1170 con 0x557db4aa6b80 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.323+0000 7f621d6aeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557db4b0e6d0 con 0x557db4aa15c0 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.323+0000 7f621d6aeb80 10 client.0 _ll_register_callbacks cb 0x7f61f00134e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.323+0000 7f6202a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4a151c0 0x557db4ab1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6203400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa6b80 0x557db4ab08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa15c0 0x557db4ab0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4a151c0 0x557db4ab1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57126/0 (socket says 172.21.15.16:57126) 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa15c0 0x557db4ab0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:51000/0 (socket says 172.21.15.16:51000) 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6203400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa6b80 0x557db4ab08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52554/0 (socket says 172.21.15.16:52554) 2024-04-05T23:38:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202a00700 1 -- 172.21.15.16:0/2907516367 learned_addr learned my addr 172.21.15.16:0/2907516367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202000700 1 -- 172.21.15.16:0/2907516367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa6b80 msgr2=0x557db4ab08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202000700 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa6b80 0x557db4ab08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202000700 1 -- 172.21.15.16:0/2907516367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4a151c0 msgr2=0x557db4ab1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202000700 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4a151c0 0x557db4ab1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202000700 1 -- 172.21.15.16:0/2907516367 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f40040f0 con 0x557db4aa15c0 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6203400700 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa6b80 0x557db4ab08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202a00700 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4a151c0 0x557db4ab1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f6202000700 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa15c0 0x557db4ab0350 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x557db4aabdf0 tx=0x7f61f00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccb376b681a316b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.324+0000 7f61ecc00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.325+0000 7f61ecc00700 1 -- 172.21.15.16:0/2907516367 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0011b20 con 0x557db4aa15c0 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.325+0000 7f61ecc00700 1 -- 172.21.15.16:0/2907516367 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f61f0011c80 con 0x557db4aa15c0 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.325+0000 7f621d6aeb80 1 -- 172.21.15.16:0/2907516367 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61fc09be90 con 0x557db4aa15c0 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.325+0000 7f61ecc00700 1 -- 172.21.15.16:0/2907516367 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61f0011e40 con 0x557db4aa15c0 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.325+0000 7f621d6aeb80 10 client.26290 fetch_fsmap learned FSMap version 44 2024-04-05T23:38:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.325+0000 7f621d6aeb80 1 -- 172.21.15.16:0/2907516367 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x557db4877c70 con 0x557db4aa15c0 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.325+0000 7f61ecc00700 1 -- 172.21.15.16:0/2907516367 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61fc09be90 con 0x557db4aa15c0 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.326+0000 7f61ecc00700 1 -- 172.21.15.16:0/2907516367 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f61f00127f0 con 0x557db4aa15c0 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.326+0000 7f621d6aeb80 10 client.26290 fetch_fsmap finished waiting for FSMap version 44 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.326+0000 7f621d6aeb80 10 client.26290 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.326+0000 7f621d6aeb80 20 client.26290 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.326+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557db4aa74a0 0x557db4b11ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.326+0000 7f621d6aeb80 4 client.26290 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.326+0000 7f621d6aeb80 1 -- 172.21.15.16:0/2907516367 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557db4840c90 con 0x557db4aa74a0 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.326+0000 7f6202a00700 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557db4aa74a0 0x557db4b11ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.327+0000 7f61ecc00700 10 client.26290 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.327+0000 7f6202a00700 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557db4aa74a0 0x557db4b11ea0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1df1357016126463 server_cookie=865f4d291c0a9e3a in_seq=0 out_seq=0 2024-04-05T23:38:31.367 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f61ecc00700 1 -- 172.21.15.16:0/2907516367 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557db4840c90 con 0x557db4aa74a0 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f61ecc00700 10 client.26290 handle_command_reply: tid=0 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 client.26290 shutdown 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 -- 172.21.15.16:0/2907516367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa15c0 msgr2=0x557db4ab0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa15c0 0x557db4ab0350 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x557db4aabdf0 tx=0x7f61f00047c0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 -- 172.21.15.16:0/2907516367 shutdown_connections 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557db4aa74a0 0x557db4b11ea0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557db4a151c0 0x557db4ab1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557db4aa6b80 0x557db4ab08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 --2- 172.21.15.16:0/2907516367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557db4aa15c0 0x557db4ab0350 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f61ecc00700 0 client.26290 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 -- 172.21.15.16:0/2907516367 >> 172.21.15.16:0/2907516367 conn(0x557db4aa1cb0 msgr2=0x557db4aa59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 -- 172.21.15.16:0/2907516367 shutdown_connections 2024-04-05T23:38:31.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 1 -- 172.21.15.16:0/2907516367 wait complete. 2024-04-05T23:38:31.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f621d6aeb80 20 client.26290 trim_cache size 0 max 16384 2024-04-05T23:38:31.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:38:31.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:38:31.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:38:31.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:38:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 -- 172.21.15.16:0/2080000447 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e80a1290 msgr2=0x7f61e80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e80a1290 0x7f61e80a3750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f620419ec60 tx=0x7f61f0009ac0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 -- 172.21.15.16:0/2080000447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6204111dd0 msgr2=0x7f620419d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6204111dd0 0x7f620419d840 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f61fc006070 tx=0x7f61fc0078c0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 -- 172.21.15.16:0/2080000447 shutdown_connections 2024-04-05T23:38:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f61e80a1290 0x7f61e80a3750 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6204111dd0 0x7f620419d840 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6204073280 0x7f62041a3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 --2- 172.21.15.16:0/2080000447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6204073ba0 0x7f62041a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 -- 172.21.15.16:0/2080000447 >> 172.21.15.16:0/2080000447 conn(0x7f62040fc8b0 msgr2=0x7f62040fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 -- 172.21.15.16:0/2080000447 shutdown_connections 2024-04-05T23:38:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:31.376+0000 7f6209c00700 1 -- 172.21.15.16:0/2080000447 wait complete. 2024-04-05T23:38:31.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 300947 8.77 MB/sec execute 111 sec latency 41.254 ms 2024-04-05T23:38:32.009 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:31 smithi130 ceph-mon[79417]: pgmap v153: 97 pgs: 97 active+clean; 114 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 341 op/s 2024-04-05T23:38:32.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:31 smithi016 ceph-mon[38248]: pgmap v153: 97 pgs: 97 active+clean; 114 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 341 op/s 2024-04-05T23:38:32.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:31 smithi027 ceph-mon[78597]: pgmap v153: 97 pgs: 97 active+clean; 114 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 341 op/s 2024-04-05T23:38:32.333 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:38:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: ignoring --setuser ceph since I am not root 2024-04-05T23:38:32.333 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:38:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: ignoring --setgroup ceph since I am not root 2024-04-05T23:38:32.333 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:38:32 smithi130 ceph-mds[121021]: -- 172.21.15.130:0/3002769949 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cf227161a0 con 0x55cf226ecc00 2024-04-05T23:38:32.333 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:38:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: starting mds.f at 2024-04-05T23:38:32.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 302233 8.75 MB/sec execute 112 sec latency 16.006 ms 2024-04-05T23:38:33.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:32 smithi016 ceph-mon[38248]: daemon mds.f restarted 2024-04-05T23:38:33.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:32 smithi016 ceph-mon[38248]: osdmap e85: 12 total, 12 up, 12 in 2024-04-05T23:38:33.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:32 smithi130 ceph-mon[79417]: daemon mds.f restarted 2024-04-05T23:38:33.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:32 smithi130 ceph-mon[79417]: osdmap e85: 12 total, 12 up, 12 in 2024-04-05T23:38:33.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:32 smithi027 ceph-mon[78597]: daemon mds.f restarted 2024-04-05T23:38:33.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:32 smithi027 ceph-mon[78597]: osdmap e85: 12 total, 12 up, 12 in 2024-04-05T23:38:33.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 303214 8.72 MB/sec execute 113 sec latency 450.496 ms 2024-04-05T23:38:34.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:33 smithi016 ceph-mon[38248]: pgmap v155: 97 pgs: 97 active+clean; 114 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 257 op/s 2024-04-05T23:38:34.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:33 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6834/2605400145,v1:172.21.15.130:6835/2605400145] up:boot 2024-04-05T23:38:34.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:33 smithi016 ceph-mon[38248]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-05T23:38:34.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:33 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:38:34.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:33 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:38:34.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:33 smithi130 ceph-mon[79417]: pgmap v155: 97 pgs: 97 active+clean; 114 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 257 op/s 2024-04-05T23:38:34.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:33 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6834/2605400145,v1:172.21.15.130:6835/2605400145] up:boot 2024-04-05T23:38:34.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:33 smithi130 ceph-mon[79417]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-05T23:38:34.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:33 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:38:34.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:33 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:38:34.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:33 smithi027 ceph-mon[78597]: pgmap v155: 97 pgs: 97 active+clean; 114 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 257 op/s 2024-04-05T23:38:34.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:33 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6834/2605400145,v1:172.21.15.130:6835/2605400145] up:boot 2024-04-05T23:38:34.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:33 smithi027 ceph-mon[78597]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-05T23:38:34.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:33 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:38:34.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:33 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:38:34.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 303214 8.64 MB/sec execute 114 sec latency 1450.569 ms 2024-04-05T23:38:35.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 303214 8.57 MB/sec execute 115 sec latency 2450.680 ms 2024-04-05T23:38:36.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:35 smithi016 ceph-mon[38248]: pgmap v156: 97 pgs: 97 active+clean; 114 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 257 op/s 2024-04-05T23:38:36.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:35 smithi130 ceph-mon[79417]: pgmap v156: 97 pgs: 97 active+clean; 114 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 257 op/s 2024-04-05T23:38:36.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:35 smithi027 ceph-mon[78597]: pgmap v156: 97 pgs: 97 active+clean; 114 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 257 op/s 2024-04-05T23:38:36.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 303930 8.51 MB/sec execute 116 sec latency 3168.416 ms 2024-04-05T23:38:37.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 305479 8.54 MB/sec execute 117 sec latency 11.830 ms 2024-04-05T23:38:38.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:37 smithi016 ceph-mon[38248]: pgmap v157: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2024-04-05T23:38:38.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:37 smithi130 ceph-mon[79417]: pgmap v157: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2024-04-05T23:38:38.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:37 smithi027 ceph-mon[78597]: pgmap v157: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2024-04-05T23:38:38.997 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306658 8.51 MB/sec execute 118 sec latency 20.744 ms 2024-04-05T23:38:39.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:38 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:38:39.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:38 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:38:39.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:38 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:38:39.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306757 8.44 MB/sec execute 119 sec latency 882.769 ms 2024-04-05T23:38:40.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:39 smithi016 ceph-mon[38248]: pgmap v158: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2024-04-05T23:38:40.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:39 smithi130 ceph-mon[79417]: pgmap v158: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2024-04-05T23:38:40.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:39 smithi027 ceph-mon[78597]: pgmap v158: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2024-04-05T23:38:40.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306757 8.37 MB/sec execute 120 sec latency 1882.843 ms 2024-04-05T23:38:41.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 307426 8.31 MB/sec execute 121 sec latency 2614.554 ms 2024-04-05T23:38:42.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:41 smithi016 ceph-mon[38248]: pgmap v159: 97 pgs: 97 active+clean; 112 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-04-05T23:38:42.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:41 smithi130 ceph-mon[79417]: pgmap v159: 97 pgs: 97 active+clean; 112 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-04-05T23:38:42.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:41 smithi027 ceph-mon[78597]: pgmap v159: 97 pgs: 97 active+clean; 112 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-04-05T23:38:42.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 308992 8.34 MB/sec execute 122 sec latency 12.587 ms 2024-04-05T23:38:43.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 310121 8.32 MB/sec execute 123 sec latency 17.020 ms 2024-04-05T23:38:44.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:43 smithi016 ceph-mon[38248]: pgmap v160: 97 pgs: 97 active+clean; 112 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 195 op/s 2024-04-05T23:38:44.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:43 smithi130 ceph-mon[79417]: pgmap v160: 97 pgs: 97 active+clean; 112 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 195 op/s 2024-04-05T23:38:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:43 smithi027 ceph-mon[78597]: pgmap v160: 97 pgs: 97 active+clean; 112 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 195 op/s 2024-04-05T23:38:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.526+0000 7f2fe1c00700 1 -- 172.21.15.16:0/120978028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fdc101030 msgr2=0x7f2fdc101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.526+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/120978028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fdc101030 0x7f2fdc101410 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009990 tx=0x7f2fcc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.526+0000 7f2fe1c00700 1 -- 172.21.15.16:0/120978028 shutdown_connections 2024-04-05T23:38:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.526+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/120978028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fdc101030 0x7f2fdc101410 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.526+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/120978028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fdc106150 0x7f2fdc10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.526+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/120978028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fdc1019e0 0x7f2fdc105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.526+0000 7f2fe1c00700 1 -- 172.21.15.16:0/120978028 >> 172.21.15.16:0/120978028 conn(0x7f2fdc0fc8b0 msgr2=0x7f2fdc0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.526+0000 7f2fe1c00700 1 -- 172.21.15.16:0/120978028 shutdown_connections 2024-04-05T23:38:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.526+0000 7f2fe1c00700 1 -- 172.21.15.16:0/120978028 wait complete. 2024-04-05T23:38:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.527+0000 7f2fe1c00700 1 Processor -- start 2024-04-05T23:38:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.527+0000 7f2fe1c00700 1 -- start start 2024-04-05T23:38:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.527+0000 7f2fe1c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fdc101030 0x7f2fdc1994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.527+0000 7f2fe1c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fdc106150 0x7f2fdc1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.527+0000 7f2fe1c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fdc19dd90 0x7f2fdc19e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.527+0000 7f2fe1c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc19e810 con 0x7f2fdc19dd90 2024-04-05T23:38:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.527+0000 7f2fe1c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc19e980 con 0x7f2fdc101030 2024-04-05T23:38:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.527+0000 7f2fe1c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc19eb40 con 0x7f2fdc106150 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fdc101030 0x7f2fdc1994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fdc19dd90 0x7f2fdc19e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fdc101030 0x7f2fdc1994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47198/0 (socket says 172.21.15.16:47198) 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdaa00700 1 -- 172.21.15.16:0/2153046841 learned_addr learned my addr 172.21.15.16:0/2153046841 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fda000700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fdc106150 0x7f2fdc1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdaa00700 1 -- 172.21.15.16:0/2153046841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fdc106150 msgr2=0x7f2fdc1999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fdc106150 0x7f2fdc1999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdaa00700 1 -- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fdc19dd90 msgr2=0x7f2fdc19e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fdc19dd90 0x7f2fdc19e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdaa00700 1 -- 172.21.15.16:0/2153046841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc0095f0 con 0x7f2fdc101030 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdb400700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fdc19dd90 0x7f2fdc19e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.528+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fdc101030 0x7f2fdc1994b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc005910 tx=0x7f2fcc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.529+0000 7f2fd3400700 1 -- 172.21.15.16:0/2153046841 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc01d070 con 0x7f2fdc101030 2024-04-05T23:38:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.529+0000 7f2fe1c00700 1 -- 172.21.15.16:0/2153046841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc1a6040 con 0x7f2fdc101030 2024-04-05T23:38:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.529+0000 7f2fd3400700 1 -- 172.21.15.16:0/2153046841 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fcc022470 con 0x7f2fdc101030 2024-04-05T23:38:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.529+0000 7f2fd3400700 1 -- 172.21.15.16:0/2153046841 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc017650 con 0x7f2fdc101030 2024-04-05T23:38:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.529+0000 7f2fe1c00700 1 -- 172.21.15.16:0/2153046841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc1a64c0 con 0x7f2fdc101030 2024-04-05T23:38:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.532+0000 7f2fd3400700 1 -- 172.21.15.16:0/2153046841 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f2fcc0225e0 con 0x7f2fdc101030 2024-04-05T23:38:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2ff57a9b80 1 Processor -- start 2024-04-05T23:38:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2fd3400700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2fc00a1290 0x7f2fc00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2ff57a9b80 1 -- start start 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2fd3400700 1 -- 172.21.15.16:0/2153046841 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f2fcc0cdf70 con 0x7f2fdc101030 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2fdb400700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2fc00a1290 0x7f2fc00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea31f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 0x556fdea2c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea2c5c0 0x556fdea2ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fde992b80 con 0x556fdea324a0 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fde983930 con 0x556fdea31b80 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.533+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fde9812f0 con 0x556fdea2c5c0 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea31f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fda000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 0x556fdea2c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea31f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47202/0 (socket says 172.21.15.16:47202) 2024-04-05T23:38:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1508220851 learned_addr learned my addr 172.21.15.16:0/1508220851 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdb400700 1 --2- 172.21.15.16:0/1508220851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea2c5c0 0x556fdea2ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdb400700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2fc00a1290 0x7f2fc00a3750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4003e80 tx=0x7f2fd400b040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1508220851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea2c5c0 msgr2=0x556fdea2ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/1508220851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea2c5c0 0x556fdea2ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1508220851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 msgr2=0x556fdea2c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/1508220851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 0x556fdea2c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1508220851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fde7fe350 con 0x556fdea31b80 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.534+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/1508220851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea31f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0a30c0 tx=0x7f2fcc009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3237a8e1ff832e23 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2fc7e00700 1 -- 172.21.15.16:0/1508220851 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc09ab60 con 0x556fdea31b80 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2fc7e00700 1 -- 172.21.15.16:0/1508220851 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fcc09b900 con 0x556fdea31b80 2024-04-05T23:38:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2fc7e00700 1 -- 172.21.15.16:0/1508220851 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0a10c0 con 0x556fdea31b80 2024-04-05T23:38:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1508220851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 msgr2=0x556fdea31f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1508220851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea31f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0a30c0 tx=0x7f2fcc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1508220851 shutdown_connections 2024-04-05T23:38:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1508220851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea2c5c0 0x556fdea2ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1508220851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea31f60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1508220851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 0x556fdea2c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1508220851 >> 172.21.15.16:0/1508220851 conn(0x556fde9a01c0 msgr2=0x556fdea26be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1508220851 shutdown_connections 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.535+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1508220851 wait complete. 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.536+0000 7f2ff57a9b80 1 -- start start 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.536+0000 7f2ff57a9b80 1 Processor -- start 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea2c5c0 0x556fdea38540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea35830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 0x556fdea35f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fde97cbe0 con 0x556fdea2c5c0 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fde918d20 con 0x556fdea31b80 2024-04-05T23:38:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fde9189c0 con 0x556fdea324a0 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2ff57a9b80 10 client.0 _ll_register_callbacks cb 0x556fde784340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea2c5c0 0x556fdea38540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea2c5c0 0x556fdea38540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42964/0 (socket says 172.21.15.16:42964) 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1657610306 learned_addr learned my addr 172.21.15.16:0/1657610306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdb400700 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 0x556fdea35f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fda000700 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea35830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1657610306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 msgr2=0x556fdea35f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 0x556fdea35f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1657610306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 msgr2=0x556fdea35830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea35830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1657610306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc1a64c0 con 0x556fdea2c5c0 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.537+0000 7f2fdb400700 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 0x556fdea35f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.538+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea2c5c0 0x556fdea38540 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0cbc10 tx=0x7f2fcc022af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359e1cb96867f46b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.538+0000 7f2fc4c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.538+0000 7f2fc4c00700 1 -- 172.21.15.16:0/1657610306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc09ab60 con 0x556fdea2c5c0 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.538+0000 7f2fc4c00700 1 -- 172.21.15.16:0/1657610306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f2fcc09b200 con 0x556fdea2c5c0 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.538+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1657610306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556fdea1f1e0 con 0x556fdea2c5c0 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.538+0000 7f2fc4c00700 1 -- 172.21.15.16:0/1657610306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fcc0a35f0 con 0x556fdea2c5c0 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.538+0000 7f2ff57a9b80 10 client.16224 fetch_fsmap learned FSMap version 47 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.538+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1657610306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x556fde7aab00 con 0x556fdea2c5c0 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.538+0000 7f2fc4c00700 1 -- 172.21.15.16:0/1657610306 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556fdea1f1e0 con 0x556fdea2c5c0 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.539+0000 7f2fc4c00700 1 -- 172.21.15.16:0/1657610306 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f2fcc0b3d20 con 0x556fdea2c5c0 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.539+0000 7f2ff57a9b80 10 client.16224 fetch_fsmap finished waiting for FSMap version 47 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.539+0000 7f2ff57a9b80 10 client.16224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.539+0000 7f2ff57a9b80 20 client.16224 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.539+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556fdea458d0 0x556fdea47cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.539+0000 7f2ff57a9b80 4 client.16224 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:38:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.539+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1657610306 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556fde9756a0 con 0x556fdea458d0 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.539+0000 7f2fda000700 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556fdea458d0 0x556fdea47cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.540+0000 7f2fda000700 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556fdea458d0 0x556fdea47cf0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b6de664eaac50c server_cookie=bbab5233ea992cb9 in_seq=0 out_seq=0 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.540+0000 7f2fc4c00700 10 client.16224 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.540+0000 7f2fc4c00700 1 -- 172.21.15.16:0/1657610306 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556fde9756a0 con 0x556fdea458d0 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.540+0000 7f2fc4c00700 10 client.16224 handle_command_reply: tid=0 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.540+0000 7f2ff57a9b80 1 client.16224 shutdown 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.540+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1657610306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea2c5c0 msgr2=0x556fdea38540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.540+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea2c5c0 0x556fdea38540 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0cbc10 tx=0x7f2fcc022af0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1657610306 shutdown_connections 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556fdea458d0 0x556fdea47cf0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea2c5c0 0x556fdea38540 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 0x556fdea35f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1657610306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea31b80 0x556fdea35830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2fc4c00700 0 client.16224 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1657610306 >> 172.21.15.16:0/1657610306 conn(0x556fde9a01c0 msgr2=0x556fdea26280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1657610306 shutdown_connections 2024-04-05T23:38:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1657610306 wait complete. 2024-04-05T23:38:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.541+0000 7f2ff57a9b80 20 client.16224 trim_cache size 0 max 16384 2024-04-05T23:38:44.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.559+0000 7f2ff57a9b80 1 Processor -- start 2024-04-05T23:38:44.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2ff57a9b80 1 -- start start 2024-04-05T23:38:44.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea2c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea31b80 0x556fdea357d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 0x556fdea3ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fde800640 con 0x556fdea324a0 2024-04-05T23:38:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fdea23aa0 con 0x556fdea2c5c0 2024-04-05T23:38:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fdea36e00 con 0x556fdea31b80 2024-04-05T23:38:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea2c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea2c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47206/0 (socket says 172.21.15.16:47206) 2024-04-05T23:38:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1729380794 learned_addr learned my addr 172.21.15.16:0/1729380794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.560+0000 7f2fdb400700 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 0x556fdea3ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fda000700 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea31b80 0x556fdea357d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1729380794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea31b80 msgr2=0x556fdea357d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea31b80 0x556fdea357d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1729380794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 msgr2=0x556fdea3ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 0x556fdea3ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fdaa00700 1 -- 172.21.15.16:0/1729380794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fde942e40 con 0x556fdea2c5c0 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fda000700 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea31b80 0x556fdea357d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fdb400700 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 0x556fdea3ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea2c030 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc099550 tx=0x7f2fcc022f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=801dbbe33fbfc793 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fd0c00700 1 -- 172.21.15.16:0/1729380794 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0a3400 con 0x556fdea2c5c0 2024-04-05T23:38:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fd0c00700 1 -- 172.21.15.16:0/1729380794 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fcc09ab60 con 0x556fdea2c5c0 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.561+0000 7f2fd0c00700 1 -- 172.21.15.16:0/1729380794 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc09a900 con 0x556fdea2c5c0 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1729380794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 msgr2=0x556fdea2c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea2c030 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc099550 tx=0x7f2fcc022f90 comp rx=0 tx=0).stop 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1729380794 shutdown_connections 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea324a0 0x556fdea3ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea2c030 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/1729380794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea31b80 0x556fdea357d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1729380794 >> 172.21.15.16:0/1729380794 conn(0x556fde9a01c0 msgr2=0x556fdea350c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1729380794 shutdown_connections 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/1729380794 wait complete. 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.562+0000 7f2ff57a9b80 1 -- start start 2024-04-05T23:38:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2ff57a9b80 1 Processor -- start 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea38ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea31b80 0x556fdea391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2ff57a9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 0x556fdea39720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fdea3d050 con 0x556fdea31b80 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fdea3d1c0 con 0x556fdea2c5c0 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2ff57a9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fdea986b0 con 0x556fdea324a0 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2ff57a9b80 10 client.0 _ll_register_callbacks cb 0x7f2fc8009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea38ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2fdaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea38ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47208/0 (socket says 172.21.15.16:47208) 2024-04-05T23:38:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2fdaa00700 1 -- 172.21.15.16:0/3300814964 learned_addr learned my addr 172.21.15.16:0/3300814964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2fda000700 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea31b80 0x556fdea391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.563+0000 7f2fdb400700 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 0x556fdea39720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fdaa00700 1 -- 172.21.15.16:0/3300814964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 msgr2=0x556fdea39720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 0x556fdea39720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fdaa00700 1 -- 172.21.15.16:0/3300814964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea31b80 msgr2=0x556fdea391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea31b80 0x556fdea391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fdaa00700 1 -- 172.21.15.16:0/3300814964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556fde7aab00 con 0x556fdea2c5c0 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fdaa00700 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea38ca0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc07ef00 tx=0x7f2fcc0cb770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22b1728d54e9eaa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fc4c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fc4c00700 1 -- 172.21.15.16:0/3300814964 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0a3400 con 0x556fdea2c5c0 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fc4c00700 1 -- 172.21.15.16:0/3300814964 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f2fcc09f030 con 0x556fdea2c5c0 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/3300814964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fcc09b900 con 0x556fdea2c5c0 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2fc4c00700 1 -- 172.21.15.16:0/3300814964 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fcc09b200 con 0x556fdea2c5c0 2024-04-05T23:38:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.564+0000 7f2ff57a9b80 10 client.26279 fetch_fsmap learned FSMap version 47 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/3300814964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x556fde802c70 con 0x556fdea2c5c0 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2fc4c00700 1 -- 172.21.15.16:0/3300814964 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fcc09b900 con 0x556fdea2c5c0 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2fc4c00700 1 -- 172.21.15.16:0/3300814964 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 47) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f2fcc0b3d20 con 0x556fdea2c5c0 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2ff57a9b80 10 client.26279 fetch_fsmap finished waiting for FSMap version 47 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2ff57a9b80 10 client.26279 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2ff57a9b80 20 client.26279 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556fdea998a0 0x556fdea9bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2ff57a9b80 4 client.26279 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/3300814964 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556fde7cbc90 con 0x556fdea998a0 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.565+0000 7f2fda000700 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556fdea998a0 0x556fdea9bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.566+0000 7f2fda000700 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556fdea998a0 0x556fdea9bc90 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb2f6f5c61b65a07 server_cookie=434c1b7ce611f31d in_seq=0 out_seq=0 2024-04-05T23:38:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.566+0000 7f2fc4c00700 10 client.26279 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2fc4c00700 1 -- 172.21.15.16:0/3300814964 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556fde7cbc90 con 0x556fdea998a0 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2fc4c00700 10 client.26279 handle_command_reply: tid=0 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 client.26279 shutdown 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/3300814964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 msgr2=0x556fdea38ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea38ca0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc07ef00 tx=0x7f2fcc0cb770 comp rx=0 tx=0).stop 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/3300814964 shutdown_connections 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556fdea998a0 0x556fdea9bc90 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556fdea324a0 0x556fdea39720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2fc4c00700 0 client.26279 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556fdea2c5c0 0x556fdea38ca0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 --2- 172.21.15.16:0/3300814964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fdea31b80 0x556fdea391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/3300814964 >> 172.21.15.16:0/3300814964 conn(0x556fde9a01c0 msgr2=0x556fdea41420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/3300814964 shutdown_connections 2024-04-05T23:38:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 1 -- 172.21.15.16:0/3300814964 wait complete. 2024-04-05T23:38:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.590+0000 7f2ff57a9b80 20 client.26279 trim_cache size 0 max 16384 2024-04-05T23:38:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:38:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:38:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:38:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 -- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2fc00a1290 msgr2=0x7f2fc00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2fc00a1290 0x7f2fc00a3750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4003e80 tx=0x7f2fd400b040 comp rx=0 tx=0).stop 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 -- 172.21.15.16:0/2153046841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fdc101030 msgr2=0x7f2fdc1994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fdc101030 0x7f2fdc1994b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc005910 tx=0x7f2fcc004730 comp rx=0 tx=0).stop 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 -- 172.21.15.16:0/2153046841 shutdown_connections 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f2fc00a1290 0x7f2fc00a3750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fdc19dd90 0x7f2fdc19e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fdc101030 0x7f2fdc1994b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 --2- 172.21.15.16:0/2153046841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fdc106150 0x7f2fdc1999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 -- 172.21.15.16:0/2153046841 >> 172.21.15.16:0/2153046841 conn(0x7f2fdc0fc8b0 msgr2=0x7f2fdc0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 -- 172.21.15.16:0/2153046841 shutdown_connections 2024-04-05T23:38:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:44.591+0000 7f2fe1c00700 1 -- 172.21.15.16:0/2153046841 wait complete. 2024-04-05T23:38:44.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 310305 8.25 MB/sec execute 124 sec latency 757.830 ms 2024-04-05T23:38:45.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 310305 8.19 MB/sec execute 125 sec latency 1757.906 ms 2024-04-05T23:38:46.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:45 smithi016 ceph-mon[38248]: pgmap v161: 97 pgs: 97 active+clean; 112 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-04-05T23:38:46.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:45 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:38:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:45 smithi130 ceph-mon[79417]: pgmap v161: 97 pgs: 97 active+clean; 112 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-04-05T23:38:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:45 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:38:46.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:45 smithi027 ceph-mon[78597]: pgmap v161: 97 pgs: 97 active+clean; 112 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-04-05T23:38:46.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:45 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:38:46.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 310305 8.12 MB/sec execute 126 sec latency 2758.024 ms 2024-04-05T23:38:47.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:46 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:38:47.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:46 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:38:47.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:46 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:38:47.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:46 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:38:47.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:46 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:38:47.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:46 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:38:47.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:46 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:38:47.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:46 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:38:47.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:46 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:38:47.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 311982 8.13 MB/sec execute 127 sec latency 2973.790 ms 2024-04-05T23:38:48.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:48 smithi016 ceph-mon[38248]: pgmap v162: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 238 op/s 2024-04-05T23:38:48.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:48 smithi130 ceph-mon[79417]: pgmap v162: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 238 op/s 2024-04-05T23:38:48.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:48 smithi027 ceph-mon[78597]: pgmap v162: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 238 op/s 2024-04-05T23:38:48.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 313135 8.11 MB/sec execute 128 sec latency 15.296 ms 2024-04-05T23:38:49.998 INFO:tasks.workunit.client.0.smithi016.stdout: 1 313847 8.08 MB/sec execute 129 sec latency 332.679 ms 2024-04-05T23:38:50.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:50 smithi016 ceph-mon[38248]: pgmap v163: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-04-05T23:38:50.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:50 smithi130 ceph-mon[79417]: pgmap v163: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-04-05T23:38:50.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:50 smithi027 ceph-mon[78597]: pgmap v163: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 1.4 MiB/s wr, 133 op/s 2024-04-05T23:38:50.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 313847 8.02 MB/sec execute 130 sec latency 1332.771 ms 2024-04-05T23:38:51.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 313847 7.96 MB/sec execute 131 sec latency 2332.855 ms 2024-04-05T23:38:52.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:52 smithi016 ceph-mon[38248]: pgmap v164: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-05T23:38:52.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:52 smithi130 ceph-mon[79417]: pgmap v164: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-05T23:38:52.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:52 smithi027 ceph-mon[78597]: pgmap v164: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-05T23:38:52.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 315512 7.97 MB/sec execute 132 sec latency 2547.979 ms 2024-04-05T23:38:53.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 316684 7.95 MB/sec execute 133 sec latency 17.903 ms 2024-04-05T23:38:54.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:54 smithi016 ceph-mon[38248]: pgmap v165: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-04-05T23:38:54.587 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:54 smithi130 ceph-mon[79417]: pgmap v165: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-04-05T23:38:54.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:54 smithi027 ceph-mon[78597]: pgmap v165: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-04-05T23:38:55.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 317396 7.92 MB/sec execute 134 sec latency 320.281 ms 2024-04-05T23:38:55.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318975 7.93 MB/sec execute 135 sec latency 417.683 ms 2024-04-05T23:38:56.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:56 smithi016 ceph-mon[38248]: pgmap v166: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-04-05T23:38:56.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:56 smithi130 ceph-mon[79417]: pgmap v166: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-04-05T23:38:56.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:56 smithi027 ceph-mon[78597]: pgmap v166: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-04-05T23:38:56.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 320149 7.91 MB/sec execute 136 sec latency 17.479 ms 2024-04-05T23:38:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.744+0000 7f56b0c00700 1 -- 172.21.15.16:0/2000378424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac106160 msgr2=0x7f56ac10dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.744+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2000378424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac106160 0x7f56ac10dce0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f56a400b0a0 tx=0x7f56a400b3b0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.744+0000 7f56b0c00700 1 -- 172.21.15.16:0/2000378424 shutdown_connections 2024-04-05T23:38:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.744+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2000378424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac106160 0x7f56ac10dce0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.744+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2000378424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56ac101040 0x7f56ac101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.744+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2000378424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56ac1019f0 0x7f56ac105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.744+0000 7f56b0c00700 1 -- 172.21.15.16:0/2000378424 >> 172.21.15.16:0/2000378424 conn(0x7f56ac0fc8e0 msgr2=0x7f56ac0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.744+0000 7f56b0c00700 1 -- 172.21.15.16:0/2000378424 shutdown_connections 2024-04-05T23:38:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.744+0000 7f56b0c00700 1 -- 172.21.15.16:0/2000378424 wait complete. 2024-04-05T23:38:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56b0c00700 1 Processor -- start 2024-04-05T23:38:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56b0c00700 1 -- start start 2024-04-05T23:38:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56b0c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac101040 0x7f56ac0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56b0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56ac106160 0x7f56ac073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56b0c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56ac073c60 0x7f56ac1a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56b0c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac1a5bf0 con 0x7f56ac073c60 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56b0c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac1a5d30 con 0x7f56ac106160 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56b0c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac1a5ef0 con 0x7f56ac101040 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56a9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac101040 0x7f56ac0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.745+0000 7f56aa800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56ac073c60 0x7f56ac1a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac101040 0x7f56ac0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:58150/0 (socket says 172.21.15.16:58150) 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a9e00700 1 -- 172.21.15.16:0/2243864559 learned_addr learned my addr 172.21.15.16:0/2243864559 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a9400700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56ac106160 0x7f56ac073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a9e00700 1 -- 172.21.15.16:0/2243864559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56ac106160 msgr2=0x7f56ac073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a9e00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56ac106160 0x7f56ac073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a9e00700 1 -- 172.21.15.16:0/2243864559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56ac073c60 msgr2=0x7f56ac1a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a9e00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56ac073c60 0x7f56ac1a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a9e00700 1 -- 172.21.15.16:0/2243864559 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a4009d00 con 0x7f56ac101040 2024-04-05T23:38:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a9e00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac101040 0x7f56ac0731e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f569c00e9a0 tx=0x7f569c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a2a00700 1 -- 172.21.15.16:0/2243864559 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c009790 con 0x7f56ac101040 2024-04-05T23:38:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a2a00700 1 -- 172.21.15.16:0/2243864559 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f569c010430 con 0x7f56ac101040 2024-04-05T23:38:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56b0c00700 1 -- 172.21.15.16:0/2243864559 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ac1a61d0 con 0x7f56ac101040 2024-04-05T23:38:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.746+0000 7f56a2a00700 1 -- 172.21.15.16:0/2243864559 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c018680 con 0x7f56ac101040 2024-04-05T23:38:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.747+0000 7f56b0c00700 1 -- 172.21.15.16:0/2243864559 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ac1a6730 con 0x7f56ac101040 2024-04-05T23:38:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.749+0000 7f56a2a00700 1 -- 172.21.15.16:0/2243864559 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f569c0105a0 con 0x7f56ac101040 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56c46e1b80 1 Processor -- start 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56a2a00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56900a1310 0x7f56900a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56c46e1b80 1 -- start start 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56a2a00700 1 -- 172.21.15.16:0/2243864559 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f569c014070 con 0x7f56ac101040 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56aa800700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56900a1310 0x7f56900a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 0x555aa2a63f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a5e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a5ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa29c4b80 con 0x555aa2a63b80 2024-04-05T23:38:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa29b5930 con 0x555aa2a644a0 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.751+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa29b32f0 con 0x555aa2a5e5c0 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 0x555aa2a63f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a5e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a5e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51198/0 (socket says 172.21.15.16:51198) 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9400700 1 -- 172.21.15.16:0/4002615519 learned_addr learned my addr 172.21.15.16:0/4002615519 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56aa800700 1 --2- 172.21.15.16:0/4002615519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a5ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56aa800700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56900a1310 0x7f56900a37d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f56a400b780 tx=0x7f56a4006210 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9400700 1 -- 172.21.15.16:0/4002615519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 msgr2=0x555aa2a5ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9400700 1 --2- 172.21.15.16:0/4002615519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a5ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9400700 1 -- 172.21.15.16:0/4002615519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 msgr2=0x555aa2a63f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9400700 1 --2- 172.21.15.16:0/4002615519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 0x555aa2a63f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9400700 1 -- 172.21.15.16:0/4002615519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555aa2830350 con 0x555aa2a644a0 2024-04-05T23:38:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.752+0000 7f56a9400700 1 --2- 172.21.15.16:0/4002615519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a5e030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5698009cb0 tx=0x7f5698011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48916849e267546d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f5697000700 1 -- 172.21.15.16:0/4002615519 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5698011d10 con 0x555aa2a644a0 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f5697000700 1 -- 172.21.15.16:0/4002615519 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5698011e70 con 0x555aa2a644a0 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f5697000700 1 -- 172.21.15.16:0/4002615519 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5698005280 con 0x555aa2a644a0 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f56c46e1b80 1 -- 172.21.15.16:0/4002615519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 msgr2=0x555aa2a5e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/4002615519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a5e030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5698009cb0 tx=0x7f5698011440 comp rx=0 tx=0).stop 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f56c46e1b80 1 -- 172.21.15.16:0/4002615519 shutdown_connections 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/4002615519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 0x555aa2a63f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/4002615519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a5ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/4002615519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a5e030 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f56c46e1b80 1 -- 172.21.15.16:0/4002615519 >> 172.21.15.16:0/4002615519 conn(0x555aa29d21c0 msgr2=0x555aa2a58be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f56c46e1b80 1 -- 172.21.15.16:0/4002615519 shutdown_connections 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.753+0000 7f56c46e1b80 1 -- 172.21.15.16:0/4002615519 wait complete. 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.754+0000 7f56c46e1b80 1 -- start start 2024-04-05T23:38:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.754+0000 7f56c46e1b80 1 Processor -- start 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.754+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a63b80 0x555aa2a678e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a67fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa29aebe0 con 0x555aa2a5e5c0 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa294ad20 con 0x555aa2a644a0 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa294a9c0 con 0x555aa2a63b80 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56c46e1b80 10 client.0 _ll_register_callbacks cb 0x555aa27b6340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56a9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56a9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47788/0 (socket says 172.21.15.16:47788) 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56a9e00700 1 -- 172.21.15.16:0/3760963845 learned_addr learned my addr 172.21.15.16:0/3760963845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56aa800700 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a67fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56a9400700 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a63b80 0x555aa2a678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56a9e00700 1 -- 172.21.15.16:0/3760963845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a63b80 msgr2=0x555aa2a678e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56a9e00700 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a63b80 0x555aa2a678e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56a9e00700 1 -- 172.21.15.16:0/3760963845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 msgr2=0x555aa2a67fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56a9e00700 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a67fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.755+0000 7f56a9e00700 1 -- 172.21.15.16:0/3760963845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56980098e0 con 0x555aa2a5e5c0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f56a9e00700 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6a5a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f569c0ca5e0 tx=0x7f569c099f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1676fdbe623e2d43 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f568b400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f568b400700 1 -- 172.21.15.16:0/3760963845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c099630 con 0x555aa2a5e5c0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f568b400700 1 -- 172.21.15.16:0/3760963845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f569c0a23c0 con 0x555aa2a5e5c0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f56c46e1b80 1 -- 172.21.15.16:0/3760963845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555aa2a511e0 con 0x555aa2a5e5c0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f568b400700 1 -- 172.21.15.16:0/3760963845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f569c0a1f30 con 0x555aa2a5e5c0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f56c46e1b80 10 client.16242 fetch_fsmap learned FSMap version 47 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f56c46e1b80 1 -- 172.21.15.16:0/3760963845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x555aa27dcb00 con 0x555aa2a5e5c0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f568b400700 1 -- 172.21.15.16:0/3760963845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555aa2a511e0 con 0x555aa2a5e5c0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f568b400700 1 -- 172.21.15.16:0/3760963845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 47) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f569c094be0 con 0x555aa2a5e5c0 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f56c46e1b80 10 client.16242 fetch_fsmap finished waiting for FSMap version 47 2024-04-05T23:38:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f56c46e1b80 10 client.16242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.756+0000 7f56c46e1b80 20 client.16242 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.757+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555aa2a77f70 0x555aa2a7a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.757+0000 7f56c46e1b80 4 client.16242 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:38:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.757+0000 7f56c46e1b80 1 -- 172.21.15.16:0/3760963845 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555aa29a76a0 con 0x555aa2a77f70 2024-04-05T23:38:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.757+0000 7f56a9400700 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555aa2a77f70 0x555aa2a7a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.757+0000 7f568b400700 10 client.16242 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.757+0000 7f56a9400700 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555aa2a77f70 0x555aa2a7a390 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1e1ba8a8db822e6 server_cookie=38437287c71b7053 in_seq=0 out_seq=0 2024-04-05T23:38:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f568b400700 1 -- 172.21.15.16:0/3760963845 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555aa29a76a0 con 0x555aa2a77f70 2024-04-05T23:38:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f568b400700 10 client.16242 handle_command_reply: tid=0 2024-04-05T23:38:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 client.16242 shutdown 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 -- 172.21.15.16:0/3760963845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 msgr2=0x555aa2a6a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6a5a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f569c0ca5e0 tx=0x7f569c099f40 comp rx=0 tx=0).stop 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 -- 172.21.15.16:0/3760963845 shutdown_connections 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555aa2a77f70 0x555aa2a7a390 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6a5a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a67fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f568b400700 0 client.16242 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/3760963845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a63b80 0x555aa2a678e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 -- 172.21.15.16:0/3760963845 >> 172.21.15.16:0/3760963845 conn(0x555aa29d21c0 msgr2=0x555aa2a58280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 -- 172.21.15.16:0/3760963845 shutdown_connections 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.758+0000 7f56c46e1b80 1 -- 172.21.15.16:0/3760963845 wait complete. 2024-04-05T23:38:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.759+0000 7f56c46e1b80 20 client.16242 trim_cache size 0 max 16384 2024-04-05T23:38:57.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.780+0000 7f56c46e1b80 1 Processor -- start 2024-04-05T23:38:57.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.780+0000 7f56c46e1b80 1 -- start start 2024-04-05T23:38:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a58a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 0x555aa2a6a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a736e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa2832640 con 0x555aa2a63b80 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa2a65e50 con 0x555aa2a644a0 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa2a62560 con 0x555aa2a5e5c0 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56a9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a58a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56a9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a58a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:58178/0 (socket says 172.21.15.16:58178) 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56a9e00700 1 -- 172.21.15.16:0/615514979 learned_addr learned my addr 172.21.15.16:0/615514979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56a9400700 1 --2- 172.21.15.16:0/615514979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 0x555aa2a6a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.781+0000 7f56aa800700 1 --2- 172.21.15.16:0/615514979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.782+0000 7f56a9e00700 1 -- 172.21.15.16:0/615514979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 msgr2=0x555aa2a736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.782+0000 7f56a9e00700 1 --2- 172.21.15.16:0/615514979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.782+0000 7f56a9e00700 1 -- 172.21.15.16:0/615514979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 msgr2=0x555aa2a6a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.782+0000 7f56a9e00700 1 --2- 172.21.15.16:0/615514979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 0x555aa2a6a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.782+0000 7f56a9e00700 1 -- 172.21.15.16:0/615514979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555aa2974e40 con 0x555aa2a5e5c0 2024-04-05T23:38:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.782+0000 7f56a9e00700 1 --2- 172.21.15.16:0/615514979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a58a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f569c099ed0 tx=0x7f569c010a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c701eafd81eb6c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.782+0000 7f5697a00700 1 -- 172.21.15.16:0/615514979 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c098f40 con 0x555aa2a5e5c0 2024-04-05T23:38:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.782+0000 7f5697a00700 1 -- 172.21.15.16:0/615514979 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f569c0a23c0 con 0x555aa2a5e5c0 2024-04-05T23:38:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.782+0000 7f5697a00700 1 -- 172.21.15.16:0/615514979 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c0a2190 con 0x555aa2a5e5c0 2024-04-05T23:38:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.784+0000 7f56c46e1b80 1 -- 172.21.15.16:0/615514979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 msgr2=0x555aa2a58a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:38:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.784+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/615514979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a58a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f569c099ed0 tx=0x7f569c010a90 comp rx=0 tx=0).stop 2024-04-05T23:38:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.784+0000 7f56c46e1b80 1 -- 172.21.15.16:0/615514979 shutdown_connections 2024-04-05T23:38:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.784+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/615514979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a644a0 0x555aa2a736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.784+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/615514979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a5e5c0 0x555aa2a58a00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.784+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/615514979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a63b80 0x555aa2a6a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.784+0000 7f56c46e1b80 1 -- 172.21.15.16:0/615514979 >> 172.21.15.16:0/615514979 conn(0x555aa29d21c0 msgr2=0x555aa2a5e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:38:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.784+0000 7f56c46e1b80 1 -- 172.21.15.16:0/615514979 shutdown_connections 2024-04-05T23:38:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.784+0000 7f56c46e1b80 1 -- 172.21.15.16:0/615514979 wait complete. 2024-04-05T23:38:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.785+0000 7f56c46e1b80 1 -- start start 2024-04-05T23:38:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.785+0000 7f56c46e1b80 1 Processor -- start 2024-04-05T23:38:57.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.786+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.786+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a63b80 0x555aa2a6e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.786+0000 7f56c46e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a644a0 0x555aa2acada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.786+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa2a6ef50 con 0x555aa2a5e5c0 2024-04-05T23:38:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.786+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa2acb2e0 con 0x555aa2a63b80 2024-04-05T23:38:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.786+0000 7f56c46e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aa2acb420 con 0x555aa2a644a0 2024-04-05T23:38:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.786+0000 7f56c46e1b80 10 client.0 _ll_register_callbacks cb 0x7f569c099970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:38:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.786+0000 7f56aa800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a644a0 0x555aa2acada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.787+0000 7f56aa800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a644a0 0x555aa2acada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:58182/0 (socket says 172.21.15.16:58182) 2024-04-05T23:38:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.787+0000 7f56aa800700 1 -- 172.21.15.16:0/526428777 learned_addr learned my addr 172.21.15.16:0/526428777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:38:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.787+0000 7f56a9400700 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a63b80 0x555aa2a6e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.788+0000 7f56aa800700 1 -- 172.21.15.16:0/526428777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a63b80 msgr2=0x555aa2a6e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.788+0000 7f56aa800700 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a63b80 0x555aa2a6e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.788+0000 7f56aa800700 1 -- 172.21.15.16:0/526428777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 msgr2=0x555aa2a6e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:38:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.788+0000 7f56a9e00700 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.788+0000 7f56aa800700 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:38:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.789+0000 7f56aa800700 1 -- 172.21.15.16:0/526428777 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555aa27dcb00 con 0x555aa2a644a0 2024-04-05T23:38:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.789+0000 7f56a9400700 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a63b80 0x555aa2a6e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:38:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.789+0000 7f56a9e00700 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:38:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.790+0000 7f56aa800700 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a644a0 0x555aa2acada0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f56a400b0a0 tx=0x7f56a40099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a908da75e45c42b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:38:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.790+0000 7f568b400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:38:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.791+0000 7f568b400700 1 -- 172.21.15.16:0/526428777 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a400e040 con 0x555aa2a644a0 2024-04-05T23:38:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.791+0000 7f568b400700 1 -- 172.21.15.16:0/526428777 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f56a4007810 con 0x555aa2a644a0 2024-04-05T23:38:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.792+0000 7f56c46e1b80 1 -- 172.21.15.16:0/526428777 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5698011e70 con 0x555aa2a644a0 2024-04-05T23:38:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.792+0000 7f568b400700 1 -- 172.21.15.16:0/526428777 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56a401c8a0 con 0x555aa2a644a0 2024-04-05T23:38:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.792+0000 7f56c46e1b80 10 client.26329 fetch_fsmap learned FSMap version 47 2024-04-05T23:38:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.792+0000 7f56c46e1b80 1 -- 172.21.15.16:0/526428777 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x555aa2834c70 con 0x555aa2a644a0 2024-04-05T23:38:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.793+0000 7f568b400700 1 -- 172.21.15.16:0/526428777 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5698011e70 con 0x555aa2a644a0 2024-04-05T23:38:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.793+0000 7f568b400700 1 -- 172.21.15.16:0/526428777 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 47) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f56a4033a20 con 0x555aa2a644a0 2024-04-05T23:38:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.793+0000 7f56c46e1b80 10 client.26329 fetch_fsmap finished waiting for FSMap version 47 2024-04-05T23:38:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.793+0000 7f56c46e1b80 10 client.26329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:38:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.793+0000 7f56c46e1b80 20 client.26329 populate_metadata read hostname 'smithi016' 2024-04-05T23:38:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.793+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555aa2ad53c0 0x555aa2ad77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:38:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.793+0000 7f56c46e1b80 4 client.26329 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:38:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.793+0000 7f56c46e1b80 1 -- 172.21.15.16:0/526428777 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555aa27fdc90 con 0x555aa2ad53c0 2024-04-05T23:38:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.794+0000 7f56a9e00700 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555aa2ad53c0 0x555aa2ad77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:38:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.794+0000 7f568b400700 10 client.26329 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:38:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:38:57.794+0000 7f56a9e00700 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555aa2ad53c0 0x555aa2ad77e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf4a1aa3d7110486 server_cookie=6c92746c53eae47a in_seq=0 out_seq=0 2024-04-05T23:38:57.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 320938 7.89 MB/sec execute 137 sec latency 422.862 ms 2024-04-05T23:38:58.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:38:58 smithi016 ceph-mon[38248]: pgmap v167: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 195 op/s 2024-04-05T23:38:58.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:38:58 smithi130 ceph-mon[79417]: pgmap v167: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 195 op/s 2024-04-05T23:38:58.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:38:58 smithi027 ceph-mon[78597]: pgmap v167: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 195 op/s 2024-04-05T23:38:58.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 320938 7.83 MB/sec execute 138 sec latency 1422.998 ms 2024-04-05T23:38:59.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 320938 7.77 MB/sec execute 139 sec latency 2423.170 ms 2024-04-05T23:39:00.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:00 smithi016 ceph-mon[38248]: pgmap v168: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-04-05T23:39:00.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:00 smithi130 ceph-mon[79417]: pgmap v168: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-04-05T23:39:00.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:00 smithi027 ceph-mon[78597]: pgmap v168: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 130 op/s 2024-04-05T23:39:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.923+0000 7f568b400700 1 -- 172.21.15.16:0/526428777 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555aa27fdc90 con 0x555aa2ad53c0 2024-04-05T23:39:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.923+0000 7f568b400700 10 client.26329 handle_command_reply: tid=0 2024-04-05T23:39:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.923+0000 7f56c46e1b80 1 client.26329 shutdown 2024-04-05T23:39:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 -- 172.21.15.16:0/526428777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a644a0 msgr2=0x555aa2acada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a644a0 0x555aa2acada0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f56a400b0a0 tx=0x7f56a40099e0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 -- 172.21.15.16:0/526428777 shutdown_connections 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555aa2ad53c0 0x555aa2ad77e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555aa2a5e5c0 0x555aa2a6e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555aa2a644a0 0x555aa2acada0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 --2- 172.21.15.16:0/526428777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555aa2a63b80 0x555aa2a6e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f568b400700 0 client.26329 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 -- 172.21.15.16:0/526428777 >> 172.21.15.16:0/526428777 conn(0x555aa29d21c0 msgr2=0x555aa2a63200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 -- 172.21.15.16:0/526428777 shutdown_connections 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 1 -- 172.21.15.16:0/526428777 wait complete. 2024-04-05T23:39:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.924+0000 7f56c46e1b80 20 client.26329 trim_cache size 0 max 16384 2024-04-05T23:39:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:39:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:39:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:39:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:39:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 -- 172.21.15.16:0/2243864559 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56900a1310 msgr2=0x7f56900a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56900a1310 0x7f56900a37d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f56a400b780 tx=0x7f56a4006210 comp rx=0 tx=0).stop 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 -- 172.21.15.16:0/2243864559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac101040 msgr2=0x7f56ac0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac101040 0x7f56ac0731e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f569c00e9a0 tx=0x7f569c00ecb0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 -- 172.21.15.16:0/2243864559 shutdown_connections 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f56900a1310 0x7f56900a37d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56ac073c60 0x7f56ac1a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56ac101040 0x7f56ac0731e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 --2- 172.21.15.16:0/2243864559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56ac106160 0x7f56ac073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 -- 172.21.15.16:0/2243864559 >> 172.21.15.16:0/2243864559 conn(0x7f56ac0fc8e0 msgr2=0x7f56ac0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.925+0000 7f56b0c00700 1 -- 172.21.15.16:0/2243864559 shutdown_connections 2024-04-05T23:39:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:00.926+0000 7f56b0c00700 1 -- 172.21.15.16:0/2243864559 wait complete. 2024-04-05T23:39:00.999 INFO:tasks.workunit.client.0.smithi016.stdout: 1 321107 7.72 MB/sec execute 140 sec latency 3319.974 ms 2024-04-05T23:39:02.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 322596 7.73 MB/sec execute 141 sec latency 29.803 ms 2024-04-05T23:39:02.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:02 smithi016 ceph-mon[38248]: pgmap v169: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 194 op/s 2024-04-05T23:39:02.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:02 smithi130 ceph-mon[79417]: pgmap v169: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 194 op/s 2024-04-05T23:39:02.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:02 smithi027 ceph-mon[78597]: pgmap v169: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 194 op/s 2024-04-05T23:39:03.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 323678 7.71 MB/sec execute 142 sec latency 49.216 ms 2024-04-05T23:39:04.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 324487 7.69 MB/sec execute 143 sec latency 488.129 ms 2024-04-05T23:39:04.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:04 smithi016 ceph-mon[38248]: pgmap v170: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 138 op/s 2024-04-05T23:39:04.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:04 smithi130 ceph-mon[79417]: pgmap v170: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 138 op/s 2024-04-05T23:39:04.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:04 smithi027 ceph-mon[78597]: pgmap v170: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 138 op/s 2024-04-05T23:39:05.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 324487 7.64 MB/sec execute 144 sec latency 1488.287 ms 2024-04-05T23:39:06.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 324487 7.59 MB/sec execute 145 sec latency 2488.414 ms 2024-04-05T23:39:06.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:06 smithi016 ceph-mon[38248]: pgmap v171: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 138 op/s 2024-04-05T23:39:06.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:06 smithi130 ceph-mon[79417]: pgmap v171: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 138 op/s 2024-04-05T23:39:06.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:06 smithi027 ceph-mon[78597]: pgmap v171: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 138 op/s 2024-04-05T23:39:07.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 325184 7.55 MB/sec execute 146 sec latency 3216.829 ms 2024-04-05T23:39:08.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 326708 7.58 MB/sec execute 147 sec latency 20.008 ms 2024-04-05T23:39:08.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:08 smithi016 ceph-mon[38248]: pgmap v172: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 247 op/s 2024-04-05T23:39:08.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:08 smithi130 ceph-mon[79417]: pgmap v172: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 247 op/s 2024-04-05T23:39:08.717 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:08 smithi027 ceph-mon[78597]: pgmap v172: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 247 op/s 2024-04-05T23:39:09.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 328029 7.56 MB/sec execute 148 sec latency 17.401 ms 2024-04-05T23:39:10.000 INFO:tasks.workunit.client.0.smithi016.stdout: 1 328029 7.51 MB/sec execute 149 sec latency 1001.293 ms 2024-04-05T23:39:10.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:10 smithi016 ceph-mon[38248]: pgmap v173: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-04-05T23:39:10.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:10 smithi130 ceph-mon[79417]: pgmap v173: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-04-05T23:39:10.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:10 smithi027 ceph-mon[78597]: pgmap v173: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-04-05T23:39:11.001 INFO:tasks.workunit.client.0.smithi016.stdout: 1 328029 7.46 MB/sec execute 150 sec latency 2001.475 ms 2024-04-05T23:39:11.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:11 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:11.585 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:11 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:11.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:11 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:12.001 INFO:tasks.workunit.client.0.smithi016.stdout: 1 328029 7.41 MB/sec execute 151 sec latency 3001.631 ms 2024-04-05T23:39:12.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:12 smithi016 ceph-mon[38248]: pgmap v174: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 231 op/s 2024-04-05T23:39:12.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:12 smithi130 ceph-mon[79417]: pgmap v174: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 231 op/s 2024-04-05T23:39:12.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:12 smithi027 ceph-mon[78597]: pgmap v174: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 231 op/s 2024-04-05T23:39:13.001 INFO:tasks.workunit.client.0.smithi016.stdout: 1 329610 7.42 MB/sec execute 152 sec latency 3214.113 ms 2024-04-05T23:39:14.001 INFO:tasks.workunit.client.0.smithi016.stdout: 1 329610 7.37 MB/sec execute 153 sec latency 1015.690 ms 2024-04-05T23:39:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.096+0000 7f8365a00700 1 -- 172.21.15.16:0/1517774431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 msgr2=0x7f8360101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.096+0000 7f8365a00700 1 --2- 172.21.15.16:0/1517774431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 0x7f8360101430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8350009990 tx=0x7f8350009ca0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.096+0000 7f8365a00700 1 -- 172.21.15.16:0/1517774431 shutdown_connections 2024-04-05T23:39:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.096+0000 7f8365a00700 1 --2- 172.21.15.16:0/1517774431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 0x7f8360101430 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.096+0000 7f8365a00700 1 --2- 172.21.15.16:0/1517774431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8360106170 0x7f836010dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.096+0000 7f8365a00700 1 --2- 172.21.15.16:0/1517774431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8360101a00 0x7f8360105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.096+0000 7f8365a00700 1 -- 172.21.15.16:0/1517774431 >> 172.21.15.16:0/1517774431 conn(0x7f83600fc8d0 msgr2=0x7f83600fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.096+0000 7f8365a00700 1 -- 172.21.15.16:0/1517774431 shutdown_connections 2024-04-05T23:39:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.096+0000 7f8365a00700 1 -- 172.21.15.16:0/1517774431 wait complete. 2024-04-05T23:39:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.098+0000 7f8365a00700 1 Processor -- start 2024-04-05T23:39:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.098+0000 7f8365a00700 1 -- start start 2024-04-05T23:39:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f8365a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 0x7f83601995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f8365a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8360106170 0x7f8360199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f8365a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f836019de30 0x7f836019e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f8365a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836019e8b0 con 0x7f8360101050 2024-04-05T23:39:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f8365a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836019ea20 con 0x7f836019de30 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f8365a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836019ebe0 con 0x7f8360106170 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 0x7f83601995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 0x7f83601995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44466/0 (socket says 172.21.15.16:44466) 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835ea00700 1 -- 172.21.15.16:0/3004512173 learned_addr learned my addr 172.21.15.16:0/3004512173 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f8357e00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8360106170 0x7f8360199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835f400700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f836019de30 0x7f836019e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835ea00700 1 -- 172.21.15.16:0/3004512173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8360106170 msgr2=0x7f8360199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835ea00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8360106170 0x7f8360199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835ea00700 1 -- 172.21.15.16:0/3004512173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f836019de30 msgr2=0x7f836019e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835ea00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f836019de30 0x7f836019e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835ea00700 1 -- 172.21.15.16:0/3004512173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83500095f0 con 0x7f8360101050 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.099+0000 7f835f400700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f836019de30 0x7f836019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.100+0000 7f835ea00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 0x7f83601995a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8350000c00 tx=0x7f8350005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.100+0000 7f8357400700 1 -- 172.21.15.16:0/3004512173 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835001d070 con 0x7f8360101050 2024-04-05T23:39:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.100+0000 7f8357400700 1 -- 172.21.15.16:0/3004512173 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8350022470 con 0x7f8360101050 2024-04-05T23:39:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.100+0000 7f8357400700 1 -- 172.21.15.16:0/3004512173 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350017650 con 0x7f8360101050 2024-04-05T23:39:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.100+0000 7f8365a00700 1 -- 172.21.15.16:0/3004512173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83601a60e0 con 0x7f8360101050 2024-04-05T23:39:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.100+0000 7f8365a00700 1 -- 172.21.15.16:0/3004512173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83601a64b0 con 0x7f8360101050 2024-04-05T23:39:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.105+0000 7f8357400700 1 -- 172.21.15.16:0/3004512173 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f83500225e0 con 0x7f8360101050 2024-04-05T23:39:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8379497b80 1 Processor -- start 2024-04-05T23:39:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8379497b80 1 -- start start 2024-04-05T23:39:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8357400700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83440a1290 0x7f83440a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8357400700 1 -- 172.21.15.16:0/3004512173 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f83500cde70 con 0x7f8360101050 2024-04-05T23:39:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8379497b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d12854ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f835f400700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83440a1290 0x7f83440a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8379497b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1285504a0 0x55d12854a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8379497b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d12854a5c0 0x55d12854aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8379497b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1284b0b80 con 0x55d12854fb80 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8379497b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1284a1930 con 0x55d12854a5c0 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.106+0000 7f8379497b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12849f2f0 con 0x55d1285504a0 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d12854ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d12854ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44470/0 (socket says 172.21.15.16:44470) 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835ea00700 1 -- 172.21.15.16:0/664980582 learned_addr learned my addr 172.21.15.16:0/664980582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835f400700 1 --2- 172.21.15.16:0/664980582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d12854a5c0 0x55d12854aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f8357e00700 1 --2- 172.21.15.16:0/664980582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1285504a0 0x55d12854a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835ea00700 1 -- 172.21.15.16:0/664980582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1285504a0 msgr2=0x55d12854a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835ea00700 1 --2- 172.21.15.16:0/664980582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1285504a0 0x55d12854a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835ea00700 1 -- 172.21.15.16:0/664980582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d12854a5c0 msgr2=0x55d12854aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835ea00700 1 --2- 172.21.15.16:0/664980582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d12854a5c0 0x55d12854aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835ea00700 1 -- 172.21.15.16:0/664980582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d12831c350 con 0x55d12854fb80 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835f400700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83440a1290 0x7f83440a3750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8358009fd0 tx=0x7f8358007b00 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.107+0000 7f835ea00700 1 --2- 172.21.15.16:0/664980582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d12854ff60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f83500a2ea0 tx=0x7f8350009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe989f1f31c179b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f834fe00700 1 -- 172.21.15.16:0/664980582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835009b6e0 con 0x55d12854fb80 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f834fe00700 1 -- 172.21.15.16:0/664980582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f835009afe0 con 0x55d12854fb80 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f834fe00700 1 -- 172.21.15.16:0/664980582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835009b240 con 0x55d12854fb80 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f8379497b80 1 -- 172.21.15.16:0/664980582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 msgr2=0x55d12854ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f8379497b80 1 --2- 172.21.15.16:0/664980582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d12854ff60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f83500a2ea0 tx=0x7f8350009ca0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f8379497b80 1 -- 172.21.15.16:0/664980582 shutdown_connections 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f8379497b80 1 --2- 172.21.15.16:0/664980582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d12854ff60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f8379497b80 1 --2- 172.21.15.16:0/664980582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d12854a5c0 0x55d12854aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f8379497b80 1 --2- 172.21.15.16:0/664980582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1285504a0 0x55d12854a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.108+0000 7f8379497b80 1 -- 172.21.15.16:0/664980582 >> 172.21.15.16:0/664980582 conn(0x55d1284be1c0 msgr2=0x55d128544be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.109+0000 7f8379497b80 1 -- 172.21.15.16:0/664980582 shutdown_connections 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.109+0000 7f8379497b80 1 -- 172.21.15.16:0/664980582 wait complete. 2024-04-05T23:39:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.109+0000 7f8379497b80 1 -- start start 2024-04-05T23:39:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.109+0000 7f8379497b80 1 Processor -- start 2024-04-05T23:39:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.110+0000 7f8379497b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854a5c0 0x55d128556470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.110+0000 7f8379497b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d128553740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.110+0000 7f8379497b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d128553d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.110+0000 7f8379497b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12849abe0 con 0x55d12854fb80 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.110+0000 7f8379497b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d128436d20 con 0x55d1285504a0 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.110+0000 7f8379497b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1284369c0 con 0x55d12854a5c0 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.110+0000 7f8379497b80 10 client.0 _ll_register_callbacks cb 0x55d1282a2340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f835ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854a5c0 0x55d128556470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f835f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d128553d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f8357e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d128553740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f835f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d128553d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:53090/0 (socket says 172.21.15.16:53090) 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f835f400700 1 -- 172.21.15.16:0/2871349286 learned_addr learned my addr 172.21.15.16:0/2871349286 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f8357e00700 1 -- 172.21.15.16:0/2871349286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854a5c0 msgr2=0x55d128556470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f8357e00700 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854a5c0 0x55d128556470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f8357e00700 1 -- 172.21.15.16:0/2871349286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 msgr2=0x55d128553d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f8357e00700 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d128553d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f8357e00700 1 -- 172.21.15.16:0/2871349286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83601a64b0 con 0x55d12854fb80 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f835f400700 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d128553d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f8357e00700 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d128553740 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f836019aaa0 tx=0x7f8348012a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75384fe2b32a011a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f834cc00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.111+0000 7f834cc00700 1 -- 172.21.15.16:0/2871349286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348017070 con 0x55d12854fb80 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f834cc00700 1 -- 172.21.15.16:0/2871349286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f834800f480 con 0x55d12854fb80 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f8379497b80 1 -- 172.21.15.16:0/2871349286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d12853d1e0 con 0x55d12854fb80 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f834cc00700 1 -- 172.21.15.16:0/2871349286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f834800f640 con 0x55d12854fb80 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f8379497b80 10 client.16257 fetch_fsmap learned FSMap version 48 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f8379497b80 1 -- 172.21.15.16:0/2871349286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) v3 -- 0x55d1282c8b00 con 0x55d12854fb80 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f834cc00700 1 -- 172.21.15.16:0/2871349286 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8348028080 con 0x55d12854fb80 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f834cc00700 1 -- 172.21.15.16:0/2871349286 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 48) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f8348010830 con 0x55d12854fb80 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f8379497b80 10 client.16257 fetch_fsmap finished waiting for FSMap version 48 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f8379497b80 10 client.16257 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:39:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f8379497b80 20 client.16257 populate_metadata read hostname 'smithi016' 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f8379497b80 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55d128563da0 0x55d1285661c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f8379497b80 4 client.16257 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.112+0000 7f8379497b80 1 -- 172.21.15.16:0/2871349286 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1284936a0 con 0x55d128563da0 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.113+0000 7f835ea00700 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55d128563da0 0x55d1285661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.113+0000 7f835ea00700 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55d128563da0 0x55d1285661c0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f15a7b99142204 server_cookie=52b172ef351ce181 in_seq=0 out_seq=0 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.113+0000 7f834cc00700 10 client.16257 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f834cc00700 1 -- 172.21.15.16:0/2871349286 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d1284936a0 con 0x55d128563da0 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f834cc00700 10 client.16257 handle_command_reply: tid=0 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 client.16257 shutdown 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 -- 172.21.15.16:0/2871349286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 msgr2=0x55d128553740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d128553740 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f836019aaa0 tx=0x7f8348012a20 comp rx=0 tx=0).stop 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 -- 172.21.15.16:0/2871349286 shutdown_connections 2024-04-05T23:39:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55d128563da0 0x55d1285661c0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d128553d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f834cc00700 0 client.16257 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854a5c0 0x55d128556470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 --2- 172.21.15.16:0/2871349286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854fb80 0x55d128553740 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 -- 172.21.15.16:0/2871349286 >> 172.21.15.16:0/2871349286 conn(0x55d1284be1c0 msgr2=0x55d128544280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 -- 172.21.15.16:0/2871349286 shutdown_connections 2024-04-05T23:39:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.114+0000 7f8379497b80 1 -- 172.21.15.16:0/2871349286 wait complete. 2024-04-05T23:39:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.115+0000 7f8379497b80 20 client.16257 trim_cache size 0 max 16384 2024-04-05T23:39:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.130+0000 7f8379497b80 1 Processor -- start 2024-04-05T23:39:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.130+0000 7f8379497b80 1 -- start start 2024-04-05T23:39:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.130+0000 7f8379497b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d128553700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.130+0000 7f8379497b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d128552fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.130+0000 7f8379497b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d12854a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.130+0000 7f8379497b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12831e640 con 0x55d12854a5c0 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.130+0000 7f8379497b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1285612a0 con 0x55d1285504a0 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.130+0000 7f8379497b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d128546450 con 0x55d12854fb80 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d128553700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f8357e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d128552fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d128553700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44492/0 (socket says 172.21.15.16:44492) 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835ea00700 1 -- 172.21.15.16:0/2347782357 learned_addr learned my addr 172.21.15.16:0/2347782357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835f400700 1 --2- 172.21.15.16:0/2347782357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d12854a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835ea00700 1 -- 172.21.15.16:0/2347782357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 msgr2=0x55d128552fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835ea00700 1 --2- 172.21.15.16:0/2347782357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d128552fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835ea00700 1 -- 172.21.15.16:0/2347782357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 msgr2=0x55d12854a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835ea00700 1 --2- 172.21.15.16:0/2347782357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d12854a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835ea00700 1 -- 172.21.15.16:0/2347782357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d128460e40 con 0x55d12854a5c0 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f835ea00700 1 --2- 172.21.15.16:0/2347782357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d128553700 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f835009af70 tx=0x7f83500a1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6377ec931eedce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.131+0000 7f8354c00700 1 -- 172.21.15.16:0/2347782357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83500a33d0 con 0x55d12854a5c0 2024-04-05T23:39:14.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8354c00700 1 -- 172.21.15.16:0/2347782357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83500a10b0 con 0x55d12854a5c0 2024-04-05T23:39:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8354c00700 1 -- 172.21.15.16:0/2347782357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835009a4d0 con 0x55d12854a5c0 2024-04-05T23:39:14.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8379497b80 1 -- 172.21.15.16:0/2347782357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 msgr2=0x55d128553700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:14.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8379497b80 1 --2- 172.21.15.16:0/2347782357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d128553700 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f835009af70 tx=0x7f83500a1040 comp rx=0 tx=0).stop 2024-04-05T23:39:14.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8379497b80 1 -- 172.21.15.16:0/2347782357 shutdown_connections 2024-04-05T23:39:14.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8379497b80 1 --2- 172.21.15.16:0/2347782357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d128553700 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8379497b80 1 --2- 172.21.15.16:0/2347782357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d12854a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8379497b80 1 --2- 172.21.15.16:0/2347782357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d128552fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8379497b80 1 -- 172.21.15.16:0/2347782357 >> 172.21.15.16:0/2347782357 conn(0x55d1284be1c0 msgr2=0x55d12856aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8379497b80 1 -- 172.21.15.16:0/2347782357 shutdown_connections 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.132+0000 7f8379497b80 1 -- 172.21.15.16:0/2347782357 wait complete. 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f8379497b80 1 -- start start 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f8379497b80 1 Processor -- start 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f8379497b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d12855aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f8379497b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d12854b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f8379497b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d12854bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f8379497b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12854c2c0 con 0x55d12854a5c0 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f8379497b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12854c430 con 0x55d1285504a0 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f8379497b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12854c5f0 con 0x55d12854fb80 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f8379497b80 10 client.0 _ll_register_callbacks cb 0x7f8348009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.133+0000 7f835ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d12855aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f8357e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d12854b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f8357e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d12854b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:34658/0 (socket says 172.21.15.16:34658) 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f835ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d12855aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44504/0 (socket says 172.21.15.16:44504) 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f8357e00700 1 -- 172.21.15.16:0/1704219151 learned_addr learned my addr 172.21.15.16:0/1704219151 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f835f400700 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d12854bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f8357e00700 1 -- 172.21.15.16:0/1704219151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 msgr2=0x55d12854bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f8357e00700 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d12854bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f8357e00700 1 -- 172.21.15.16:0/1704219151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 msgr2=0x55d12855aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f8357e00700 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d12855aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f8357e00700 1 -- 172.21.15.16:0/1704219151 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d12831c350 con 0x55d12854fb80 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f835ea00700 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d12855aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f8357e00700 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d12854b5a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8348000c00 tx=0x7f8348011ff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3e11568cd08031e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.134+0000 7f834cc00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.135+0000 7f834cc00700 1 -- 172.21.15.16:0/1704219151 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348017040 con 0x55d12854fb80 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.135+0000 7f834cc00700 1 -- 172.21.15.16:0/1704219151 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f834802b070 con 0x55d12854fb80 2024-04-05T23:39:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.135+0000 7f8379497b80 1 -- 172.21.15.16:0/1704219151 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f835009afe0 con 0x55d12854fb80 2024-04-05T23:39:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.135+0000 7f834cc00700 1 -- 172.21.15.16:0/1704219151 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83480109e0 con 0x55d12854fb80 2024-04-05T23:39:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.135+0000 7f8379497b80 10 client.26347 fetch_fsmap learned FSMap version 48 2024-04-05T23:39:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.135+0000 7f8379497b80 1 -- 172.21.15.16:0/1704219151 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) v3 -- 0x55d128320c70 con 0x55d12854fb80 2024-04-05T23:39:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.136+0000 7f834cc00700 1 -- 172.21.15.16:0/1704219151 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f835009afe0 con 0x55d12854fb80 2024-04-05T23:39:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.136+0000 7f834cc00700 1 -- 172.21.15.16:0/1704219151 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 48) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f8348003bc0 con 0x55d12854fb80 2024-04-05T23:39:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.136+0000 7f8379497b80 10 client.26347 fetch_fsmap finished waiting for FSMap version 48 2024-04-05T23:39:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.136+0000 7f8379497b80 10 client.26347 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:39:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.136+0000 7f8379497b80 20 client.26347 populate_metadata read hostname 'smithi016' 2024-04-05T23:39:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.136+0000 7f8379497b80 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55d1285b84a0 0x55d1285ba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.136+0000 7f8379497b80 4 client.26347 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:39:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.136+0000 7f8379497b80 1 -- 172.21.15.16:0/1704219151 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d1282e9c90 con 0x55d1285b84a0 2024-04-05T23:39:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.136+0000 7f835ea00700 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55d1285b84a0 0x55d1285ba8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.137+0000 7f835ea00700 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55d1285b84a0 0x55d1285ba8c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=875b0655348fa9f4 server_cookie=c77608e78eb06681 in_seq=0 out_seq=0 2024-04-05T23:39:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.137+0000 7f834cc00700 10 client.26347 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:14.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f834cc00700 1 -- 172.21.15.16:0/1704219151 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d1282e9c90 con 0x55d1285b84a0 2024-04-05T23:39:14.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f834cc00700 10 client.26347 handle_command_reply: tid=0 2024-04-05T23:39:14.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 client.26347 shutdown 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 -- 172.21.15.16:0/1704219151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 msgr2=0x55d12854b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d12854b5a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8348000c00 tx=0x7f8348011ff0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 -- 172.21.15.16:0/1704219151 shutdown_connections 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55d1285b84a0 0x55d1285ba8c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f834cc00700 0 client.26347 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d12854a5c0 0x55d12855aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d1285504a0 0x55d12854bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 --2- 172.21.15.16:0/1704219151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d12854fb80 0x55d12854b5a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 -- 172.21.15.16:0/1704219151 >> 172.21.15.16:0/1704219151 conn(0x55d1284be1c0 msgr2=0x55d12854a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 -- 172.21.15.16:0/1704219151 shutdown_connections 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.167+0000 7f8379497b80 1 -- 172.21.15.16:0/1704219151 wait complete. 2024-04-05T23:39:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8379497b80 20 client.26347 trim_cache size 0 max 16384 2024-04-05T23:39:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:39:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:39:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:39:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:39:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 -- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83440a1290 msgr2=0x7f83440a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83440a1290 0x7f83440a3750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8358009fd0 tx=0x7f8358007b00 comp rx=0 tx=0).stop 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 -- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 msgr2=0x7f83601995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 0x7f83601995a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8350000c00 tx=0x7f8350005740 comp rx=0 tx=0).stop 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 -- 172.21.15.16:0/3004512173 shutdown_connections 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f83440a1290 0x7f83440a3750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8360101050 0x7f83601995a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f836019de30 0x7f836019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 --2- 172.21.15.16:0/3004512173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8360106170 0x7f8360199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 -- 172.21.15.16:0/3004512173 >> 172.21.15.16:0/3004512173 conn(0x7f83600fc8d0 msgr2=0x7f83600fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 -- 172.21.15.16:0/3004512173 shutdown_connections 2024-04-05T23:39:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:14.168+0000 7f8365a00700 1 -- 172.21.15.16:0/3004512173 wait complete. 2024-04-05T23:39:14.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:14 smithi016 ceph-mon[38248]: pgmap v175: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 167 op/s 2024-04-05T23:39:14.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:14 smithi130 ceph-mon[79417]: pgmap v175: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 167 op/s 2024-04-05T23:39:14.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:14 smithi027 ceph-mon[78597]: pgmap v175: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 167 op/s 2024-04-05T23:39:15.001 INFO:tasks.workunit.client.0.smithi016.stdout: 1 329610 7.33 MB/sec execute 154 sec latency 2015.789 ms 2024-04-05T23:39:16.001 INFO:tasks.workunit.client.0.smithi016.stdout: 1 329610 7.28 MB/sec execute 155 sec latency 3015.888 ms 2024-04-05T23:39:16.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:16 smithi016 ceph-mon[38248]: pgmap v176: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 167 op/s 2024-04-05T23:39:16.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:16 smithi130 ceph-mon[79417]: pgmap v176: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 167 op/s 2024-04-05T23:39:16.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:16 smithi027 ceph-mon[78597]: pgmap v176: 97 pgs: 97 active+clean; 118 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 167 op/s 2024-04-05T23:39:17.001 INFO:tasks.workunit.client.0.smithi016.stdout: 1 329610 7.23 MB/sec execute 156 sec latency 4016.007 ms 2024-04-05T23:39:18.001 INFO:tasks.workunit.client.0.smithi016.stdout: 1 329610 7.19 MB/sec execute 157 sec latency 5016.112 ms 2024-04-05T23:39:18.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:18 smithi016 ceph-mon[38248]: pgmap v177: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-04-05T23:39:18.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:18 smithi130 ceph-mon[79417]: pgmap v177: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-04-05T23:39:18.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:18 smithi027 ceph-mon[78597]: pgmap v177: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-04-05T23:39:19.002 INFO:tasks.workunit.client.0.smithi016.stdout: 1 329610 7.14 MB/sec execute 158 sec latency 6016.281 ms 2024-04-05T23:39:20.002 INFO:tasks.workunit.client.0.smithi016.stdout: 1 329610 7.09 MB/sec execute 159 sec latency 7016.421 ms 2024-04-05T23:39:20.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:20 smithi016 ceph-mon[38248]: pgmap v178: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 942 KiB/s wr, 94 op/s 2024-04-05T23:39:20.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:20 smithi130 ceph-mon[79417]: pgmap v178: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 942 KiB/s wr, 94 op/s 2024-04-05T23:39:20.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:20 smithi027 ceph-mon[78597]: pgmap v178: 97 pgs: 97 active+clean; 119 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 942 KiB/s wr, 94 op/s 2024-04-05T23:39:21.002 INFO:tasks.workunit.client.0.smithi016.stdout: 1 330608 7.08 MB/sec execute 160 sec latency 7112.799 ms 2024-04-05T23:39:22.002 INFO:tasks.workunit.client.0.smithi016.stdout: 1 331577 7.07 MB/sec execute 161 sec latency 382.920 ms 2024-04-05T23:39:22.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:22 smithi016 ceph-mon[38248]: pgmap v179: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.2 MiB/s wr, 119 op/s 2024-04-05T23:39:22.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:22 smithi130 ceph-mon[79417]: pgmap v179: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.2 MiB/s wr, 119 op/s 2024-04-05T23:39:22.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:22 smithi027 ceph-mon[78597]: pgmap v179: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.2 MiB/s wr, 119 op/s 2024-04-05T23:39:23.002 INFO:tasks.workunit.client.0.smithi016.stdout: 1 333241 7.08 MB/sec execute 162 sec latency 591.912 ms 2024-04-05T23:39:24.002 INFO:tasks.workunit.client.0.smithi016.stdout: 1 334365 7.07 MB/sec execute 163 sec latency 11.589 ms 2024-04-05T23:39:24.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:24 smithi016 ceph-mon[38248]: pgmap v180: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 649 KiB/s wr, 61 op/s 2024-04-05T23:39:24.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:24 smithi130 ceph-mon[79417]: pgmap v180: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 649 KiB/s wr, 61 op/s 2024-04-05T23:39:24.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:24 smithi027 ceph-mon[78597]: pgmap v180: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 649 KiB/s wr, 61 op/s 2024-04-05T23:39:25.002 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335120 7.06 MB/sec execute 164 sec latency 503.279 ms 2024-04-05T23:39:26.002 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335120 7.01 MB/sec execute 165 sec latency 1503.425 ms 2024-04-05T23:39:26.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:26 smithi016 ceph-mon[38248]: pgmap v181: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 649 KiB/s wr, 61 op/s 2024-04-05T23:39:26.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:26 smithi130 ceph-mon[79417]: pgmap v181: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 649 KiB/s wr, 61 op/s 2024-04-05T23:39:26.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:26 smithi027 ceph-mon[78597]: pgmap v181: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 649 KiB/s wr, 61 op/s 2024-04-05T23:39:27.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335796 6.98 MB/sec execute 166 sec latency 2230.426 ms 2024-04-05T23:39:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.329+0000 7f0e3b200700 1 -- 172.21.15.16:0/1543549503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e341019e0 msgr2=0x7f0e34105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.329+0000 7f0e3b200700 1 --2- 172.21.15.16:0/1543549503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e341019e0 0x7f0e34105a20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009990 tx=0x7f0e20009ca0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.329+0000 7f0e3b200700 1 -- 172.21.15.16:0/1543549503 shutdown_connections 2024-04-05T23:39:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.329+0000 7f0e3b200700 1 --2- 172.21.15.16:0/1543549503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e34106150 0x7f0e3410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.329+0000 7f0e3b200700 1 --2- 172.21.15.16:0/1543549503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e34101030 0x7f0e34101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.329+0000 7f0e3b200700 1 --2- 172.21.15.16:0/1543549503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e341019e0 0x7f0e34105a20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.329+0000 7f0e3b200700 1 -- 172.21.15.16:0/1543549503 >> 172.21.15.16:0/1543549503 conn(0x7f0e340fc8b0 msgr2=0x7f0e340fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.329+0000 7f0e3b200700 1 -- 172.21.15.16:0/1543549503 shutdown_connections 2024-04-05T23:39:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.329+0000 7f0e3b200700 1 -- 172.21.15.16:0/1543549503 wait complete. 2024-04-05T23:39:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.330+0000 7f0e3b200700 1 Processor -- start 2024-04-05T23:39:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.331+0000 7f0e3b200700 1 -- start start 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.331+0000 7f0e3b200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e34101030 0x7f0e341972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.331+0000 7f0e3b200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e34106150 0x7f0e34197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.331+0000 7f0e3b200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e3419bbd0 0x7f0e3419c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.331+0000 7f0e3b200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3419c650 con 0x7f0e3419bbd0 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.331+0000 7f0e3b200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3419c7c0 con 0x7f0e34106150 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.331+0000 7f0e3b200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3419c980 con 0x7f0e34101030 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.331+0000 7f0e33e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e34101030 0x7f0e341972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e33e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e34101030 0x7f0e341972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39030/0 (socket says 172.21.15.16:39030) 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e33e00700 1 -- 172.21.15.16:0/10021190 learned_addr learned my addr 172.21.15.16:0/10021190 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e33400700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e34106150 0x7f0e34197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e39200700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e3419bbd0 0x7f0e3419c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e33400700 1 -- 172.21.15.16:0/10021190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e34101030 msgr2=0x7f0e341972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e33400700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e34101030 0x7f0e341972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e33400700 1 -- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e3419bbd0 msgr2=0x7f0e3419c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e33400700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e3419bbd0 0x7f0e3419c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e33400700 1 -- 172.21.15.16:0/10021190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e200095f0 con 0x7f0e34106150 2024-04-05T23:39:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.332+0000 7f0e33400700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e34106150 0x7f0e34197830 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0e20000c00 tx=0x7f0e20009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.333+0000 7f0e30c00700 1 -- 172.21.15.16:0/10021190 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2001d070 con 0x7f0e34106150 2024-04-05T23:39:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.333+0000 7f0e30c00700 1 -- 172.21.15.16:0/10021190 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e20021410 con 0x7f0e34106150 2024-04-05T23:39:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.333+0000 7f0e30c00700 1 -- 172.21.15.16:0/10021190 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20017650 con 0x7f0e34106150 2024-04-05T23:39:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.333+0000 7f0e3b200700 1 -- 172.21.15.16:0/10021190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e341a3e80 con 0x7f0e34106150 2024-04-05T23:39:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.333+0000 7f0e3b200700 1 -- 172.21.15.16:0/10021190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e341a4240 con 0x7f0e34106150 2024-04-05T23:39:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.336+0000 7f0e30c00700 1 -- 172.21.15.16:0/10021190 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f0e20021780 con 0x7f0e34106150 2024-04-05T23:39:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.337+0000 7f0e30c00700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e180a1310 0x7f0e180a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.337+0000 7f0e30c00700 1 -- 172.21.15.16:0/10021190 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f0e200ce070 con 0x7f0e34106150 2024-04-05T23:39:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.337+0000 7f0e4edf7b80 1 Processor -- start 2024-04-05T23:39:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.337+0000 7f0e33e00700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e180a1310 0x7f0e180a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.337+0000 7f0e4edf7b80 1 -- start start 2024-04-05T23:39:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e1129f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e1124030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e1124a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e108ab80 con 0x55b2e112a4a0 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e107b930 con 0x55b2e11245c0 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e10792f0 con 0x55b2e1129b80 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e33e00700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e180a1310 0x7f0e180a37d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009c90 tx=0x7f0e24009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e33e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e1129f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e39200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e1124a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e33400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e1124030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e33e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e1129f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39040/0 (socket says 172.21.15.16:39040) 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e33e00700 1 -- 172.21.15.16:0/3891727134 learned_addr learned my addr 172.21.15.16:0/3891727134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.338+0000 7f0e39200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e1124a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37226/0 (socket says 172.21.15.16:37226) 2024-04-05T23:39:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e33e00700 1 -- 172.21.15.16:0/3891727134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 msgr2=0x55b2e1124a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e33e00700 1 --2- 172.21.15.16:0/3891727134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e1124a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e33e00700 1 -- 172.21.15.16:0/3891727134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 msgr2=0x55b2e1124030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e33e00700 1 --2- 172.21.15.16:0/3891727134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e1124030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e33e00700 1 -- 172.21.15.16:0/3891727134 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2e0ef6350 con 0x55b2e1129b80 2024-04-05T23:39:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e33400700 1 --2- 172.21.15.16:0/3891727134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e1124030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e33e00700 1 --2- 172.21.15.16:0/3891727134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e1129f60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0e24011f80 tx=0x7f0e24014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cc942ff8f78263f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e29600700 1 -- 172.21.15.16:0/3891727134 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24012dc0 con 0x55b2e1129b80 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e29600700 1 -- 172.21.15.16:0/3891727134 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e24016070 con 0x55b2e1129b80 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.339+0000 7f0e29600700 1 -- 172.21.15.16:0/3891727134 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24027620 con 0x55b2e1129b80 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.340+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3891727134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 msgr2=0x55b2e1129f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.340+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/3891727134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e1129f60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0e24011f80 tx=0x7f0e24014000 comp rx=0 tx=0).stop 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.340+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3891727134 shutdown_connections 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.340+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/3891727134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e1124a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.340+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/3891727134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e1129f60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.340+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/3891727134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e1124030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.340+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3891727134 >> 172.21.15.16:0/3891727134 conn(0x55b2e10981c0 msgr2=0x55b2e111ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.340+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3891727134 shutdown_connections 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.340+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3891727134 wait complete. 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.341+0000 7f0e4edf7b80 1 -- start start 2024-04-05T23:39:27.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.341+0000 7f0e4edf7b80 1 Processor -- start 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.341+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e112d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.341+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e112dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.341+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e112a4a0 0x55b2e112e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.341+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e1074be0 con 0x55b2e11245c0 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.341+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e1010d20 con 0x55b2e112a4a0 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.341+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e10109c0 con 0x55b2e1129b80 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.341+0000 7f0e4edf7b80 10 client.0 _ll_register_callbacks cb 0x55b2e0e7c340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e112d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e112d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51930/0 (socket says 172.21.15.16:51930) 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33e00700 1 -- 172.21.15.16:0/1437879268 learned_addr learned my addr 172.21.15.16:0/1437879268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e112dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e39200700 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e112a4a0 0x55b2e112e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33e00700 1 -- 172.21.15.16:0/1437879268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 msgr2=0x55b2e112dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33e00700 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e112dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33e00700 1 -- 172.21.15.16:0/1437879268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e112a4a0 msgr2=0x55b2e112e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33e00700 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e112a4a0 0x55b2e112e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33e00700 1 -- 172.21.15.16:0/1437879268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e341a4240 con 0x55b2e11245c0 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e39200700 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e112a4a0 0x55b2e112e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e33e00700 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e112d850 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0e24012a10 tx=0x7f0e2401f650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffe95cd46414170 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.342+0000 7f0e1e000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e1e000700 1 -- 172.21.15.16:0/1437879268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2402c070 con 0x55b2e11245c0 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1437879268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b2e11171e0 con 0x55b2e11245c0 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e1e000700 1 -- 172.21.15.16:0/1437879268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f0e24018070 con 0x55b2e11245c0 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e1e000700 1 -- 172.21.15.16:0/1437879268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e24009e80 con 0x55b2e11245c0 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e4edf7b80 10 client.16275 fetch_fsmap learned FSMap version 48 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1437879268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) v3 -- 0x7f0e24011c60 con 0x55b2e11245c0 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e1e000700 1 -- 172.21.15.16:0/1437879268 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e2403f020 con 0x55b2e11245c0 2024-04-05T23:39:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e1e000700 1 -- 172.21.15.16:0/1437879268 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 48) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f0e24025020 con 0x55b2e11245c0 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e4edf7b80 10 client.16275 fetch_fsmap finished waiting for FSMap version 48 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.343+0000 7f0e4edf7b80 10 client.16275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.344+0000 7f0e4edf7b80 20 client.16275 populate_metadata read hostname 'smithi016' 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.344+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2e1146730 0x55b2e1148b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.344+0000 7f0e4edf7b80 4 client.16275 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.344+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1437879268 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b2e106d6a0 con 0x55b2e1146730 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.344+0000 7f0e33400700 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2e1146730 0x55b2e1148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.344+0000 7f0e33400700 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2e1146730 0x55b2e1148b50 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5710ec9b53f9575b server_cookie=88d2b53fdfaa7c64 in_seq=0 out_seq=0 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.344+0000 7f0e1e000700 10 client.16275 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e1e000700 1 -- 172.21.15.16:0/1437879268 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b2e106d6a0 con 0x55b2e1146730 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e1e000700 10 client.16275 handle_command_reply: tid=0 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 client.16275 shutdown 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1437879268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 msgr2=0x55b2e112d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e112d850 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0e24012a10 tx=0x7f0e2401f650 comp rx=0 tx=0).stop 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1437879268 shutdown_connections 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2e1146730 0x55b2e1148b50 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e112d850 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e112a4a0 0x55b2e112e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e1e000700 0 client.16275 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1437879268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e112dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1437879268 >> 172.21.15.16:0/1437879268 conn(0x55b2e10981c0 msgr2=0x55b2e111e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1437879268 shutdown_connections 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.345+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1437879268 wait complete. 2024-04-05T23:39:27.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.346+0000 7f0e4edf7b80 20 client.16275 trim_cache size 0 max 16384 2024-04-05T23:39:27.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.361+0000 7f0e4edf7b80 1 Processor -- start 2024-04-05T23:39:27.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.361+0000 7f0e4edf7b80 1 -- start start 2024-04-05T23:39:27.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e113d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e112c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e111ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e0ef8640 con 0x55b2e112a4a0 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e1130a00 con 0x55b2e11245c0 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e111baa0 con 0x55b2e1129b80 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e113d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e113d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37240/0 (socket says 172.21.15.16:37240) 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33e00700 1 -- 172.21.15.16:0/3187193191 learned_addr learned my addr 172.21.15.16:0/3187193191 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33400700 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e112c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e39200700 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e111ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33e00700 1 -- 172.21.15.16:0/3187193191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 msgr2=0x55b2e112c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33e00700 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e112c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33e00700 1 -- 172.21.15.16:0/3187193191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 msgr2=0x55b2e111ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33e00700 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e111ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33e00700 1 -- 172.21.15.16:0/3187193191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2e0ea2b00 con 0x55b2e11245c0 2024-04-05T23:39:27.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e39200700 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e111ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:27.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.362+0000 7f0e33400700 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e112c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:27.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.363+0000 7f0e33e00700 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e113d810 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005a80 tx=0x7f0e24009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dacd014e26f0fd1e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.363+0000 7f0e2a000700 1 -- 172.21.15.16:0/3187193191 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2402c070 con 0x55b2e11245c0 2024-04-05T23:39:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.363+0000 7f0e2a000700 1 -- 172.21.15.16:0/3187193191 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e2403f070 con 0x55b2e11245c0 2024-04-05T23:39:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.363+0000 7f0e2a000700 1 -- 172.21.15.16:0/3187193191 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e240052a0 con 0x55b2e11245c0 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.363+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3187193191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 msgr2=0x55b2e113d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.363+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e113d810 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005a80 tx=0x7f0e24009b00 comp rx=0 tx=0).stop 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.364+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3187193191 shutdown_connections 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.364+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e112a4a0 0x55b2e111ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.364+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e11245c0 0x55b2e113d810 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.364+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/3187193191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e1129b80 0x55b2e112c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.364+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3187193191 >> 172.21.15.16:0/3187193191 conn(0x55b2e10981c0 msgr2=0x55b2e112c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.364+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3187193191 shutdown_connections 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.364+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/3187193191 wait complete. 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.364+0000 7f0e4edf7b80 1 -- start start 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.364+0000 7f0e4edf7b80 1 Processor -- start 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e113d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e1129b80 0x55b2e113ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e4edf7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e112a4a0 0x55b2e113b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e113b9c0 con 0x55b2e11245c0 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e113bb30 con 0x55b2e1129b80 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e4edf7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2e1199560 con 0x55b2e112a4a0 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e4edf7b80 10 client.0 _ll_register_callbacks cb 0x7f0e20005c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e1129b80 0x55b2e113ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e1129b80 0x55b2e113ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37250/0 (socket says 172.21.15.16:37250) 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33400700 1 -- 172.21.15.16:0/1490641031 learned_addr learned my addr 172.21.15.16:0/1490641031 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33e00700 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e113d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e39200700 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e112a4a0 0x55b2e113b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33400700 1 -- 172.21.15.16:0/1490641031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e112a4a0 msgr2=0x55b2e113b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33400700 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e112a4a0 0x55b2e113b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33400700 1 -- 172.21.15.16:0/1490641031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 msgr2=0x55b2e113d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33400700 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e113d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33400700 1 -- 172.21.15.16:0/1490641031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e341a3e80 con 0x55b2e1129b80 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.365+0000 7f0e33e00700 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e113d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e33400700 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e1129b80 0x55b2e113ad50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000b2e0 tx=0x7f0e2000f840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2754bfe3bcb3598 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e1e000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e1e000700 1 -- 172.21.15.16:0/1490641031 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2000fcd0 con 0x55b2e1129b80 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1490641031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e24016070 con 0x55b2e1129b80 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e1e000700 1 -- 172.21.15.16:0/1490641031 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f0e20021580 con 0x55b2e1129b80 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e1e000700 1 -- 172.21.15.16:0/1490641031 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e2009b8e0 con 0x55b2e1129b80 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e4edf7b80 10 client.26333 fetch_fsmap learned FSMap version 48 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1490641031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) v3 -- 0x7f0e2c007500 con 0x55b2e1129b80 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e1e000700 1 -- 172.21.15.16:0/1490641031 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e24016070 con 0x55b2e1129b80 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e1e000700 1 -- 172.21.15.16:0/1490641031 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 48) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f0e2009b440 con 0x55b2e1129b80 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e4edf7b80 10 client.26333 fetch_fsmap finished waiting for FSMap version 48 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e4edf7b80 10 client.26333 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:39:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e4edf7b80 20 client.26333 populate_metadata read hostname 'smithi016' 2024-04-05T23:39:27.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2e119aad0 0x55b2e119cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:27.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e4edf7b80 4 client.26333 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:39:27.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.366+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1490641031 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b2e0ec3c90 con 0x55b2e119aad0 2024-04-05T23:39:27.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.367+0000 7f0e33e00700 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2e119aad0 0x55b2e119cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:27.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.367+0000 7f0e33e00700 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2e119aad0 0x55b2e119cef0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d27b41cbea546f1 server_cookie=9707de282ef757ac in_seq=0 out_seq=0 2024-04-05T23:39:27.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.367+0000 7f0e1e000700 10 client.26333 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.393+0000 7f0e1e000700 1 -- 172.21.15.16:0/1490641031 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b2e0ec3c90 con 0x55b2e119aad0 2024-04-05T23:39:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.393+0000 7f0e1e000700 10 client.26333 handle_command_reply: tid=0 2024-04-05T23:39:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.393+0000 7f0e4edf7b80 1 client.26333 shutdown 2024-04-05T23:39:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1490641031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e1129b80 msgr2=0x55b2e113ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e1129b80 0x55b2e113ad50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000b2e0 tx=0x7f0e2000f840 comp rx=0 tx=0).stop 2024-04-05T23:39:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1490641031 shutdown_connections 2024-04-05T23:39:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2e119aad0 0x55b2e119cef0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2e11245c0 0x55b2e113d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e1e000700 0 client.26333 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2e112a4a0 0x55b2e113b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 --2- 172.21.15.16:0/1490641031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2e1129b80 0x55b2e113ad50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1490641031 >> 172.21.15.16:0/1490641031 conn(0x55b2e10981c0 msgr2=0x55b2e112e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1490641031 shutdown_connections 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 1 -- 172.21.15.16:0/1490641031 wait complete. 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.394+0000 7f0e4edf7b80 20 client.26333 trim_cache size 0 max 16384 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:39:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:39:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 -- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e180a1310 msgr2=0x7f0e180a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e180a1310 0x7f0e180a37d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009c90 tx=0x7f0e24009260 comp rx=0 tx=0).stop 2024-04-05T23:39:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 -- 172.21.15.16:0/10021190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e34106150 msgr2=0x7f0e34197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e34106150 0x7f0e34197830 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0e20000c00 tx=0x7f0e20009510 comp rx=0 tx=0).stop 2024-04-05T23:39:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 -- 172.21.15.16:0/10021190 shutdown_connections 2024-04-05T23:39:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f0e180a1310 0x7f0e180a37d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e3419bbd0 0x7f0e3419c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e34101030 0x7f0e341972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 --2- 172.21.15.16:0/10021190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e34106150 0x7f0e34197830 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 -- 172.21.15.16:0/10021190 >> 172.21.15.16:0/10021190 conn(0x7f0e340fc8b0 msgr2=0x7f0e340fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 -- 172.21.15.16:0/10021190 shutdown_connections 2024-04-05T23:39:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:27.395+0000 7f0e3b200700 1 -- 172.21.15.16:0/10021190 wait complete. 2024-04-05T23:39:28.002 INFO:tasks.workunit.client.0.smithi016.stdout: 1 337313 7.01 MB/sec execute 167 sec latency 21.732 ms 2024-04-05T23:39:28.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:28 smithi016 ceph-mon[38248]: pgmap v182: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-05T23:39:28.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:28 smithi130 ceph-mon[79417]: pgmap v182: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-05T23:39:28.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:28 smithi027 ceph-mon[78597]: pgmap v182: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-05T23:39:29.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338627 7.00 MB/sec execute 168 sec latency 18.821 ms 2024-04-05T23:39:30.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338670 6.96 MB/sec execute 169 sec latency 962.849 ms 2024-04-05T23:39:30.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:30 smithi016 ceph-mon[38248]: pgmap v183: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 990 KiB/s wr, 104 op/s 2024-04-05T23:39:30.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:30 smithi130 ceph-mon[79417]: pgmap v183: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 990 KiB/s wr, 104 op/s 2024-04-05T23:39:30.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:30 smithi027 ceph-mon[78597]: pgmap v183: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 990 KiB/s wr, 104 op/s 2024-04-05T23:39:31.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338670 6.92 MB/sec execute 170 sec latency 1962.926 ms 2024-04-05T23:39:32.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338670 6.88 MB/sec execute 171 sec latency 2962.987 ms 2024-04-05T23:39:32.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:32 smithi016 ceph-mon[38248]: pgmap v184: 97 pgs: 97 active+clean; 123 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-05T23:39:32.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:32 smithi130 ceph-mon[79417]: pgmap v184: 97 pgs: 97 active+clean; 123 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-05T23:39:32.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:32 smithi027 ceph-mon[78597]: pgmap v184: 97 pgs: 97 active+clean; 123 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-05T23:39:33.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 340347 6.89 MB/sec execute 172 sec latency 3172.911 ms 2024-04-05T23:39:34.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 341508 6.88 MB/sec execute 173 sec latency 12.832 ms 2024-04-05T23:39:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:34 smithi016 ceph-mon[38248]: pgmap v185: 97 pgs: 97 active+clean; 123 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-05T23:39:34.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:34 smithi130 ceph-mon[79417]: pgmap v185: 97 pgs: 97 active+clean; 123 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-05T23:39:34.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:34 smithi027 ceph-mon[78597]: pgmap v185: 97 pgs: 97 active+clean; 123 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-05T23:39:35.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 342210 6.87 MB/sec execute 174 sec latency 598.172 ms 2024-04-05T23:39:36.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 342210 6.83 MB/sec execute 175 sec latency 1598.308 ms 2024-04-05T23:39:36.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:36 smithi016 ceph-mon[38248]: pgmap v186: 97 pgs: 97 active+clean; 123 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-05T23:39:36.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:36 smithi130 ceph-mon[79417]: pgmap v186: 97 pgs: 97 active+clean; 123 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-05T23:39:36.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:36 smithi027 ceph-mon[78597]: pgmap v186: 97 pgs: 97 active+clean; 123 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 145 op/s 2024-04-05T23:39:37.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 342210 6.79 MB/sec execute 176 sec latency 2598.401 ms 2024-04-05T23:39:38.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 344007 6.82 MB/sec execute 177 sec latency 2810.688 ms 2024-04-05T23:39:38.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:38 smithi016 ceph-mon[38248]: pgmap v187: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 200 op/s 2024-04-05T23:39:38.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:38 smithi130 ceph-mon[79417]: pgmap v187: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 200 op/s 2024-04-05T23:39:38.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:38 smithi027 ceph-mon[78597]: pgmap v187: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 200 op/s 2024-04-05T23:39:39.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345046 6.80 MB/sec execute 178 sec latency 15.842 ms 2024-04-05T23:39:40.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345759 6.79 MB/sec execute 179 sec latency 554.651 ms 2024-04-05T23:39:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.566+0000 7f27e7c00700 1 -- 172.21.15.16:0/2634847147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00feed0 msgr2=0x7f27e00ff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.566+0000 7f27e7c00700 1 --2- 172.21.15.16:0/2634847147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00feed0 0x7f27e00ff350 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009990 tx=0x7f27cc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.566+0000 7f27e7c00700 1 -- 172.21.15.16:0/2634847147 shutdown_connections 2024-04-05T23:39:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.566+0000 7f27e7c00700 1 --2- 172.21.15.16:0/2634847147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27e00ff890 0x7f27e010dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.566+0000 7f27e7c00700 1 --2- 172.21.15.16:0/2634847147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27e0105e60 0x7f27e0106240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.566+0000 7f27e7c00700 1 --2- 172.21.15.16:0/2634847147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00feed0 0x7f27e00ff350 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.566+0000 7f27e7c00700 1 -- 172.21.15.16:0/2634847147 >> 172.21.15.16:0/2634847147 conn(0x7f27e0075e90 msgr2=0x7f27e00fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.567+0000 7f27e7c00700 1 -- 172.21.15.16:0/2634847147 shutdown_connections 2024-04-05T23:39:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.567+0000 7f27e7c00700 1 -- 172.21.15.16:0/2634847147 wait complete. 2024-04-05T23:39:40.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.568+0000 7f27e7c00700 1 Processor -- start 2024-04-05T23:39:40.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.568+0000 7f27e7c00700 1 -- start start 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.568+0000 7f27e7c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27e00feed0 0x7f27e0199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27e7c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00ff890 0x7f27e0199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27e7c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27e0105e60 0x7f27e019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27e7c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e019e350 con 0x7f27e0105e60 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27e7c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e019e4c0 con 0x7f27e00ff890 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27e7c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e019e680 con 0x7f27e00feed0 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00ff890 0x7f27e0199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27e5200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27e00feed0 0x7f27e0199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00ff890 0x7f27e0199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48266/0 (socket says 172.21.15.16:48266) 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27dfe00700 1 -- 172.21.15.16:0/4053473871 learned_addr learned my addr 172.21.15.16:0/4053473871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27e5c00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27e0105e60 0x7f27e019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27dfe00700 1 -- 172.21.15.16:0/4053473871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27e00feed0 msgr2=0x7f27e0199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:40.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27dfe00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27e00feed0 0x7f27e0199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27dfe00700 1 -- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27e0105e60 msgr2=0x7f27e019dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:40.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27dfe00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27e0105e60 0x7f27e019dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.569+0000 7f27dfe00700 1 -- 172.21.15.16:0/4053473871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0095f0 con 0x7f27e00ff890 2024-04-05T23:39:40.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.570+0000 7f27dfe00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00ff890 0x7f27e0199970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005770 tx=0x7f27cc0057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:40.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.570+0000 7f27dd600700 1 -- 172.21.15.16:0/4053473871 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc01d070 con 0x7f27e00ff890 2024-04-05T23:39:40.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.570+0000 7f27dd600700 1 -- 172.21.15.16:0/4053473871 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27cc021410 con 0x7f27e00ff890 2024-04-05T23:39:40.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.570+0000 7f27e7c00700 1 -- 172.21.15.16:0/4053473871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e019e900 con 0x7f27e00ff890 2024-04-05T23:39:40.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.570+0000 7f27dd600700 1 -- 172.21.15.16:0/4053473871 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc017650 con 0x7f27e00ff890 2024-04-05T23:39:40.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.570+0000 7f27e7c00700 1 -- 172.21.15.16:0/4053473871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e01a61a0 con 0x7f27e00ff890 2024-04-05T23:39:40.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.573+0000 7f27dd600700 1 -- 172.21.15.16:0/4053473871 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f27cc00fb40 con 0x7f27e00ff890 2024-04-05T23:39:40.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.574+0000 7f27dd600700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27c80a1310 0x7f27c80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.575+0000 7f27dd600700 1 -- 172.21.15.16:0/4053473871 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f27cc0cdfe0 con 0x7f27e00ff890 2024-04-05T23:39:40.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.575+0000 7f27fb645b80 1 Processor -- start 2024-04-05T23:39:40.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.575+0000 7f27e5200700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27c80a1310 0x7f27c80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.575+0000 7f27fb645b80 1 -- start start 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.575+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8deb80 0x557f9e8def60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8d9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 0x557f9e8d9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27e5200700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27c80a1310 0x7f27c80a37d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009be0 tx=0x7f27d4009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27fb645b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e83fb80 con 0x557f9e8d95c0 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27fb645b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e830930 con 0x557f9e8df4a0 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27fb645b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e82e2f0 con 0x557f9e8deb80 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8d9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8d9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48282/0 (socket says 172.21.15.16:48282) 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27dfe00700 1 -- 172.21.15.16:0/1699183078 learned_addr learned my addr 172.21.15.16:0/1699183078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27e5c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 0x557f9e8d9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27e5200700 1 --2- 172.21.15.16:0/1699183078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8deb80 0x557f9e8def60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27dfe00700 1 -- 172.21.15.16:0/1699183078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8deb80 msgr2=0x557f9e8def60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27dfe00700 1 --2- 172.21.15.16:0/1699183078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8deb80 0x557f9e8def60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27dfe00700 1 -- 172.21.15.16:0/1699183078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 msgr2=0x557f9e8d9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27dfe00700 1 --2- 172.21.15.16:0/1699183078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 0x557f9e8d9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.576+0000 7f27dfe00700 1 -- 172.21.15.16:0/1699183078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f9e6ab350 con 0x557f9e8df4a0 2024-04-05T23:39:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.577+0000 7f27dfe00700 1 --2- 172.21.15.16:0/1699183078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8d9030 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0098e0 tx=0x7f27cc09adc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da31c8ef8c3a1408 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.577+0000 7f27d2000700 1 -- 172.21.15.16:0/1699183078 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc09b190 con 0x557f9e8df4a0 2024-04-05T23:39:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.577+0000 7f27d2000700 1 -- 172.21.15.16:0/1699183078 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27cc0cbe90 con 0x557f9e8df4a0 2024-04-05T23:39:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.577+0000 7f27d2000700 1 -- 172.21.15.16:0/1699183078 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc09b3f0 con 0x557f9e8df4a0 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.577+0000 7f27fb645b80 1 -- 172.21.15.16:0/1699183078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 msgr2=0x557f9e8d9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.577+0000 7f27fb645b80 1 --2- 172.21.15.16:0/1699183078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8d9030 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0098e0 tx=0x7f27cc09adc0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.578+0000 7f27fb645b80 1 -- 172.21.15.16:0/1699183078 shutdown_connections 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.578+0000 7f27fb645b80 1 --2- 172.21.15.16:0/1699183078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 0x557f9e8d9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.578+0000 7f27fb645b80 1 --2- 172.21.15.16:0/1699183078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8deb80 0x557f9e8def60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.578+0000 7f27fb645b80 1 --2- 172.21.15.16:0/1699183078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8d9030 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.578+0000 7f27fb645b80 1 -- 172.21.15.16:0/1699183078 >> 172.21.15.16:0/1699183078 conn(0x557f9e84d1c0 msgr2=0x557f9e8d3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.578+0000 7f27fb645b80 1 -- 172.21.15.16:0/1699183078 shutdown_connections 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.578+0000 7f27fb645b80 1 -- 172.21.15.16:0/1699183078 wait complete. 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.578+0000 7f27fb645b80 1 -- start start 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.579+0000 7f27fb645b80 1 Processor -- start 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.579+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 0x557f9e8e5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.579+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8deb80 0x557f9e8e2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.579+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.579+0000 7f27fb645b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e829be0 con 0x557f9e8d95c0 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.579+0000 7f27fb645b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e7c5d20 con 0x557f9e8deb80 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.579+0000 7f27fb645b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e7c59c0 con 0x557f9e8df4a0 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27fb645b80 10 client.0 _ll_register_callbacks cb 0x557f9e631340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27e5200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 0x557f9e8e5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27e5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8deb80 0x557f9e8e2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27e5200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 0x557f9e8e5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44236/0 (socket says 172.21.15.16:44236) 2024-04-05T23:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27e5200700 1 -- 172.21.15.16:0/3359706072 learned_addr learned my addr 172.21.15.16:0/3359706072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:40.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27dfe00700 1 -- 172.21.15.16:0/3359706072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 msgr2=0x557f9e8e2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:40.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27dfe00700 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27dfe00700 1 -- 172.21.15.16:0/3359706072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 msgr2=0x557f9e8e5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:40.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27dfe00700 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 0x557f9e8e5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.580+0000 7f27dfe00700 1 -- 172.21.15.16:0/3359706072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e01a61a0 con 0x557f9e8deb80 2024-04-05T23:39:40.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.581+0000 7f27dfe00700 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8deb80 0x557f9e8e2810 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x557f9e8e5b00 tx=0x7f27cc021b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f86c73f54ebdb51 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:40.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.581+0000 7f27c6a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:39:40.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.581+0000 7f27c6a00700 1 -- 172.21.15.16:0/3359706072 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc09a880 con 0x557f9e8deb80 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.581+0000 7f27c6a00700 1 -- 172.21.15.16:0/3359706072 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f27cc0a3580 con 0x557f9e8deb80 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.581+0000 7f27fb645b80 1 -- 172.21.15.16:0/3359706072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557f9e8cc1e0 con 0x557f9e8deb80 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.581+0000 7f27c6a00700 1 -- 172.21.15.16:0/3359706072 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27cc09b190 con 0x557f9e8deb80 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.581+0000 7f27fb645b80 10 client.26345 fetch_fsmap learned FSMap version 48 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.581+0000 7f27fb645b80 1 -- 172.21.15.16:0/3359706072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) v3 -- 0x557f9e657b00 con 0x557f9e8deb80 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.582+0000 7f27c6a00700 1 -- 172.21.15.16:0/3359706072 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557f9e8cc1e0 con 0x557f9e8deb80 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.582+0000 7f27c6a00700 1 -- 172.21.15.16:0/3359706072 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 48) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f27cc09a0f0 con 0x557f9e8deb80 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.582+0000 7f27fb645b80 10 client.26345 fetch_fsmap finished waiting for FSMap version 48 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.582+0000 7f27fb645b80 10 client.26345 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.582+0000 7f27fb645b80 20 client.26345 populate_metadata read hostname 'smithi016' 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.582+0000 7f27fb645b80 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557f9e8f28a0 0x557f9e8f4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.582+0000 7f27fb645b80 4 client.26345 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.582+0000 7f27fb645b80 1 -- 172.21.15.16:0/3359706072 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557f9e8226a0 con 0x557f9e8f28a0 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.582+0000 7f27e5200700 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557f9e8f28a0 0x557f9e8f4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.583+0000 7f27e5200700 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557f9e8f28a0 0x557f9e8f4cc0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a6be4faf95e56f6 server_cookie=1283831fa61d5a3f in_seq=0 out_seq=0 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.583+0000 7f27c6a00700 10 client.26345 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.583+0000 7f27c6a00700 1 -- 172.21.15.16:0/3359706072 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557f9e8226a0 con 0x557f9e8f28a0 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.583+0000 7f27c6a00700 10 client.26345 handle_command_reply: tid=0 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.583+0000 7f27fb645b80 1 client.26345 shutdown 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 -- 172.21.15.16:0/3359706072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8deb80 msgr2=0x557f9e8e2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8deb80 0x557f9e8e2810 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x557f9e8e5b00 tx=0x7f27cc021b50 comp rx=0 tx=0).stop 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 -- 172.21.15.16:0/3359706072 shutdown_connections 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557f9e8f28a0 0x557f9e8f4cc0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8d95c0 0x557f9e8e5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27c6a00700 0 client.26345 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 --2- 172.21.15.16:0/3359706072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8deb80 0x557f9e8e2810 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 -- 172.21.15.16:0/3359706072 >> 172.21.15.16:0/3359706072 conn(0x557f9e84d1c0 msgr2=0x557f9e8d3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 -- 172.21.15.16:0/3359706072 shutdown_connections 2024-04-05T23:39:40.578 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 1 -- 172.21.15.16:0/3359706072 wait complete. 2024-04-05T23:39:40.578 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.584+0000 7f27fb645b80 20 client.26345 trim_cache size 0 max 16384 2024-04-05T23:39:40.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:40 smithi016 ceph-mon[38248]: pgmap v188: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-05T23:39:40.583 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:40 smithi130 ceph-mon[79417]: pgmap v188: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-05T23:39:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27fb645b80 1 Processor -- start 2024-04-05T23:39:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27fb645b80 1 -- start start 2024-04-05T23:39:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8d95c0 0x557f9e8de150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 0x557f9e8e2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27fb645b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e6ad640 con 0x557f9e8deb80 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27fb645b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e8e37e0 con 0x557f9e8d95c0 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27fb645b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e8e3e10 con 0x557f9e8df4a0 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27e5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27e5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8d95c0 0x557f9e8de150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27e5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:41114/0 (socket says 172.21.15.16:41114) 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27e5c00700 1 -- 172.21.15.16:0/2129763076 learned_addr learned my addr 172.21.15.16:0/2129763076 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27e5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8d95c0 0x557f9e8de150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48306/0 (socket says 172.21.15.16:48306) 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.601+0000 7f27dfe00700 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 0x557f9e8e2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27e5200700 1 -- 172.21.15.16:0/2129763076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 msgr2=0x557f9e8e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27e5200700 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27e5200700 1 -- 172.21.15.16:0/2129763076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 msgr2=0x557f9e8e2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27e5200700 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 0x557f9e8e2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27e5200700 1 -- 172.21.15.16:0/2129763076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f9e7efe40 con 0x557f9e8d95c0 2024-04-05T23:39:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27e5c00700 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27dfe00700 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 0x557f9e8e2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:39:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27e5200700 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8d95c0 0x557f9e8de150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x557f9e8f52a0 tx=0x7f27d4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a1c883345f32cd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27d2a00700 1 -- 172.21.15.16:0/2129763076 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4024040 con 0x557f9e8d95c0 2024-04-05T23:39:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27d2a00700 1 -- 172.21.15.16:0/2129763076 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27d4029dc0 con 0x557f9e8d95c0 2024-04-05T23:39:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.602+0000 7f27d2a00700 1 -- 172.21.15.16:0/2129763076 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4003dd0 con 0x557f9e8d95c0 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.603+0000 7f27fb645b80 1 -- 172.21.15.16:0/2129763076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8d95c0 msgr2=0x557f9e8de150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.603+0000 7f27fb645b80 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8d95c0 0x557f9e8de150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x557f9e8f52a0 tx=0x7f27d4014000 comp rx=0 tx=0).stop 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.603+0000 7f27fb645b80 1 -- 172.21.15.16:0/2129763076 shutdown_connections 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.603+0000 7f27fb645b80 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8df4a0 0x557f9e8e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.603+0000 7f27fb645b80 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8d95c0 0x557f9e8de150 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.603+0000 7f27fb645b80 1 --2- 172.21.15.16:0/2129763076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 0x557f9e8e2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.603+0000 7f27fb645b80 1 -- 172.21.15.16:0/2129763076 >> 172.21.15.16:0/2129763076 conn(0x557f9e84d1c0 msgr2=0x557f9e8ee420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.603+0000 7f27fb645b80 1 -- 172.21.15.16:0/2129763076 shutdown_connections 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.603+0000 7f27fb645b80 1 -- 172.21.15.16:0/2129763076 wait complete. 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.604+0000 7f27fb645b80 1 -- start start 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.604+0000 7f27fb645b80 1 Processor -- start 2024-04-05T23:39:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.604+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8d95c0 0x557f9e93f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.604+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 0x557f9e93fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.604+0000 7f27fb645b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8e7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.604+0000 7f27fb645b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e8e81f0 con 0x557f9e8deb80 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.604+0000 7f27fb645b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e8e8360 con 0x557f9e8df4a0 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.604+0000 7f27fb645b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9e8e8520 con 0x557f9e8d95c0 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27fb645b80 10 client.0 _ll_register_callbacks cb 0x557f9e631340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8d95c0 0x557f9e93f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8d95c0 0x557f9e93f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:41124/0 (socket says 172.21.15.16:41124) 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5200700 1 -- 172.21.15.16:0/4189370765 learned_addr learned my addr 172.21.15.16:0/4189370765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8e7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27dfe00700 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 0x557f9e93fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5200700 1 -- 172.21.15.16:0/4189370765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 msgr2=0x557f9e8e7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5200700 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8e7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5200700 1 -- 172.21.15.16:0/4189370765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 msgr2=0x557f9e93fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5200700 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 0x557f9e93fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5200700 1 -- 172.21.15.16:0/4189370765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d8009b90 con 0x557f9e8d95c0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27e5200700 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8d95c0 0x557f9e93f8c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009210 tx=0x7f27d4011db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14c4e54bc24082ed server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.605+0000 7f27c6a00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.606+0000 7f27c6a00700 1 -- 172.21.15.16:0/4189370765 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4024040 con 0x557f9e8d95c0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.606+0000 7f27c6a00700 1 -- 172.21.15.16:0/4189370765 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f27d4029b00 con 0x557f9e8d95c0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.606+0000 7f27fb645b80 1 -- 172.21.15.16:0/4189370765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f27cc0cbe90 con 0x557f9e8d95c0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.606+0000 7f27c6a00700 1 -- 172.21.15.16:0/4189370765 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27d4003ab0 con 0x557f9e8d95c0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.606+0000 7f27fb645b80 10 client.26383 fetch_fsmap learned FSMap version 48 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.606+0000 7f27fb645b80 1 -- 172.21.15.16:0/4189370765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) v3 -- 0x7f27d4017070 con 0x557f9e8d95c0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.606+0000 7f27c6a00700 1 -- 172.21.15.16:0/4189370765 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27cc0cbe90 con 0x557f9e8d95c0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.606+0000 7f27c6a00700 1 -- 172.21.15.16:0/4189370765 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 48) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f27d401fcc0 con 0x557f9e8d95c0 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.607+0000 7f27fb645b80 10 client.26383 fetch_fsmap finished waiting for FSMap version 48 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.607+0000 7f27fb645b80 10 client.26383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:39:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.607+0000 7f27fb645b80 20 client.26383 populate_metadata read hostname 'smithi016' 2024-04-05T23:39:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.607+0000 7f27fb645b80 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557f9e947730 0x557f9e949b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.607+0000 7f27fb645b80 4 client.26383 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:39:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.607+0000 7f27fb645b80 1 -- 172.21.15.16:0/4189370765 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557f9e678c90 con 0x557f9e947730 2024-04-05T23:39:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.607+0000 7f27dfe00700 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557f9e947730 0x557f9e949b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.607+0000 7f27dfe00700 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557f9e947730 0x557f9e949b50 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=178db073d040b758 server_cookie=5770d97b05ea2ed8 in_seq=0 out_seq=0 2024-04-05T23:39:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.607+0000 7f27c6a00700 10 client.26383 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27c6a00700 1 -- 172.21.15.16:0/4189370765 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557f9e678c90 con 0x557f9e947730 2024-04-05T23:39:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27c6a00700 10 client.26383 handle_command_reply: tid=0 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 client.26383 shutdown 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 -- 172.21.15.16:0/4189370765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8d95c0 msgr2=0x557f9e93f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8d95c0 0x557f9e93f8c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009210 tx=0x7f27d4011db0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 -- 172.21.15.16:0/4189370765 shutdown_connections 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x557f9e947730 0x557f9e949b50 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557f9e8df4a0 0x557f9e8e7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f9e8d95c0 0x557f9e93f8c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27c6a00700 0 client.26383 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 --2- 172.21.15.16:0/4189370765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f9e8deb80 0x557f9e93fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 -- 172.21.15.16:0/4189370765 >> 172.21.15.16:0/4189370765 conn(0x557f9e84d1c0 msgr2=0x557f9e8ee420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 -- 172.21.15.16:0/4189370765 shutdown_connections 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.634+0000 7f27fb645b80 1 -- 172.21.15.16:0/4189370765 wait complete. 2024-04-05T23:39:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27fb645b80 20 client.26383 trim_cache size 0 max 16384 2024-04-05T23:39:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:39:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:39:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:39:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:39:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 -- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27c80a1310 msgr2=0x7f27c80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27c80a1310 0x7f27c80a37d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009be0 tx=0x7f27d4009260 comp rx=0 tx=0).stop 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 -- 172.21.15.16:0/4053473871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00ff890 msgr2=0x7f27e0199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00ff890 0x7f27e0199970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005770 tx=0x7f27cc0057a0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 -- 172.21.15.16:0/4053473871 shutdown_connections 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f27c80a1310 0x7f27c80a37d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27e0105e60 0x7f27e019dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27e00feed0 0x7f27e0199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 --2- 172.21.15.16:0/4053473871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27e00ff890 0x7f27e0199970 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 -- 172.21.15.16:0/4053473871 >> 172.21.15.16:0/4053473871 conn(0x7f27e0075e90 msgr2=0x7f27e01041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 -- 172.21.15.16:0/4053473871 shutdown_connections 2024-04-05T23:39:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:40.635+0000 7f27e7c00700 1 -- 172.21.15.16:0/4053473871 wait complete. 2024-04-05T23:39:40.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:40 smithi027 ceph-mon[78597]: pgmap v188: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-05T23:39:41.003 INFO:tasks.workunit.client.0.smithi016.stdout: 1 347682 6.82 MB/sec execute 180 sec latency 648.162 ms 2024-04-05T23:39:42.004 INFO:tasks.workunit.client.0.smithi016.stdout: 1 348718 6.80 MB/sec execute 181 sec latency 14.354 ms 2024-04-05T23:39:42.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:42 smithi016 ceph-mon[38248]: pgmap v189: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-04-05T23:39:42.584 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:42 smithi130 ceph-mon[79417]: pgmap v189: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-04-05T23:39:42.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:42 smithi027 ceph-mon[78597]: pgmap v189: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-04-05T23:39:43.004 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349302 6.78 MB/sec execute 182 sec latency 629.040 ms 2024-04-05T23:39:43.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:43 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:43.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:43 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:43.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:43 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:44.004 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349302 6.74 MB/sec execute 183 sec latency 1629.187 ms 2024-04-05T23:39:44.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:44 smithi027 ceph-mon[78597]: pgmap v190: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-05T23:39:44.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:44 smithi016 ceph-mon[38248]: pgmap v190: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-05T23:39:44.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:44 smithi130 ceph-mon[79417]: pgmap v190: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-05T23:39:45.004 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349302 6.70 MB/sec execute 184 sec latency 2629.336 ms 2024-04-05T23:39:46.004 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349515 6.67 MB/sec execute 185 sec latency 3518.298 ms 2024-04-05T23:39:46.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:46 smithi027 ceph-mon[78597]: pgmap v191: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-05T23:39:46.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:46 smithi016 ceph-mon[38248]: pgmap v191: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-05T23:39:46.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:46 smithi130 ceph-mon[79417]: pgmap v191: 97 pgs: 97 active+clean; 125 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-05T23:39:47.004 INFO:tasks.workunit.client.0.smithi016.stdout: 1 351403 6.71 MB/sec execute 186 sec latency 14.872 ms 2024-04-05T23:39:47.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:47 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:39:47.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:47 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:39:47.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:47 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:39:47.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:47 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:39:47.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:47 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:39:47.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:47 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:39:47.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:47 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:39:47.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:47 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:39:47.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:47 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:39:47.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:47 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:39:47.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:47 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:39:47.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:47 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:39:48.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352664 6.70 MB/sec execute 187 sec latency 15.549 ms 2024-04-05T23:39:48.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:48 smithi027 ceph-mon[78597]: pgmap v192: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-04-05T23:39:48.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:48 smithi016 ceph-mon[38248]: pgmap v192: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-04-05T23:39:48.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:48 smithi130 ceph-mon[79417]: pgmap v192: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-04-05T23:39:49.004 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352850 6.66 MB/sec execute 188 sec latency 862.349 ms 2024-04-05T23:39:50.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352850 6.63 MB/sec execute 189 sec latency 1862.533 ms 2024-04-05T23:39:50.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:50 smithi027 ceph-mon[78597]: pgmap v193: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-05T23:39:50.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:50 smithi016 ceph-mon[38248]: pgmap v193: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-05T23:39:50.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:50 smithi130 ceph-mon[79417]: pgmap v193: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-05T23:39:51.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352850 6.59 MB/sec execute 190 sec latency 2862.640 ms 2024-04-05T23:39:52.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 353548 6.57 MB/sec execute 191 sec latency 3589.166 ms 2024-04-05T23:39:52.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:52 smithi027 ceph-mon[78597]: pgmap v194: 97 pgs: 97 active+clean; 130 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-04-05T23:39:52.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:52 smithi016 ceph-mon[38248]: pgmap v194: 97 pgs: 97 active+clean; 130 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-04-05T23:39:52.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:52 smithi130 ceph-mon[79417]: pgmap v194: 97 pgs: 97 active+clean; 130 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 169 op/s 2024-04-05T23:39:53.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 355065 6.60 MB/sec execute 192 sec latency 18.628 ms 2024-04-05T23:39:53.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:53 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.804+0000 7f43f9200700 1 -- 172.21.15.16:0/1016729425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43f4068be0 msgr2=0x7f43f410dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.804+0000 7f43f9200700 1 --2- 172.21.15.16:0/1016729425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43f4068be0 0x7f43f410dbe0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00b0a0 tx=0x7f43ec00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.804+0000 7f43f9200700 1 -- 172.21.15.16:0/1016729425 shutdown_connections 2024-04-05T23:39:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.804+0000 7f43f9200700 1 --2- 172.21.15.16:0/1016729425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4106f40 0x7f43f4107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.804+0000 7f43f9200700 1 --2- 172.21.15.16:0/1016729425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43f4068be0 0x7f43f410dbe0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.804+0000 7f43f9200700 1 --2- 172.21.15.16:0/1016729425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4068220 0x7f43f40686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.804+0000 7f43f9200700 1 -- 172.21.15.16:0/1016729425 >> 172.21.15.16:0/1016729425 conn(0x7f43f4076e50 msgr2=0x7f43f4077260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.804+0000 7f43f9200700 1 -- 172.21.15.16:0/1016729425 shutdown_connections 2024-04-05T23:39:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.804+0000 7f43f9200700 1 -- 172.21.15.16:0/1016729425 wait complete. 2024-04-05T23:39:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.805+0000 7f43f9200700 1 Processor -- start 2024-04-05T23:39:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.805+0000 7f43f9200700 1 -- start start 2024-04-05T23:39:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f9200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4068220 0x7f43f41995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f9200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4068be0 0x7f43f4199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f9200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43f4106f40 0x7f43f419dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f9200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f419e4e0 con 0x7f43f4068be0 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f9200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f419e650 con 0x7f43f4068220 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f9200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f419e810 con 0x7f43f4106f40 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4068220 0x7f43f41995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4068220 0x7f43f41995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56036/0 (socket says 172.21.15.16:56036) 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f1e00700 1 -- 172.21.15.16:0/442902730 learned_addr learned my addr 172.21.15.16:0/442902730 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f2800700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43f4106f40 0x7f43f419dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.806+0000 7f43f1400700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4068be0 0x7f43f4199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43f1e00700 1 -- 172.21.15.16:0/442902730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43f4106f40 msgr2=0x7f43f419dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43f1e00700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43f4106f40 0x7f43f419dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43f1e00700 1 -- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4068be0 msgr2=0x7f43f4199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43f1e00700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4068be0 0x7f43f4199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43f1e00700 1 -- 172.21.15.16:0/442902730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec009d00 con 0x7f43f4068220 2024-04-05T23:39:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43f1400700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4068be0 0x7f43f4199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:39:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43f1e00700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4068220 0x7f43f41995c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f43e400e9a0 tx=0x7f43e400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43eaa00700 1 -- 172.21.15.16:0/442902730 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4009790 con 0x7f43f4068220 2024-04-05T23:39:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43f9200700 1 -- 172.21.15.16:0/442902730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f419eaf0 con 0x7f43f4068220 2024-04-05T23:39:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43eaa00700 1 -- 172.21.15.16:0/442902730 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43e400cbd0 con 0x7f43f4068220 2024-04-05T23:39:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43eaa00700 1 -- 172.21.15.16:0/442902730 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4018680 con 0x7f43f4068220 2024-04-05T23:39:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.807+0000 7f43f9200700 1 -- 172.21.15.16:0/442902730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f41a6330 con 0x7f43f4068220 2024-04-05T23:39:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.810+0000 7f43eaa00700 1 -- 172.21.15.16:0/442902730 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f43e400cd40 con 0x7f43f4068220 2024-04-05T23:39:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.812+0000 7f43eaa00700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43d80a1290 0x7f43d80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.812+0000 7f43eaa00700 1 -- 172.21.15.16:0/442902730 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f43e4014070 con 0x7f43f4068220 2024-04-05T23:39:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.812+0000 7f440cd88b80 1 Processor -- start 2024-04-05T23:39:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.812+0000 7f43f1400700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43d80a1290 0x7f43d80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.812+0000 7f440cd88b80 1 -- start start 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c48b80 0x563861c48f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c43030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c435c0 0x563861c43a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f440cd88b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861ba9b80 con 0x563861c48b80 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f440cd88b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861b9a930 con 0x563861c494a0 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f440cd88b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861b982f0 con 0x563861c435c0 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f43f1400700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43d80a1290 0x7f43d80a3750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f43f419ab10 tx=0x7f43e0009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f43f1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c48b80 0x563861c48f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f43f1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c43030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f43f1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c48b80 0x563861c48f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33484/0 (socket says 172.21.15.16:33484) 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f43f1e00700 1 -- 172.21.15.16:0/3404542951 learned_addr learned my addr 172.21.15.16:0/3404542951 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.813+0000 7f43f2800700 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c435c0 0x563861c43a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43f1e00700 1 -- 172.21.15.16:0/3404542951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c435c0 msgr2=0x563861c43a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43f1e00700 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c435c0 0x563861c43a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43f1e00700 1 -- 172.21.15.16:0/3404542951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 msgr2=0x563861c43030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43f1e00700 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c43030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43f1e00700 1 -- 172.21.15.16:0/3404542951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563861a15350 con 0x563861c48b80 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43f1400700 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c43030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43f2800700 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c435c0 0x563861c43a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43f1e00700 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c48b80 0x563861c48f60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f43e4096a70 tx=0x7f43e4099960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e290a6b985056a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43df400700 1 -- 172.21.15.16:0/3404542951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e409a000 con 0x563861c48b80 2024-04-05T23:39:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43df400700 1 -- 172.21.15.16:0/3404542951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43e409de30 con 0x563861c48b80 2024-04-05T23:39:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.814+0000 7f43df400700 1 -- 172.21.15.16:0/3404542951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4099780 con 0x563861c48b80 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.815+0000 7f440cd88b80 1 -- 172.21.15.16:0/3404542951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c48b80 msgr2=0x563861c48f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.815+0000 7f440cd88b80 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c48b80 0x563861c48f60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f43e4096a70 tx=0x7f43e4099960 comp rx=0 tx=0).stop 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.815+0000 7f440cd88b80 1 -- 172.21.15.16:0/3404542951 shutdown_connections 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.815+0000 7f440cd88b80 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c48b80 0x563861c48f60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.815+0000 7f440cd88b80 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c435c0 0x563861c43a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.815+0000 7f440cd88b80 1 --2- 172.21.15.16:0/3404542951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c43030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.815+0000 7f440cd88b80 1 -- 172.21.15.16:0/3404542951 >> 172.21.15.16:0/3404542951 conn(0x563861bb71c0 msgr2=0x563861c3dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.815+0000 7f440cd88b80 1 -- 172.21.15.16:0/3404542951 shutdown_connections 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.815+0000 7f440cd88b80 1 -- 172.21.15.16:0/3404542951 wait complete. 2024-04-05T23:39:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.816+0000 7f440cd88b80 1 -- start start 2024-04-05T23:39:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.816+0000 7f440cd88b80 1 Processor -- start 2024-04-05T23:39:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c4c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c4ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c4d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f440cd88b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861b93be0 con 0x563861c435c0 2024-04-05T23:39:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f440cd88b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861b2fd20 con 0x563861c494a0 2024-04-05T23:39:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f440cd88b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861b2f9c0 con 0x563861c48b80 2024-04-05T23:39:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f440cd88b80 10 client.0 _ll_register_callbacks cb 0x56386199b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:39:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f43f1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c4ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f43f1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c4ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:47824/0 (socket says 172.21.15.16:47824) 2024-04-05T23:39:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.817+0000 7f43f1400700 1 -- 172.21.15.16:0/703298890 learned_addr learned my addr 172.21.15.16:0/703298890 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.818+0000 7f43f2800700 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c4d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.818+0000 7f43f1e00700 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c4c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.818+0000 7f43f1400700 1 -- 172.21.15.16:0/703298890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 msgr2=0x563861c4d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.818+0000 7f43f1400700 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c4d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.818+0000 7f43f1400700 1 -- 172.21.15.16:0/703298890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 msgr2=0x563861c4c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.818+0000 7f43f1400700 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c4c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.818+0000 7f43f1400700 1 -- 172.21.15.16:0/703298890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e0011c60 con 0x563861c48b80 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.819+0000 7f43f1400700 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c4ce00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f43e0003930 tx=0x7f43e00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80ab08f3b7078069 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.819+0000 7f43d3e00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.819+0000 7f43d3e00700 1 -- 172.21.15.16:0/703298890 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0012860 con 0x563861c48b80 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.819+0000 7f440cd88b80 1 -- 172.21.15.16:0/703298890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563861c361e0 con 0x563861c48b80 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.819+0000 7f43d3e00700 1 -- 172.21.15.16:0/703298890 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f43e0018030 con 0x563861c48b80 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.819+0000 7f43d3e00700 1 -- 172.21.15.16:0/703298890 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43e0009e70 con 0x563861c48b80 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.819+0000 7f440cd88b80 10 client.26395 fetch_fsmap learned FSMap version 50 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.819+0000 7f440cd88b80 1 -- 172.21.15.16:0/703298890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) v3 -- 0x7f43f41a6330 con 0x563861c48b80 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.820+0000 7f43d3e00700 1 -- 172.21.15.16:0/703298890 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563861c361e0 con 0x563861c48b80 2024-04-05T23:39:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.820+0000 7f43d3e00700 1 -- 172.21.15.16:0/703298890 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 50) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f43e0030070 con 0x563861c48b80 2024-04-05T23:39:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.820+0000 7f440cd88b80 10 client.26395 fetch_fsmap finished waiting for FSMap version 50 2024-04-05T23:39:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.820+0000 7f440cd88b80 10 client.26395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:39:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.820+0000 7f440cd88b80 20 client.26395 populate_metadata read hostname 'smithi016' 2024-04-05T23:39:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.820+0000 7f440cd88b80 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563861c658e0 0x563861c67d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.821+0000 7f440cd88b80 4 client.26395 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:39:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.821+0000 7f440cd88b80 1 -- 172.21.15.16:0/703298890 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563861b8c6a0 con 0x563861c658e0 2024-04-05T23:39:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.821+0000 7f43f1e00700 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563861c658e0 0x563861c67d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.822+0000 7f43f1e00700 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563861c658e0 0x563861c67d00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=debaa7e7e5e3b604 server_cookie=b35737d978ce7ed in_seq=0 out_seq=0 2024-04-05T23:39:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.822+0000 7f43d3e00700 10 client.26395 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.823+0000 7f43d3e00700 1 -- 172.21.15.16:0/703298890 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563861b8c6a0 con 0x563861c658e0 2024-04-05T23:39:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.823+0000 7f43d3e00700 10 client.26395 handle_command_reply: tid=0 2024-04-05T23:39:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.823+0000 7f440cd88b80 1 client.26395 shutdown 2024-04-05T23:39:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.824+0000 7f440cd88b80 1 -- 172.21.15.16:0/703298890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 msgr2=0x563861c4ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.824+0000 7f440cd88b80 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c4ce00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f43e0003930 tx=0x7f43e00047c0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.825+0000 7f440cd88b80 1 -- 172.21.15.16:0/703298890 shutdown_connections 2024-04-05T23:39:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.825+0000 7f440cd88b80 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563861c658e0 0x563861c67d00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.825+0000 7f440cd88b80 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c4c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.825+0000 7f440cd88b80 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c4d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.825+0000 7f440cd88b80 1 --2- 172.21.15.16:0/703298890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c4ce00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.825+0000 7f440cd88b80 1 -- 172.21.15.16:0/703298890 >> 172.21.15.16:0/703298890 conn(0x563861bb71c0 msgr2=0x563861c3d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.825+0000 7f43d3e00700 0 client.26395 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.826+0000 7f440cd88b80 1 -- 172.21.15.16:0/703298890 shutdown_connections 2024-04-05T23:39:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.826+0000 7f440cd88b80 1 -- 172.21.15.16:0/703298890 wait complete. 2024-04-05T23:39:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.826+0000 7f440cd88b80 20 client.26395 trim_cache size 0 max 16384 2024-04-05T23:39:53.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:53 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:53.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:53 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.844+0000 7f440cd88b80 1 Processor -- start 2024-04-05T23:39:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.844+0000 7f440cd88b80 1 -- start start 2024-04-05T23:39:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c43030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c3da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c469e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f440cd88b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861a17640 con 0x563861c435c0 2024-04-05T23:39:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f440cd88b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861c4f390 con 0x563861c494a0 2024-04-05T23:39:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f440cd88b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861c3f650 con 0x563861c48b80 2024-04-05T23:39:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f43f1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c43030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f43f2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f43f1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c3da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f43f2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c469e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56078/0 (socket says 172.21.15.16:56078) 2024-04-05T23:39:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.845+0000 7f43f2800700 1 -- 172.21.15.16:0/1861113676 learned_addr learned my addr 172.21.15.16:0/1861113676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.846+0000 7f43f2800700 1 -- 172.21.15.16:0/1861113676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 msgr2=0x563861c3da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.846+0000 7f43f2800700 1 --2- 172.21.15.16:0/1861113676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c3da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.846+0000 7f43f2800700 1 -- 172.21.15.16:0/1861113676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 msgr2=0x563861c43030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.846+0000 7f43f2800700 1 --2- 172.21.15.16:0/1861113676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c43030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.846+0000 7f43f2800700 1 -- 172.21.15.16:0/1861113676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563861c3a1c0 con 0x563861c494a0 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.846+0000 7f43f2800700 1 --2- 172.21.15.16:0/1861113676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c469e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00b0a0 tx=0x7f43ec007b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad3055afcbfea21 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.846+0000 7f43dfe00700 1 -- 172.21.15.16:0/1861113676 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec00e040 con 0x563861c494a0 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.846+0000 7f43dfe00700 1 -- 172.21.15.16:0/1861113676 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43ec021070 con 0x563861c494a0 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.846+0000 7f43dfe00700 1 -- 172.21.15.16:0/1861113676 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec007940 con 0x563861c494a0 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.847+0000 7f440cd88b80 1 -- 172.21.15.16:0/1861113676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 msgr2=0x563861c469e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.847+0000 7f440cd88b80 1 --2- 172.21.15.16:0/1861113676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c469e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00b0a0 tx=0x7f43ec007b30 comp rx=0 tx=0).stop 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.847+0000 7f440cd88b80 1 -- 172.21.15.16:0/1861113676 shutdown_connections 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.847+0000 7f440cd88b80 1 --2- 172.21.15.16:0/1861113676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c43030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.847+0000 7f440cd88b80 1 --2- 172.21.15.16:0/1861113676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c494a0 0x563861c469e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.847+0000 7f440cd88b80 1 --2- 172.21.15.16:0/1861113676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c48b80 0x563861c3da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.847+0000 7f440cd88b80 1 -- 172.21.15.16:0/1861113676 >> 172.21.15.16:0/1861113676 conn(0x563861bb71c0 msgr2=0x563861c4d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.847+0000 7f440cd88b80 1 -- 172.21.15.16:0/1861113676 shutdown_connections 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.847+0000 7f440cd88b80 1 -- 172.21.15.16:0/1861113676 wait complete. 2024-04-05T23:39:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.848+0000 7f440cd88b80 1 -- start start 2024-04-05T23:39:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.848+0000 7f440cd88b80 1 Processor -- start 2024-04-05T23:39:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c536c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c48b80 0x563861c53c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f440cd88b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c494a0 0x563861cb0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f440cd88b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861cb1250 con 0x563861c435c0 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f440cd88b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861cb13c0 con 0x563861c48b80 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f440cd88b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563861cb1580 con 0x563861c494a0 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f440cd88b80 10 client.0 _ll_register_callbacks cb 0x7f43e00043d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c48b80 0x563861c53c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f2800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c494a0 0x563861cb0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c48b80 0x563861c53c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56086/0 (socket says 172.21.15.16:56086) 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1400700 1 -- 172.21.15.16:0/562492005 learned_addr learned my addr 172.21.15.16:0/562492005 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1400700 1 -- 172.21.15.16:0/562492005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c494a0 msgr2=0x563861cb0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1400700 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c494a0 0x563861cb0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1400700 1 -- 172.21.15.16:0/562492005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 msgr2=0x563861c536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1400700 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1400700 1 -- 172.21.15.16:0/562492005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ec0093e0 con 0x563861c48b80 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.849+0000 7f43f1e00700 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f43f1400700 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c48b80 0x563861c53c00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x563861c3eda0 tx=0x7f43e0009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27450fdfb116a0af server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f43d3e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f43d3e00700 1 -- 172.21.15.16:0/562492005 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0012860 con 0x563861c48b80 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f440cd88b80 1 -- 172.21.15.16:0/562492005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43e409de30 con 0x563861c48b80 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f43d3e00700 1 -- 172.21.15.16:0/562492005 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8512+0+0 (secure 0 0 0) 0x7f43e0018040 con 0x563861c48b80 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f43d3e00700 1 -- 172.21.15.16:0/562492005 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43e0004630 con 0x563861c48b80 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f440cd88b80 10 client.26369 fetch_fsmap learned FSMap version 50 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f440cd88b80 1 -- 172.21.15.16:0/562492005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) v3 -- 0x5638619c1b00 con 0x563861c48b80 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f43d3e00700 1 -- 172.21.15.16:0/562492005 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43e409de30 con 0x563861c48b80 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.850+0000 7f43d3e00700 1 -- 172.21.15.16:0/562492005 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 50) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f43e001e040 con 0x563861c48b80 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.851+0000 7f440cd88b80 10 client.26369 fetch_fsmap finished waiting for FSMap version 50 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.851+0000 7f440cd88b80 10 client.26369 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:39:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.851+0000 7f440cd88b80 20 client.26369 populate_metadata read hostname 'smithi016' 2024-04-05T23:39:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.851+0000 7f440cd88b80 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563861cb9fc0 0x563861cbc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:39:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.851+0000 7f440cd88b80 4 client.26369 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:39:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.851+0000 7f440cd88b80 1 -- 172.21.15.16:0/562492005 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5638619e2c90 con 0x563861cb9fc0 2024-04-05T23:39:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.851+0000 7f43f1e00700 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563861cb9fc0 0x563861cbc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:39:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.851+0000 7f43d3e00700 10 client.26369 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.851+0000 7f43f1e00700 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563861cb9fc0 0x563861cbc3e0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a10068f1bfe64e7 server_cookie=4171b9f81d0af36a in_seq=0 out_seq=0 2024-04-05T23:39:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.880+0000 7f43d3e00700 1 -- 172.21.15.16:0/562492005 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5638619e2c90 con 0x563861cb9fc0 2024-04-05T23:39:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.880+0000 7f43d3e00700 10 client.26369 handle_command_reply: tid=0 2024-04-05T23:39:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.880+0000 7f440cd88b80 1 client.26369 shutdown 2024-04-05T23:39:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 -- 172.21.15.16:0/562492005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c48b80 msgr2=0x563861c53c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c48b80 0x563861c53c00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x563861c3eda0 tx=0x7f43e0009640 comp rx=0 tx=0).stop 2024-04-05T23:39:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 -- 172.21.15.16:0/562492005 shutdown_connections 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563861cb9fc0 0x563861cbc3e0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563861c435c0 0x563861c536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563861c494a0 0x563861cb0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f43d3e00700 0 client.26369 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 --2- 172.21.15.16:0/562492005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563861c48b80 0x563861c53c00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 -- 172.21.15.16:0/562492005 >> 172.21.15.16:0/562492005 conn(0x563861bb71c0 msgr2=0x563861c469e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 -- 172.21.15.16:0/562492005 shutdown_connections 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 1 -- 172.21.15.16:0/562492005 wait complete. 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.881+0000 7f440cd88b80 20 client.26369 trim_cache size 0 max 16384 2024-04-05T23:39:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:39:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:39:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:39:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:39:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 -- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43d80a1290 msgr2=0x7f43d80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43d80a1290 0x7f43d80a3750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f43f419ab10 tx=0x7f43e0009260 comp rx=0 tx=0).stop 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 -- 172.21.15.16:0/442902730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4068220 msgr2=0x7f43f41995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4068220 0x7f43f41995c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f43e400e9a0 tx=0x7f43e400ecb0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 -- 172.21.15.16:0/442902730 shutdown_connections 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f43d80a1290 0x7f43d80a3750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43f4106f40 0x7f43f419dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43f4068220 0x7f43f41995c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 --2- 172.21.15.16:0/442902730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43f4068be0 0x7f43f4199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 -- 172.21.15.16:0/442902730 >> 172.21.15.16:0/442902730 conn(0x7f43f4076e50 msgr2=0x7f43f40fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 -- 172.21.15.16:0/442902730 shutdown_connections 2024-04-05T23:39:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:39:53.882+0000 7f43f9200700 1 -- 172.21.15.16:0/442902730 wait complete. 2024-04-05T23:39:54.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356309 6.59 MB/sec execute 193 sec latency 23.387 ms 2024-04-05T23:39:54.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:54 smithi027 ceph-mon[78597]: pgmap v195: 97 pgs: 97 active+clean; 130 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 758 KiB/s wr, 95 op/s 2024-04-05T23:39:54.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:54 smithi016 ceph-mon[38248]: pgmap v195: 97 pgs: 97 active+clean; 130 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 758 KiB/s wr, 95 op/s 2024-04-05T23:39:54.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:54 smithi130 ceph-mon[79417]: pgmap v195: 97 pgs: 97 active+clean; 130 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 758 KiB/s wr, 95 op/s 2024-04-05T23:39:55.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356392 6.56 MB/sec execute 194 sec latency 943.319 ms 2024-04-05T23:39:56.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356392 6.52 MB/sec execute 195 sec latency 1943.402 ms 2024-04-05T23:39:56.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:56 smithi027 ceph-mon[78597]: pgmap v196: 97 pgs: 97 active+clean; 130 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 758 KiB/s wr, 95 op/s 2024-04-05T23:39:56.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:56 smithi016 ceph-mon[38248]: pgmap v196: 97 pgs: 97 active+clean; 130 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 758 KiB/s wr, 95 op/s 2024-04-05T23:39:56.835 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:56 smithi130 ceph-mon[79417]: pgmap v196: 97 pgs: 97 active+clean; 130 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 758 KiB/s wr, 95 op/s 2024-04-05T23:39:57.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 357077 6.50 MB/sec execute 196 sec latency 2668.540 ms 2024-04-05T23:39:58.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 358615 6.53 MB/sec execute 197 sec latency 30.479 ms 2024-04-05T23:39:58.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:58 smithi027 ceph-mon[78597]: pgmap v197: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-05T23:39:58.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:39:58 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:58.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:58 smithi016 ceph-mon[38248]: pgmap v197: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-05T23:39:58.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:39:58 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:58.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:58 smithi130 ceph-mon[79417]: pgmap v197: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-05T23:39:58.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:39:58 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:39:59.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359917 6.52 MB/sec execute 198 sec latency 23.332 ms 2024-04-05T23:40:00.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359941 6.49 MB/sec execute 199 sec latency 994.858 ms 2024-04-05T23:40:00.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a[38225]: 2024-04-05T23:39:59.998+0000 7f0cb0200700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-05T23:40:00.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a[38225]: 2024-04-05T23:39:59.998+0000 7f0cb0200700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-05T23:40:00.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a[38225]: 2024-04-05T23:39:59.998+0000 7f0cb0200700 -1 log_channel(cluster) log [ERR] : mds.c(mds.2): Metadata damage detected 2024-04-05T23:40:00.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:00 smithi027 ceph-mon[78597]: pgmap v198: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1017 KiB/s wr, 118 op/s 2024-04-05T23:40:00.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:00 smithi027 ceph-mon[78597]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-05T23:40:00.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:00 smithi027 ceph-mon[78597]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-05T23:40:00.718 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:00 smithi027 ceph-mon[78597]: mds.c(mds.2): Metadata damage detected 2024-04-05T23:40:00.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:00 smithi016 ceph-mon[38248]: pgmap v198: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1017 KiB/s wr, 118 op/s 2024-04-05T23:40:00.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:00 smithi016 ceph-mon[38248]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-05T23:40:00.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:00 smithi016 ceph-mon[38248]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-05T23:40:00.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:00 smithi016 ceph-mon[38248]: mds.c(mds.2): Metadata damage detected 2024-04-05T23:40:00.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:00 smithi130 ceph-mon[79417]: pgmap v198: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1017 KiB/s wr, 118 op/s 2024-04-05T23:40:00.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:00 smithi130 ceph-mon[79417]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-05T23:40:00.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:00 smithi130 ceph-mon[79417]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-05T23:40:00.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:00 smithi130 ceph-mon[79417]: mds.c(mds.2): Metadata damage detected 2024-04-05T23:40:01.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359941 6.46 MB/sec execute 200 sec latency 1995.012 ms 2024-04-05T23:40:02.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 360521 6.43 MB/sec execute 201 sec latency 2720.560 ms 2024-04-05T23:40:02.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:02 smithi027 ceph-mon[78597]: pgmap v199: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-05T23:40:02.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:02 smithi016 ceph-mon[38248]: pgmap v199: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-05T23:40:02.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:02 smithi130 ceph-mon[79417]: pgmap v199: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-05T23:40:03.005 INFO:tasks.workunit.client.0.smithi016.stdout: 1 361908 6.45 MB/sec execute 202 sec latency 25.968 ms 2024-04-05T23:40:04.006 INFO:tasks.workunit.client.0.smithi016.stdout: 1 362705 6.43 MB/sec execute 203 sec latency 23.044 ms 2024-04-05T23:40:04.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:04 smithi027 ceph-mon[78597]: pgmap v200: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-05T23:40:04.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:04 smithi027 ceph-mon[78597]: osdmap e86: 12 total, 12 up, 12 in 2024-04-05T23:40:04.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:04 smithi016 ceph-mon[38248]: pgmap v200: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-05T23:40:04.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:04 smithi016 ceph-mon[38248]: osdmap e86: 12 total, 12 up, 12 in 2024-04-05T23:40:04.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:04 smithi130 ceph-mon[79417]: pgmap v200: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-05T23:40:04.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:04 smithi130 ceph-mon[79417]: osdmap e86: 12 total, 12 up, 12 in 2024-04-05T23:40:05.006 INFO:tasks.workunit.client.0.smithi016.stdout: 1 363483 6.42 MB/sec execute 204 sec latency 511.478 ms 2024-04-05T23:40:06.006 INFO:tasks.workunit.client.0.smithi016.stdout: 1 365482 6.45 MB/sec execute 205 sec latency 601.609 ms 2024-04-05T23:40:06.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:06 smithi027 ceph-mon[78597]: pgmap v202: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-04-05T23:40:06.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:06 smithi016 ceph-mon[38248]: pgmap v202: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-04-05T23:40:06.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:06 smithi130 ceph-mon[79417]: pgmap v202: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-04-05T23:40:07.006 INFO:tasks.workunit.client.0.smithi016.stdout: 1 366561 6.44 MB/sec execute 206 sec latency 14.666 ms 2024-04-05T23:40:07.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.037+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2995216180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9f41019f0 msgr2=0x7fb9f4105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:07.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.037+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2995216180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9f41019f0 0x7fb9f4105a30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0009990 tx=0x7fb9e0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.037+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2995216180 shutdown_connections 2024-04-05T23:40:07.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.037+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2995216180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f4106160 0x7fb9f410dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.037+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2995216180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9f4101040 0x7fb9f4101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.037+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2995216180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9f41019f0 0x7fb9f4105a30 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.037+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2995216180 >> 172.21.15.16:0/2995216180 conn(0x7fb9f40fc8e0 msgr2=0x7fb9f40fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:07.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.037+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2995216180 shutdown_connections 2024-04-05T23:40:07.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.038+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2995216180 wait complete. 2024-04-05T23:40:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9fbc00700 1 Processor -- start 2024-04-05T23:40:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9fbc00700 1 -- start start 2024-04-05T23:40:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9fbc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9f4101040 0x7fb9f41972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9fbc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9f4106160 0x7fb9f4197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9fbc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f419bbc0 0x7fb9f419c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9fbc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f419c640 con 0x7fb9f4106160 2024-04-05T23:40:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9fbc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f419c7b0 con 0x7fb9f419bbc0 2024-04-05T23:40:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9fbc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f419c970 con 0x7fb9f4101040 2024-04-05T23:40:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9f9200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9f4101040 0x7fb9f41972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9f9200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9f4101040 0x7fb9f41972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:41690/0 (socket says 172.21.15.16:41690) 2024-04-05T23:40:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9f9200700 1 -- 172.21.15.16:0/2574928906 learned_addr learned my addr 172.21.15.16:0/2574928906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9f4106160 0x7fb9f4197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.039+0000 7fb9f9c00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f419bbc0 0x7fb9f419c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f9200700 1 -- 172.21.15.16:0/2574928906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f419bbc0 msgr2=0x7fb9f419c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f9200700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f419bbc0 0x7fb9f419c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f9200700 1 -- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9f4106160 msgr2=0x7fb9f4197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f9200700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9f4106160 0x7fb9f4197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f9200700 1 -- 172.21.15.16:0/2574928906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec009c40 con 0x7fb9f4101040 2024-04-05T23:40:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f9c00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f419bbc0 0x7fb9f419c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9f4106160 0x7fb9f4197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f9200700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9f4101040 0x7fb9f41972e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800e9a0 tx=0x7fb9e800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f1600700 1 -- 172.21.15.16:0/2574928906 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8009790 con 0x7fb9f4101040 2024-04-05T23:40:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f1600700 1 -- 172.21.15.16:0/2574928906 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9e800ce80 con 0x7fb9f4101040 2024-04-05T23:40:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2574928906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e00095f0 con 0x7fb9f4101040 2024-04-05T23:40:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.040+0000 7fb9f1600700 1 -- 172.21.15.16:0/2574928906 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8018680 con 0x7fb9f4101040 2024-04-05T23:40:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.041+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2574928906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f41a41d0 con 0x7fb9f4101040 2024-04-05T23:40:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.044+0000 7fb9f1600700 1 -- 172.21.15.16:0/2574928906 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fb9e8010430 con 0x7fb9f4101040 2024-04-05T23:40:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.045+0000 7fb9f1600700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb9dc0a1290 0x7fb9dc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.045+0000 7fb9f1600700 1 -- 172.21.15.16:0/2574928906 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fb9e8014070 con 0x7fb9f4101040 2024-04-05T23:40:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.045+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb9dc0a1290 0x7fb9dc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.046+0000 7fba0f777b80 1 Processor -- start 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.046+0000 7fba0f777b80 1 -- start start 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.046+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d99730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.046+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb9dc0a1290 0x7fb9dc0a3750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4198830 tx=0x7fb9e000b260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.046+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 0x55ca62d8c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.046+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 0x55ca62d8ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.046+0000 7fba0f777b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62cf2b80 con 0x55ca62d99c70 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.046+0000 7fba0f777b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62ce3930 con 0x55ca62d99350 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.046+0000 7fba0f777b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62ce12f0 con 0x55ca62d8c5c0 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 0x55ca62d8c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f9200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d99730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 0x55ca62d8c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47666/0 (socket says 172.21.15.16:47666) 2024-04-05T23:40:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f3e00700 1 -- 172.21.15.16:0/976217386 learned_addr learned my addr 172.21.15.16:0/976217386 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f9c00700 1 --2- 172.21.15.16:0/976217386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 0x55ca62d8ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f9200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d99730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:54988/0 (socket says 172.21.15.16:54988) 2024-04-05T23:40:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f9200700 1 -- 172.21.15.16:0/976217386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 msgr2=0x55ca62d8ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f9200700 1 --2- 172.21.15.16:0/976217386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 0x55ca62d8ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f9200700 1 -- 172.21.15.16:0/976217386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 msgr2=0x55ca62d8c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f9200700 1 --2- 172.21.15.16:0/976217386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 0x55ca62d8c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f9200700 1 -- 172.21.15.16:0/976217386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca62b5e350 con 0x55ca62d99350 2024-04-05T23:40:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.047+0000 7fb9f9200700 1 --2- 172.21.15.16:0/976217386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d99730 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80ca5e0 tx=0x7fb9e809a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=671f681f615da7fb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.048+0000 7fb9e6000700 1 -- 172.21.15.16:0/976217386 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e809ddd0 con 0x55ca62d99350 2024-04-05T23:40:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.048+0000 7fb9e6000700 1 -- 172.21.15.16:0/976217386 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9e80a2420 con 0x55ca62d99350 2024-04-05T23:40:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.048+0000 7fb9e6000700 1 -- 172.21.15.16:0/976217386 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e80996f0 con 0x55ca62d99350 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.048+0000 7fba0f777b80 1 -- 172.21.15.16:0/976217386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 msgr2=0x55ca62d99730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.048+0000 7fba0f777b80 1 --2- 172.21.15.16:0/976217386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d99730 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80ca5e0 tx=0x7fb9e809a6a0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.049+0000 7fba0f777b80 1 -- 172.21.15.16:0/976217386 shutdown_connections 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.049+0000 7fba0f777b80 1 --2- 172.21.15.16:0/976217386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 0x55ca62d8ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.049+0000 7fba0f777b80 1 --2- 172.21.15.16:0/976217386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d99730 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.049+0000 7fba0f777b80 1 --2- 172.21.15.16:0/976217386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 0x55ca62d8c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.049+0000 7fba0f777b80 1 -- 172.21.15.16:0/976217386 >> 172.21.15.16:0/976217386 conn(0x55ca62d001c0 msgr2=0x55ca62d86e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.049+0000 7fba0f777b80 1 -- 172.21.15.16:0/976217386 shutdown_connections 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.049+0000 7fba0f777b80 1 -- 172.21.15.16:0/976217386 wait complete. 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.049+0000 7fba0f777b80 1 -- start start 2024-04-05T23:40:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.050+0000 7fba0f777b80 1 Processor -- start 2024-04-05T23:40:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.050+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.050+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d9e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.050+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 0x55ca62d9ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.050+0000 7fba0f777b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62cdcbe0 con 0x55ca62d99c70 2024-04-05T23:40:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.050+0000 7fba0f777b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62c78d20 con 0x55ca62d99350 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.050+0000 7fba0f777b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62c789c0 con 0x55ca62d8c5c0 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fba0f777b80 10 client.0 _ll_register_callbacks cb 0x55ca62ae4340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f9200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d9e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f9c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 0x55ca62d9ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d9e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:54990/0 (socket says 172.21.15.16:54990) 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f3e00700 1 -- 172.21.15.16:0/2994038714 learned_addr learned my addr 172.21.15.16:0/2994038714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f3e00700 1 -- 172.21.15.16:0/2994038714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 msgr2=0x55ca62d9e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f3e00700 1 -- 172.21.15.16:0/2994038714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 msgr2=0x55ca62d9ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 0x55ca62d9ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f3e00700 1 -- 172.21.15.16:0/2994038714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f41a41d0 con 0x55ca62d99350 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.051+0000 7fb9f9200700 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.052+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d9e690 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0009990 tx=0x7fb9e0020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e92b367e83643d60 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.052+0000 7fb9daa00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.052+0000 7fb9daa00700 1 -- 172.21.15.16:0/2994038714 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e002d040 con 0x55ca62d99350 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.052+0000 7fba0f777b80 1 -- 172.21.15.16:0/2994038714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ca62d7f1e0 con 0x55ca62d99350 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.052+0000 7fb9daa00700 1 -- 172.21.15.16:0/2994038714 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fb9e0004e40 con 0x55ca62d99350 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.052+0000 7fb9daa00700 1 -- 172.21.15.16:0/2994038714 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9e00056e0 con 0x55ca62d99350 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.052+0000 7fba0f777b80 10 client.26381 fetch_fsmap learned FSMap version 51 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.052+0000 7fba0f777b80 1 -- 172.21.15.16:0/2994038714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x55ca62b0ab00 con 0x55ca62d99350 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.052+0000 7fb9daa00700 1 -- 172.21.15.16:0/2994038714 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9e003d020 con 0x55ca62d99350 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fb9daa00700 1 -- 172.21.15.16:0/2994038714 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 51) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb9e00329d0 con 0x55ca62d99350 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fba0f777b80 10 client.26381 fetch_fsmap finished waiting for FSMap version 51 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fba0f777b80 10 client.26381 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fba0f777b80 20 client.26381 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fba0f777b80 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ca62dae9e0 0x55ca62db0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fba0f777b80 4 client.26381 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:40:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fba0f777b80 1 -- 172.21.15.16:0/2994038714 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca62cd56a0 con 0x55ca62dae9e0 2024-04-05T23:40:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fb9f9200700 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ca62dae9e0 0x55ca62db0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fb9f9200700 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ca62dae9e0 0x55ca62db0e00 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=809345aa873ab237 server_cookie=6b69fd1a49292df0 in_seq=0 out_seq=0 2024-04-05T23:40:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.053+0000 7fb9daa00700 10 client.26381 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fb9daa00700 1 -- 172.21.15.16:0/2994038714 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ca62cd56a0 con 0x55ca62dae9e0 2024-04-05T23:40:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fb9daa00700 10 client.26381 handle_command_reply: tid=0 2024-04-05T23:40:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 client.26381 shutdown 2024-04-05T23:40:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 -- 172.21.15.16:0/2994038714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 msgr2=0x55ca62d9e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d9e690 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0009990 tx=0x7fb9e0020000 comp rx=0 tx=0).stop 2024-04-05T23:40:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 -- 172.21.15.16:0/2994038714 shutdown_connections 2024-04-05T23:40:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ca62dae9e0 0x55ca62db0e00 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d99c70 0x55ca62d9ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 --2- 172.21.15.16:0/2994038714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62d9e690 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fb9daa00700 0 client.26381 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 -- 172.21.15.16:0/2994038714 >> 172.21.15.16:0/2994038714 conn(0x55ca62d001c0 msgr2=0x55ca62d86280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 -- 172.21.15.16:0/2994038714 shutdown_connections 2024-04-05T23:40:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.054+0000 7fba0f777b80 1 -- 172.21.15.16:0/2994038714 wait complete. 2024-04-05T23:40:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.055+0000 7fba0f777b80 20 client.26381 trim_cache size 0 max 16384 2024-04-05T23:40:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.071+0000 7fba0f777b80 1 Processor -- start 2024-04-05T23:40:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.071+0000 7fba0f777b80 1 -- start start 2024-04-05T23:40:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99350 0x55ca62d86a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99c70 0x55ca62d86f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fba0f777b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62b60640 con 0x55ca62d8c5c0 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fba0f777b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62d9f750 con 0x55ca62d99c70 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fba0f777b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62d8c130 con 0x55ca62d99350 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99c70 0x55ca62d86f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99c70 0x55ca62d86f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:54998/0 (socket says 172.21.15.16:54998) 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47684/0 (socket says 172.21.15.16:47684) 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9c00700 1 -- 172.21.15.16:0/1533201987 learned_addr learned my addr 172.21.15.16:0/1533201987 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/1533201987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99350 0x55ca62d86a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9c00700 1 -- 172.21.15.16:0/1533201987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99350 msgr2=0x55ca62d86a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9c00700 1 --2- 172.21.15.16:0/1533201987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99350 0x55ca62d86a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9c00700 1 -- 172.21.15.16:0/1533201987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 msgr2=0x55ca62d9e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9c00700 1 --2- 172.21.15.16:0/1533201987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.072+0000 7fb9f9c00700 1 -- 172.21.15.16:0/1533201987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca62ca2e40 con 0x55ca62d99c70 2024-04-05T23:40:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.073+0000 7fb9f9c00700 1 --2- 172.21.15.16:0/1533201987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99c70 0x55ca62d86f40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec003d90 tx=0x7fb9ec009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7393cec40b89cf74 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.073+0000 7fb9e6a00700 1 -- 172.21.15.16:0/1533201987 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec00e6f0 con 0x55ca62d99c70 2024-04-05T23:40:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.073+0000 7fb9e6a00700 1 -- 172.21.15.16:0/1533201987 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9ec00f040 con 0x55ca62d99c70 2024-04-05T23:40:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.073+0000 7fb9e6a00700 1 -- 172.21.15.16:0/1533201987 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec020be0 con 0x55ca62d99c70 2024-04-05T23:40:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 -- 172.21.15.16:0/1533201987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99c70 msgr2=0x55ca62d86f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1533201987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99c70 0x55ca62d86f40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec003d90 tx=0x7fb9ec009920 comp rx=0 tx=0).stop 2024-04-05T23:40:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 -- 172.21.15.16:0/1533201987 shutdown_connections 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1533201987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62d9e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1533201987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99c70 0x55ca62d86f40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1533201987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99350 0x55ca62d86a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 -- 172.21.15.16:0/1533201987 >> 172.21.15.16:0/1533201987 conn(0x55ca62d001c0 msgr2=0x55ca62d9ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 -- 172.21.15.16:0/1533201987 shutdown_connections 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 -- 172.21.15.16:0/1533201987 wait complete. 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 -- start start 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.074+0000 7fba0f777b80 1 Processor -- start 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62da54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62da3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fba0f777b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99c70 0x55ca62da3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fba0f777b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62da4290 con 0x55ca62d8c5c0 2024-04-05T23:40:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fba0f777b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62da4400 con 0x55ca62d99350 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fba0f777b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca62da45c0 con 0x55ca62d99c70 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fba0f777b80 10 client.0 _ll_register_callbacks cb 0x7fb9e0005540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62da3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62da3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:55002/0 (socket says 172.21.15.16:55002) 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f3e00700 1 -- 172.21.15.16:0/1859698306 learned_addr learned my addr 172.21.15.16:0/1859698306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f9200700 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62da54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f9c00700 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99c70 0x55ca62da3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f3e00700 1 -- 172.21.15.16:0/1859698306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99c70 msgr2=0x55ca62da3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99c70 0x55ca62da3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f3e00700 1 -- 172.21.15.16:0/1859698306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 msgr2=0x55ca62da54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62da54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.075+0000 7fb9f3e00700 1 -- 172.21.15.16:0/1859698306 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec003a40 con 0x55ca62d99350 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.076+0000 7fb9f9200700 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62da54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:40:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.076+0000 7fb9f3e00700 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62da3810 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0032db0 tx=0x7fb9e000bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6ca729dcf2f78e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.076+0000 7fb9daa00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.076+0000 7fb9daa00700 1 -- 172.21.15.16:0/1859698306 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e002d040 con 0x55ca62d99350 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.076+0000 7fba0f777b80 1 -- 172.21.15.16:0/1859698306 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9e80a2420 con 0x55ca62d99350 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.076+0000 7fb9daa00700 1 -- 172.21.15.16:0/1859698306 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fb9e003e080 con 0x55ca62d99350 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.076+0000 7fb9daa00700 1 -- 172.21.15.16:0/1859698306 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9e0032630 con 0x55ca62d99350 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.076+0000 7fba0f777b80 10 client.26387 fetch_fsmap learned FSMap version 51 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.076+0000 7fba0f777b80 1 -- 172.21.15.16:0/1859698306 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x55ca62b62c70 con 0x55ca62d99350 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.077+0000 7fb9daa00700 1 -- 172.21.15.16:0/1859698306 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9e80a2420 con 0x55ca62d99350 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.077+0000 7fb9daa00700 1 -- 172.21.15.16:0/1859698306 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 51) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb9e0024640 con 0x55ca62d99350 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.077+0000 7fba0f777b80 10 client.26387 fetch_fsmap finished waiting for FSMap version 51 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.077+0000 7fba0f777b80 10 client.26387 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.077+0000 7fba0f777b80 20 client.26387 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.077+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ca62e037d0 0x55ca62e05bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.077+0000 7fba0f777b80 4 client.26387 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:40:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.077+0000 7fba0f777b80 1 -- 172.21.15.16:0/1859698306 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ca62b2bc90 con 0x55ca62e037d0 2024-04-05T23:40:07.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.077+0000 7fb9f9200700 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ca62e037d0 0x55ca62e05bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:07.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.078+0000 7fb9daa00700 10 client.26387 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:07.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.078+0000 7fb9f9200700 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ca62e037d0 0x55ca62e05bf0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a29fb5d43c38bf48 server_cookie=736fd38f979754a2 in_seq=0 out_seq=0 2024-04-05T23:40:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.110+0000 7fb9daa00700 1 -- 172.21.15.16:0/1859698306 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ca62b2bc90 con 0x55ca62e037d0 2024-04-05T23:40:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.110+0000 7fb9daa00700 10 client.26387 handle_command_reply: tid=0 2024-04-05T23:40:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.110+0000 7fba0f777b80 1 client.26387 shutdown 2024-04-05T23:40:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 -- 172.21.15.16:0/1859698306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 msgr2=0x55ca62da3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62da3810 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0032db0 tx=0x7fb9e000bf70 comp rx=0 tx=0).stop 2024-04-05T23:40:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 -- 172.21.15.16:0/1859698306 shutdown_connections 2024-04-05T23:40:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ca62e037d0 0x55ca62e05bf0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fb9daa00700 0 client.26387 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ca62d8c5c0 0x55ca62da54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca62d99c70 0x55ca62da3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 --2- 172.21.15.16:0/1859698306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca62d99350 0x55ca62da3810 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 -- 172.21.15.16:0/1859698306 >> 172.21.15.16:0/1859698306 conn(0x55ca62d001c0 msgr2=0x55ca62d987b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 -- 172.21.15.16:0/1859698306 shutdown_connections 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.111+0000 7fba0f777b80 1 -- 172.21.15.16:0/1859698306 wait complete. 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fba0f777b80 20 client.26387 trim_cache size 0 max 16384 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:40:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:40:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:40:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:40:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb9dc0a1290 msgr2=0x7fb9dc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb9dc0a1290 0x7fb9dc0a3750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4198830 tx=0x7fb9e000b260 comp rx=0 tx=0).stop 2024-04-05T23:40:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2574928906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9f4101040 msgr2=0x7fb9f41972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9f4101040 0x7fb9f41972e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800e9a0 tx=0x7fb9e800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2574928906 shutdown_connections 2024-04-05T23:40:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb9dc0a1290 0x7fb9dc0a3750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f419bbc0 0x7fb9f419c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9f4101040 0x7fb9f41972e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 --2- 172.21.15.16:0/2574928906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9f4106160 0x7fb9f4197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.112+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2574928906 >> 172.21.15.16:0/2574928906 conn(0x7fb9f40fc8e0 msgr2=0x7fb9f40fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.113+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2574928906 shutdown_connections 2024-04-05T23:40:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:07.113+0000 7fb9fbc00700 1 -- 172.21.15.16:0/2574928906 wait complete. 2024-04-05T23:40:08.006 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367032 6.42 MB/sec execute 207 sec latency 453.977 ms 2024-04-05T23:40:08.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:08 smithi027 ceph-mon[78597]: pgmap v203: 97 pgs: 97 active+clean; 135 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-05T23:40:08.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:08 smithi016 ceph-mon[38248]: pgmap v203: 97 pgs: 97 active+clean; 135 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-05T23:40:08.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:08 smithi130 ceph-mon[79417]: pgmap v203: 97 pgs: 97 active+clean; 135 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-05T23:40:09.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367032 6.39 MB/sec execute 208 sec latency 1454.139 ms 2024-04-05T23:40:10.006 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367032 6.36 MB/sec execute 209 sec latency 2454.266 ms 2024-04-05T23:40:10.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:10 smithi016 ceph-mon[38248]: pgmap v204: 97 pgs: 97 active+clean; 135 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-05T23:40:10.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:10 smithi016 ceph-mon[38248]: osdmap e87: 12 total, 12 up, 12 in 2024-04-05T23:40:10.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:10 smithi130 ceph-mon[79417]: pgmap v204: 97 pgs: 97 active+clean; 135 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-05T23:40:10.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:10 smithi130 ceph-mon[79417]: osdmap e87: 12 total, 12 up, 12 in 2024-04-05T23:40:10.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:10 smithi027 ceph-mon[78597]: pgmap v204: 97 pgs: 97 active+clean; 135 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-05T23:40:10.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:10 smithi027 ceph-mon[78597]: osdmap e87: 12 total, 12 up, 12 in 2024-04-05T23:40:11.006 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367256 6.33 MB/sec execute 210 sec latency 3339.699 ms 2024-04-05T23:40:12.006 INFO:tasks.workunit.client.0.smithi016.stdout: 1 369162 6.36 MB/sec execute 211 sec latency 15.148 ms 2024-04-05T23:40:12.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:12 smithi016 ceph-mon[38248]: pgmap v206: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 227 op/s 2024-04-05T23:40:12.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:12 smithi130 ceph-mon[79417]: pgmap v206: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 227 op/s 2024-04-05T23:40:12.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:12 smithi027 ceph-mon[78597]: pgmap v206: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 227 op/s 2024-04-05T23:40:13.006 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370308 6.36 MB/sec execute 212 sec latency 18.954 ms 2024-04-05T23:40:14.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370575 6.33 MB/sec execute 213 sec latency 597.955 ms 2024-04-05T23:40:14.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:14 smithi016 ceph-mon[38248]: pgmap v207: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-04-05T23:40:14.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:14 smithi130 ceph-mon[79417]: pgmap v207: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-04-05T23:40:14.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:14 smithi027 ceph-mon[78597]: pgmap v207: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-04-05T23:40:15.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370575 6.30 MB/sec execute 214 sec latency 1598.115 ms 2024-04-05T23:40:16.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370575 6.27 MB/sec execute 215 sec latency 2598.254 ms 2024-04-05T23:40:16.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:16 smithi016 ceph-mon[38248]: pgmap v208: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-04-05T23:40:16.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:16 smithi130 ceph-mon[79417]: pgmap v208: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-04-05T23:40:16.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:16 smithi027 ceph-mon[78597]: pgmap v208: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-04-05T23:40:17.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 371253 6.25 MB/sec execute 216 sec latency 3331.559 ms 2024-04-05T23:40:18.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 372804 6.28 MB/sec execute 217 sec latency 11.842 ms 2024-04-05T23:40:18.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:18 smithi016 ceph-mon[38248]: pgmap v209: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-04-05T23:40:18.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:18 smithi130 ceph-mon[79417]: pgmap v209: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-04-05T23:40:18.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:18 smithi027 ceph-mon[78597]: pgmap v209: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-04-05T23:40:19.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 373965 6.27 MB/sec execute 218 sec latency 17.963 ms 2024-04-05T23:40:20.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 374123 6.24 MB/sec execute 219 sec latency 846.115 ms 2024-04-05T23:40:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.269+0000 7f527da00700 1 -- 172.21.15.16:0/2648234733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 msgr2=0x7f5278111890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.269+0000 7f527da00700 1 --2- 172.21.15.16:0/2648234733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 0x7f5278111890 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5264009990 tx=0x7f5264009ca0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.269+0000 7f527da00700 1 -- 172.21.15.16:0/2648234733 shutdown_connections 2024-04-05T23:40:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.269+0000 7f527da00700 1 --2- 172.21.15.16:0/2648234733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5278111dd0 0x7f52781141c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.269+0000 7f527da00700 1 --2- 172.21.15.16:0/2648234733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5278073280 0x7f5278073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.269+0000 7f527da00700 1 --2- 172.21.15.16:0/2648234733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 0x7f5278111890 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.269+0000 7f527da00700 1 -- 172.21.15.16:0/2648234733 >> 172.21.15.16:0/2648234733 conn(0x7f52780fc8b0 msgr2=0x7f52780fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.269+0000 7f527da00700 1 -- 172.21.15.16:0/2648234733 shutdown_connections 2024-04-05T23:40:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.269+0000 7f527da00700 1 -- 172.21.15.16:0/2648234733 wait complete. 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.270+0000 7f527da00700 1 Processor -- start 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.270+0000 7f527da00700 1 -- start start 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.270+0000 7f527da00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5278073280 0x7f52781a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.270+0000 7f527da00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 0x7f52781a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f527da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5278111dd0 0x7f527819d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f527da00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527819de50 con 0x7f5278073ba0 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f527da00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527819dfc0 con 0x7f5278111dd0 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f527da00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527819e180 con 0x7f5278073280 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5276000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 0x7f52781a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5276000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 0x7f52781a3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41184/0 (socket says 172.21.15.16:41184) 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5276000700 1 -- 172.21.15.16:0/2406233039 learned_addr learned my addr 172.21.15.16:0/2406233039 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5276a00700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5278073280 0x7f52781a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5277400700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5278111dd0 0x7f527819d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5276000700 1 -- 172.21.15.16:0/2406233039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5278073280 msgr2=0x7f52781a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5276000700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5278073280 0x7f52781a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5276000700 1 -- 172.21.15.16:0/2406233039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5278111dd0 msgr2=0x7f527819d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5276000700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5278111dd0 0x7f527819d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.271+0000 7f5276000700 1 -- 172.21.15.16:0/2406233039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52640095f0 con 0x7f5278073ba0 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.272+0000 7f5276000700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 0x7f52781a3bf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f52640095c0 tx=0x7f526400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.272+0000 7f526f400700 1 -- 172.21.15.16:0/2406233039 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526401d070 con 0x7f5278073ba0 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.272+0000 7f526f400700 1 -- 172.21.15.16:0/2406233039 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5264005050 con 0x7f5278073ba0 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.272+0000 7f527da00700 1 -- 172.21.15.16:0/2406233039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527819e400 con 0x7f5278073ba0 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.272+0000 7f526f400700 1 -- 172.21.15.16:0/2406233039 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264004a70 con 0x7f5278073ba0 2024-04-05T23:40:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.272+0000 7f527da00700 1 -- 172.21.15.16:0/2406233039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52781aa6a0 con 0x7f5278073ba0 2024-04-05T23:40:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.275+0000 7f526f400700 1 -- 172.21.15.16:0/2406233039 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f52640051c0 con 0x7f5278073ba0 2024-04-05T23:40:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.276+0000 7f526f400700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f525c0a1290 0x7f525c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.276+0000 7f526f400700 1 -- 172.21.15.16:0/2406233039 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f52640cdb10 con 0x7f5278073ba0 2024-04-05T23:40:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.276+0000 7f52914fdb80 1 Processor -- start 2024-04-05T23:40:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.277+0000 7f5276a00700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f525c0a1290 0x7f525c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.277+0000 7f52914fdb80 1 -- start start 2024-04-05T23:40:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.277+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.277+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c34a0 0x55b3833bd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.277+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.277+0000 7f52914fdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b383323b80 con 0x55b3833c34a0 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.277+0000 7f52914fdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b383314930 con 0x55b3833bd5c0 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.277+0000 7f52914fdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3833122f0 con 0x55b3833c2b80 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.277+0000 7f5276a00700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f525c0a1290 0x7f525c0a3750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5268007be0 tx=0x7f52680073d0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5276000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c34a0 0x55b3833bd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5276a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5277400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5277400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52114/0 (socket says 172.21.15.16:52114) 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5276a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:36692/0 (socket says 172.21.15.16:36692) 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5276000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c34a0 0x55b3833bd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41196/0 (socket says 172.21.15.16:41196) 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5277400700 1 -- 172.21.15.16:0/110089539 learned_addr learned my addr 172.21.15.16:0/110089539 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5277400700 1 -- 172.21.15.16:0/110089539 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 msgr2=0x55b3833c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5277400700 1 --2- 172.21.15.16:0/110089539 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5277400700 1 -- 172.21.15.16:0/110089539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c34a0 msgr2=0x55b3833bd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5277400700 1 --2- 172.21.15.16:0/110089539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c34a0 0x55b3833bd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5277400700 1 -- 172.21.15.16:0/110089539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b38318f350 con 0x55b3833bd5c0 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5276000700 1 --2- 172.21.15.16:0/110089539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c34a0 0x55b3833bd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5276a00700 1 --2- 172.21.15.16:0/110089539 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.278+0000 7f5277400700 1 --2- 172.21.15.16:0/110089539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bda20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f52700060b0 tx=0x7f5270009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d1ff90e07b1f0a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f5263e00700 1 -- 172.21.15.16:0/110089539 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52700044d0 con 0x55b3833bd5c0 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f5263e00700 1 -- 172.21.15.16:0/110089539 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f527000f040 con 0x55b3833bd5c0 2024-04-05T23:40:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f5263e00700 1 -- 172.21.15.16:0/110089539 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52700156e0 con 0x55b3833bd5c0 2024-04-05T23:40:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f52914fdb80 1 -- 172.21.15.16:0/110089539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 msgr2=0x55b3833bda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f52914fdb80 1 --2- 172.21.15.16:0/110089539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bda20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f52700060b0 tx=0x7f5270009c20 comp rx=0 tx=0).stop 2024-04-05T23:40:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f52914fdb80 1 -- 172.21.15.16:0/110089539 shutdown_connections 2024-04-05T23:40:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f52914fdb80 1 --2- 172.21.15.16:0/110089539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bda20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f52914fdb80 1 --2- 172.21.15.16:0/110089539 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f52914fdb80 1 --2- 172.21.15.16:0/110089539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c34a0 0x55b3833bd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.279+0000 7f52914fdb80 1 -- 172.21.15.16:0/110089539 >> 172.21.15.16:0/110089539 conn(0x55b3833311c0 msgr2=0x55b3833b7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.280+0000 7f52914fdb80 1 -- 172.21.15.16:0/110089539 shutdown_connections 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.280+0000 7f52914fdb80 1 -- 172.21.15.16:0/110089539 wait complete. 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.280+0000 7f52914fdb80 1 -- start start 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.281+0000 7f52914fdb80 1 Processor -- start 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.281+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.281+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.281+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c34a0 0x55b3833c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.281+0000 7f52914fdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b38330dbe0 con 0x55b3833bd5c0 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.281+0000 7f52914fdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3832a9d20 con 0x55b3833c34a0 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.281+0000 7f52914fdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3832a99c0 con 0x55b3833c2b80 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.281+0000 7f52914fdb80 10 client.0 _ll_register_callbacks cb 0x55b383115340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41198/0 (socket says 172.21.15.16:41198) 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276a00700 1 -- 172.21.15.16:0/2308946575 learned_addr learned my addr 172.21.15.16:0/2308946575 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276000700 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5277400700 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c34a0 0x55b3833c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276a00700 1 -- 172.21.15.16:0/2308946575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 msgr2=0x55b3833c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276a00700 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276a00700 1 -- 172.21.15.16:0/2308946575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c34a0 msgr2=0x55b3833c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276a00700 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c34a0 0x55b3833c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276a00700 1 -- 172.21.15.16:0/2308946575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52781aa6a0 con 0x55b3833bd5c0 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5276a00700 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833c93c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f52680042e0 tx=0x7f52680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b402aee9684ad1d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5260c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.282+0000 7f5260c00700 1 -- 172.21.15.16:0/2308946575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268003a90 con 0x55b3833bd5c0 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f5260c00700 1 -- 172.21.15.16:0/2308946575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f5268017040 con 0x55b3833bd5c0 2024-04-05T23:40:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f52914fdb80 1 -- 172.21.15.16:0/2308946575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3833b01e0 con 0x55b3833bd5c0 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f5260c00700 1 -- 172.21.15.16:0/2308946575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5268003bf0 con 0x55b3833bd5c0 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f52914fdb80 10 client.16347 fetch_fsmap learned FSMap version 51 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f52914fdb80 1 -- 172.21.15.16:0/2308946575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x7f5270003b00 con 0x55b3833bd5c0 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f5260c00700 1 -- 172.21.15.16:0/2308946575 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b3833b01e0 con 0x55b3833bd5c0 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f5260c00700 1 -- 172.21.15.16:0/2308946575 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 51) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5268029420 con 0x55b3833bd5c0 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f52914fdb80 10 client.16347 fetch_fsmap finished waiting for FSMap version 51 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f52914fdb80 10 client.16347 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f52914fdb80 20 client.16347 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f52914fdb80 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b3833d6fd0 0x55b3833d93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f52914fdb80 4 client.16347 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.283+0000 7f52914fdb80 1 -- 172.21.15.16:0/2308946575 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3833066a0 con 0x55b3833d6fd0 2024-04-05T23:40:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.284+0000 7f5276000700 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b3833d6fd0 0x55b3833d93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.284+0000 7f5276000700 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b3833d6fd0 0x55b3833d93f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=565706ff7dbe2384 server_cookie=5087f32fd8bf3514 in_seq=0 out_seq=0 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.284+0000 7f5260c00700 10 client.16347 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f5260c00700 1 -- 172.21.15.16:0/2308946575 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b3833066a0 con 0x55b3833d6fd0 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f5260c00700 10 client.16347 handle_command_reply: tid=0 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 client.16347 shutdown 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 -- 172.21.15.16:0/2308946575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 msgr2=0x55b3833c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833c93c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f52680042e0 tx=0x7f52680047c0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 -- 172.21.15.16:0/2308946575 shutdown_connections 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b3833d6fd0 0x55b3833d93f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833c93c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c34a0 0x55b3833c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f5260c00700 0 client.16347 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 --2- 172.21.15.16:0/2308946575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c2b80 0x55b3833c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 -- 172.21.15.16:0/2308946575 >> 172.21.15.16:0/2308946575 conn(0x55b3833311c0 msgr2=0x55b3833b7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 -- 172.21.15.16:0/2308946575 shutdown_connections 2024-04-05T23:40:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 1 -- 172.21.15.16:0/2308946575 wait complete. 2024-04-05T23:40:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.285+0000 7f52914fdb80 20 client.16347 trim_cache size 0 max 16384 2024-04-05T23:40:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.300+0000 7f52914fdb80 1 Processor -- start 2024-04-05T23:40:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f52914fdb80 1 -- start start 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833bd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c2b80 0x55b3833cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f52914fdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b383191640 con 0x55b3833bd5c0 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f52914fdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3833ce490 con 0x55b3833c2b80 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f52914fdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3833c4c00 con 0x55b3833c34a0 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f5276a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833bd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f5276a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833bd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41214/0 (socket says 172.21.15.16:41214) 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f5276a00700 1 -- 172.21.15.16:0/955106661 learned_addr learned my addr 172.21.15.16:0/955106661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f5276000700 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c2b80 0x55b3833cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.301+0000 7f5277400700 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f5276a00700 1 -- 172.21.15.16:0/955106661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 msgr2=0x55b3833b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f5276a00700 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f5276a00700 1 -- 172.21.15.16:0/955106661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c2b80 msgr2=0x55b3833cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f5276a00700 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c2b80 0x55b3833cdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f5276a00700 1 -- 172.21.15.16:0/955106661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b38313bb00 con 0x55b3833bd5c0 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f5277400700 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f5276a00700 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833bd030 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f52680297a0 tx=0x7f5268010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce717b87bf076fea server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f526cc00700 1 -- 172.21.15.16:0/955106661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52680299c0 con 0x55b3833bd5c0 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f526cc00700 1 -- 172.21.15.16:0/955106661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5268003890 con 0x55b3833bd5c0 2024-04-05T23:40:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.302+0000 7f526cc00700 1 -- 172.21.15.16:0/955106661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52680116a0 con 0x55b3833bd5c0 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 -- 172.21.15.16:0/955106661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 msgr2=0x55b3833bd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833bd030 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f52680297a0 tx=0x7f5268010f90 comp rx=0 tx=0).stop 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 -- 172.21.15.16:0/955106661 shutdown_connections 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833bd5c0 0x55b3833bd030 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 --2- 172.21.15.16:0/955106661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833c2b80 0x55b3833cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 -- 172.21.15.16:0/955106661 >> 172.21.15.16:0/955106661 conn(0x55b3833311c0 msgr2=0x55b3833c4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 -- 172.21.15.16:0/955106661 shutdown_connections 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 -- 172.21.15.16:0/955106661 wait complete. 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 -- start start 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.303+0000 7f52914fdb80 1 Processor -- start 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c2b80 0x55b3833bfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f52914fdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833be600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f52914fdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3833bece0 con 0x55b3833c2b80 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f52914fdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3833bee50 con 0x55b3833bd5c0 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f52914fdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3833bf010 con 0x55b3833c34a0 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f52914fdb80 10 client.0 _ll_register_callbacks cb 0x7f5268010d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f5276000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c2b80 0x55b3833bfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f5277400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833be600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f5276a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f5277400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833be600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:36712/0 (socket says 172.21.15.16:36712) 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.304+0000 7f5277400700 1 -- 172.21.15.16:0/1727257044 learned_addr learned my addr 172.21.15.16:0/1727257044 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5277400700 1 -- 172.21.15.16:0/1727257044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 msgr2=0x55b3833bf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5277400700 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5277400700 1 -- 172.21.15.16:0/1727257044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c2b80 msgr2=0x55b3833bfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5277400700 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c2b80 0x55b3833bfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5277400700 1 -- 172.21.15.16:0/1727257044 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5270003b00 con 0x55b3833c34a0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5276a00700 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5276000700 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c2b80 0x55b3833bfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5277400700 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833be600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55b3833b97f0 tx=0x7f5270007a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46c501cbc4def683 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5260c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5260c00700 1 -- 172.21.15.16:0/1727257044 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270005810 con 0x55b3833c34a0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f5260c00700 1 -- 172.21.15.16:0/1727257044 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f5270005970 con 0x55b3833c34a0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.305+0000 7f52914fdb80 1 -- 172.21.15.16:0/1727257044 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f527000f040 con 0x55b3833c34a0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f5260c00700 1 -- 172.21.15.16:0/1727257044 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5270015480 con 0x55b3833c34a0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f52914fdb80 10 client.26437 fetch_fsmap learned FSMap version 51 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f52914fdb80 1 -- 172.21.15.16:0/1727257044 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x55b3832d3e40 con 0x55b3833c34a0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f5260c00700 1 -- 172.21.15.16:0/1727257044 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f527000f040 con 0x55b3833c34a0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f5260c00700 1 -- 172.21.15.16:0/1727257044 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 51) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5270026030 con 0x55b3833c34a0 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f52914fdb80 10 client.26437 fetch_fsmap finished waiting for FSMap version 51 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f52914fdb80 10 client.26437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f52914fdb80 20 client.26437 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f52914fdb80 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b38342bf60 0x55b38342e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f52914fdb80 4 client.26437 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:40:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.306+0000 7f52914fdb80 1 -- 172.21.15.16:0/1727257044 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b38315cc90 con 0x55b38342bf60 2024-04-05T23:40:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.307+0000 7f5276a00700 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b38342bf60 0x55b38342e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.307+0000 7f5276a00700 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b38342bf60 0x55b38342e380 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94886471bad5c2db server_cookie=703a42e14112750c in_seq=0 out_seq=0 2024-04-05T23:40:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.307+0000 7f5260c00700 10 client.26437 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.335+0000 7f5260c00700 1 -- 172.21.15.16:0/1727257044 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b38315cc90 con 0x55b38342bf60 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.335+0000 7f5260c00700 10 client.26437 handle_command_reply: tid=0 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.335+0000 7f52914fdb80 1 client.26437 shutdown 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 -- 172.21.15.16:0/1727257044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 msgr2=0x55b3833be600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833be600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55b3833b97f0 tx=0x7f5270007a20 comp rx=0 tx=0).stop 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 -- 172.21.15.16:0/1727257044 shutdown_connections 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b38342bf60 0x55b38342e380 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3833c34a0 0x55b3833be600 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3833bd5c0 0x55b3833bf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 --2- 172.21.15.16:0/1727257044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3833c2b80 0x55b3833bfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 -- 172.21.15.16:0/1727257044 >> 172.21.15.16:0/1727257044 conn(0x55b3833311c0 msgr2=0x55b3833d26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f5260c00700 0 client.26437 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:20.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 -- 172.21.15.16:0/1727257044 shutdown_connections 2024-04-05T23:40:20.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 1 -- 172.21.15.16:0/1727257044 wait complete. 2024-04-05T23:40:20.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.336+0000 7f52914fdb80 20 client.26437 trim_cache size 0 max 16384 2024-04-05T23:40:20.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:40:20.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:40:20.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:40:20.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:40:20.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 -- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f525c0a1290 msgr2=0x7f525c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:20.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f525c0a1290 0x7f525c0a3750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5268007be0 tx=0x7f52680073d0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 -- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 msgr2=0x7f52781a3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:20.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 0x7f52781a3bf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f52640095c0 tx=0x7f526400f690 comp rx=0 tx=0).stop 2024-04-05T23:40:20.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 -- 172.21.15.16:0/2406233039 shutdown_connections 2024-04-05T23:40:20.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f525c0a1290 0x7f525c0a3750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5278111dd0 0x7f527819d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5278073280 0x7f52781a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 --2- 172.21.15.16:0/2406233039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5278073ba0 0x7f52781a3bf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:20.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 -- 172.21.15.16:0/2406233039 >> 172.21.15.16:0/2406233039 conn(0x7f52780fc8b0 msgr2=0x7f52780fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:20.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 -- 172.21.15.16:0/2406233039 shutdown_connections 2024-04-05T23:40:20.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:20.337+0000 7f527da00700 1 -- 172.21.15.16:0/2406233039 wait complete. 2024-04-05T23:40:20.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:20 smithi016 ceph-mon[38248]: pgmap v210: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-04-05T23:40:20.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:20 smithi130 ceph-mon[79417]: pgmap v210: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-04-05T23:40:20.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:20 smithi027 ceph-mon[78597]: pgmap v210: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-04-05T23:40:21.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 374123 6.22 MB/sec execute 220 sec latency 1846.189 ms 2024-04-05T23:40:22.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 374818 6.20 MB/sec execute 221 sec latency 2569.923 ms 2024-04-05T23:40:22.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:22 smithi016 ceph-mon[38248]: pgmap v211: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-04-05T23:40:22.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:22 smithi130 ceph-mon[79417]: pgmap v211: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-04-05T23:40:22.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:22 smithi027 ceph-mon[78597]: pgmap v211: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-04-05T23:40:23.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376365 6.22 MB/sec execute 222 sec latency 11.310 ms 2024-04-05T23:40:23.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:23 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:40:23.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:23 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:40:23.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:23 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:40:24.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 6.19 MB/sec execute 223 sec latency 687.665 ms 2024-04-05T23:40:24.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:24 smithi016 ceph-mon[38248]: pgmap v212: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-04-05T23:40:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:24 smithi130 ceph-mon[79417]: pgmap v212: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-04-05T23:40:24.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:24 smithi027 ceph-mon[78597]: pgmap v212: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-04-05T23:40:25.007 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 6.17 MB/sec execute 224 sec latency 1687.758 ms 2024-04-05T23:40:26.008 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 6.14 MB/sec execute 225 sec latency 2687.886 ms 2024-04-05T23:40:26.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:26 smithi016 ceph-mon[38248]: pgmap v213: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-04-05T23:40:26.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:26 smithi130 ceph-mon[79417]: pgmap v213: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-04-05T23:40:26.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:26 smithi027 ceph-mon[78597]: pgmap v213: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-04-05T23:40:27.008 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 6.11 MB/sec execute 226 sec latency 3687.942 ms 2024-04-05T23:40:28.008 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 6.09 MB/sec execute 227 sec latency 4688.025 ms 2024-04-05T23:40:28.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:28 smithi016 ceph-mon[38248]: pgmap v214: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-04-05T23:40:28.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:28 smithi130 ceph-mon[79417]: pgmap v214: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-04-05T23:40:28.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:28 smithi027 ceph-mon[78597]: pgmap v214: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-04-05T23:40:29.008 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 6.06 MB/sec execute 228 sec latency 5688.175 ms 2024-04-05T23:40:30.008 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 6.03 MB/sec execute 229 sec latency 6688.300 ms 2024-04-05T23:40:30.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:30 smithi016 ceph-mon[38248]: pgmap v215: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-04-05T23:40:30.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:30 smithi130 ceph-mon[79417]: pgmap v215: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-04-05T23:40:30.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:30 smithi027 ceph-mon[78597]: pgmap v215: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-04-05T23:40:31.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 6.01 MB/sec execute 230 sec latency 7688.418 ms 2024-04-05T23:40:32.008 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 5.98 MB/sec execute 231 sec latency 8688.575 ms 2024-04-05T23:40:32.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:32 smithi016 ceph-mon[38248]: pgmap v216: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-04-05T23:40:32.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:32 smithi130 ceph-mon[79417]: pgmap v216: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-04-05T23:40:32.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:32 smithi027 ceph-mon[78597]: pgmap v216: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-04-05T23:40:33.008 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 5.95 MB/sec execute 232 sec latency 9688.733 ms 2024-04-05T23:40:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.498+0000 7fa40de00700 1 -- 172.21.15.16:0/1846259333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 msgr2=0x7fa408111890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.498+0000 7fa40de00700 1 --2- 172.21.15.16:0/1846259333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 0x7fa408111890 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009990 tx=0x7fa3f0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.498+0000 7fa40de00700 1 -- 172.21.15.16:0/1846259333 shutdown_connections 2024-04-05T23:40:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.498+0000 7fa40de00700 1 --2- 172.21.15.16:0/1846259333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa408111dd0 0x7fa4081141c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.498+0000 7fa40de00700 1 --2- 172.21.15.16:0/1846259333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa408073280 0x7fa408073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.498+0000 7fa40de00700 1 --2- 172.21.15.16:0/1846259333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 0x7fa408111890 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.498+0000 7fa40de00700 1 -- 172.21.15.16:0/1846259333 >> 172.21.15.16:0/1846259333 conn(0x7fa4080fc8b0 msgr2=0x7fa4080fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.498+0000 7fa40de00700 1 -- 172.21.15.16:0/1846259333 shutdown_connections 2024-04-05T23:40:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.498+0000 7fa40de00700 1 -- 172.21.15.16:0/1846259333 wait complete. 2024-04-05T23:40:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.499+0000 7fa40de00700 1 Processor -- start 2024-04-05T23:40:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.499+0000 7fa40de00700 1 -- start start 2024-04-05T23:40:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa40de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa408073280 0x7fa4081a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa40de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 0x7fa4081a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa40de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa408111dd0 0x7fa40819d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa40de00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40819de50 con 0x7fa408073ba0 2024-04-05T23:40:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa40de00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40819dfc0 con 0x7fa408073280 2024-04-05T23:40:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa40de00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40819e180 con 0x7fa408111dd0 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa3ffe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 0x7fa4081a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa3ffe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 0x7fa4081a3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57038/0 (socket says 172.21.15.16:57038) 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa3ffe00700 1 -- 172.21.15.16:0/4144864360 learned_addr learned my addr 172.21.15.16:0/4144864360 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa406a00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa408073280 0x7fa4081a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa407400700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa408111dd0 0x7fa40819d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa3ffe00700 1 -- 172.21.15.16:0/4144864360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa408111dd0 msgr2=0x7fa40819d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa408111dd0 0x7fa40819d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa3ffe00700 1 -- 172.21.15.16:0/4144864360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa408073280 msgr2=0x7fa4081a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa408073280 0x7fa4081a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa3ffe00700 1 -- 172.21.15.16:0/4144864360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f00095f0 con 0x7fa408073ba0 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.500+0000 7fa407400700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa408111dd0 0x7fa40819d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.501+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 0x7fa4081a3bf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0000c00 tx=0x7fa3f0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.501+0000 7fa40ca00700 1 -- 172.21.15.16:0/4144864360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f001d070 con 0x7fa408073ba0 2024-04-05T23:40:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.501+0000 7fa40ca00700 1 -- 172.21.15.16:0/4144864360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3f0022470 con 0x7fa408073ba0 2024-04-05T23:40:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.501+0000 7fa40ca00700 1 -- 172.21.15.16:0/4144864360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0017650 con 0x7fa408073ba0 2024-04-05T23:40:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.501+0000 7fa40de00700 1 -- 172.21.15.16:0/4144864360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40819e400 con 0x7fa408073ba0 2024-04-05T23:40:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.501+0000 7fa40de00700 1 -- 172.21.15.16:0/4144864360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4081aa6a0 con 0x7fa408073ba0 2024-04-05T23:40:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.504+0000 7fa40ca00700 1 -- 172.21.15.16:0/4144864360 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fa3f00225e0 con 0x7fa408073ba0 2024-04-05T23:40:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa40ca00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa3ec0a1310 0x7fa3ec0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa40ca00700 1 -- 172.21.15.16:0/4144864360 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fa3f00cdf20 con 0x7fa408073ba0 2024-04-05T23:40:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa42186bb80 1 Processor -- start 2024-04-05T23:40:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa406a00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa3ec0a1310 0x7fa3ec0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa42186bb80 1 -- start start 2024-04-05T23:40:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x559910141f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x55991013c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa42186bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599100a2b80 con 0x55991013c5c0 2024-04-05T23:40:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa42186bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559910093930 con 0x5599101424a0 2024-04-05T23:40:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.506+0000 7fa42186bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599100912f0 con 0x559910141b80 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa407400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa407400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57054/0 (socket says 172.21.15.16:57054) 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa407400700 1 -- 172.21.15.16:0/1227664387 learned_addr learned my addr 172.21.15.16:0/1227664387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/1227664387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x55991013c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa406a00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa3ec0a1310 0x7fa3ec0a37d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009c60 tx=0x7fa3f8009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa406a00700 1 --2- 172.21.15.16:0/1227664387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x559910141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa3ffe00700 1 -- 172.21.15.16:0/1227664387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 msgr2=0x559910141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/1227664387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x559910141f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa3ffe00700 1 -- 172.21.15.16:0/1227664387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 msgr2=0x55991013ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/1227664387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.507+0000 7fa3ffe00700 1 -- 172.21.15.16:0/1227664387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55990ff0e350 con 0x5599101424a0 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.508+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/1227664387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x55991013c030 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00cb650 tx=0x7fa3f0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efa15a5dd5e6011c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.508+0000 7fa3f7e00700 1 -- 172.21.15.16:0/1227664387 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f009b7a0 con 0x5599101424a0 2024-04-05T23:40:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.508+0000 7fa3f7e00700 1 -- 172.21.15.16:0/1227664387 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3f009b0a0 con 0x5599101424a0 2024-04-05T23:40:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.508+0000 7fa3f7e00700 1 -- 172.21.15.16:0/1227664387 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00229d0 con 0x5599101424a0 2024-04-05T23:40:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 -- 172.21.15.16:0/1227664387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 msgr2=0x55991013c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 --2- 172.21.15.16:0/1227664387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x55991013c030 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00cb650 tx=0x7fa3f0005fd0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 -- 172.21.15.16:0/1227664387 shutdown_connections 2024-04-05T23:40:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 --2- 172.21.15.16:0/1227664387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 --2- 172.21.15.16:0/1227664387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x559910141f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 --2- 172.21.15.16:0/1227664387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x55991013c030 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 -- 172.21.15.16:0/1227664387 >> 172.21.15.16:0/1227664387 conn(0x5599100b01c0 msgr2=0x559910136be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 -- 172.21.15.16:0/1227664387 shutdown_connections 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 -- 172.21.15.16:0/1227664387 wait complete. 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 -- start start 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.509+0000 7fa42186bb80 1 Processor -- start 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910148500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x5599101457f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x559910145e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa42186bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55991008cbe0 con 0x55991013c5c0 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa42186bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559910028d20 con 0x5599101424a0 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa42186bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599100289c0 con 0x559910141b80 2024-04-05T23:40:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa42186bb80 10 client.0 _ll_register_callbacks cb 0x55990fe94340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa406a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910148500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa406a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910148500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57070/0 (socket says 172.21.15.16:57070) 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa3ffe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x5599101457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa407400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x559910145e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.510+0000 7fa406a00700 1 -- 172.21.15.16:0/2475771695 learned_addr learned my addr 172.21.15.16:0/2475771695 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa407400700 1 -- 172.21.15.16:0/2475771695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 msgr2=0x5599101457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa407400700 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x5599101457f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa407400700 1 -- 172.21.15.16:0/2475771695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 msgr2=0x559910148500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa407400700 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910148500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa407400700 1 -- 172.21.15.16:0/2475771695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4081aa6a0 con 0x5599101424a0 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x5599101457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa407400700 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x559910145e70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa400007af0 tx=0x7fa400007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b50df30d796af14f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa3f4c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:40:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa3f4c00700 1 -- 172.21.15.16:0/2475771695 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400003c00 con 0x5599101424a0 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.511+0000 7fa42186bb80 1 -- 172.21.15.16:0/2475771695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55991012f1e0 con 0x5599101424a0 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa3f4c00700 1 -- 172.21.15.16:0/2475771695 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fa400003d60 con 0x5599101424a0 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa3f4c00700 1 -- 172.21.15.16:0/2475771695 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa400012940 con 0x5599101424a0 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa42186bb80 10 client.26417 fetch_fsmap learned FSMap version 52 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa42186bb80 1 -- 172.21.15.16:0/2475771695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) v3 -- 0x55990febab00 con 0x5599101424a0 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa3f4c00700 1 -- 172.21.15.16:0/2475771695 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55991012f1e0 con 0x5599101424a0 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa3f4c00700 1 -- 172.21.15.16:0/2475771695 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 52) v1 ==== 7498+0+0 (secure 0 0 0) 0x7fa400014040 con 0x5599101424a0 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa42186bb80 10 client.26417 fetch_fsmap finished waiting for FSMap version 52 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa42186bb80 10 client.26417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa42186bb80 20 client.26417 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x559910155cc0 0x5599101580e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa42186bb80 4 client.26417 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.512+0000 7fa42186bb80 1 -- 172.21.15.16:0/2475771695 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599100856a0 con 0x559910155cc0 2024-04-05T23:40:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.513+0000 7fa406a00700 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x559910155cc0 0x5599101580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.513+0000 7fa406a00700 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x559910155cc0 0x5599101580e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f81ee4d98812e85 server_cookie=6c1ca044664b1871 in_seq=0 out_seq=0 2024-04-05T23:40:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.513+0000 7fa3f4c00700 10 client.26417 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa3f4c00700 1 -- 172.21.15.16:0/2475771695 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5599100856a0 con 0x559910155cc0 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa3f4c00700 10 client.26417 handle_command_reply: tid=0 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa42186bb80 1 client.26417 shutdown 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa42186bb80 1 -- 172.21.15.16:0/2475771695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 msgr2=0x559910145e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x559910145e70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa400007af0 tx=0x7fa400007e00 comp rx=0 tx=0).stop 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa42186bb80 1 -- 172.21.15.16:0/2475771695 shutdown_connections 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x559910155cc0 0x5599101580e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910148500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x559910145e70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa3f4c00700 0 client.26417 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2475771695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x5599101457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.514+0000 7fa42186bb80 1 -- 172.21.15.16:0/2475771695 >> 172.21.15.16:0/2475771695 conn(0x5599100b01c0 msgr2=0x559910136280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.515+0000 7fa42186bb80 1 -- 172.21.15.16:0/2475771695 shutdown_connections 2024-04-05T23:40:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.515+0000 7fa42186bb80 1 -- 172.21.15.16:0/2475771695 wait complete. 2024-04-05T23:40:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.515+0000 7fa42186bb80 20 client.26417 trim_cache size 0 max 16384 2024-04-05T23:40:33.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.530+0000 7fa42186bb80 1 Processor -- start 2024-04-05T23:40:33.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa42186bb80 1 -- start start 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910145da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x55991014cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x559910151a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa42186bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55990ff10640 con 0x55991013c5c0 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa42186bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55991014d0e0 con 0x5599101424a0 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa42186bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559910143da0 con 0x559910141b80 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa406a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa406a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910145da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57074/0 (socket says 172.21.15.16:57074) 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa406a00700 1 -- 172.21.15.16:0/2035253376 learned_addr learned my addr 172.21.15.16:0/2035253376 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa407400700 1 --2- 172.21.15.16:0/2035253376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x559910151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/2035253376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x55991014cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa406a00700 1 -- 172.21.15.16:0/2035253376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 msgr2=0x55991014cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa406a00700 1 --2- 172.21.15.16:0/2035253376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x55991014cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa406a00700 1 -- 172.21.15.16:0/2035253376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 msgr2=0x559910151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa406a00700 1 --2- 172.21.15.16:0/2035253376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x559910151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.531+0000 7fa406a00700 1 -- 172.21.15.16:0/2035253376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559910052e40 con 0x55991013c5c0 2024-04-05T23:40:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.532+0000 7fa406a00700 1 --2- 172.21.15.16:0/2035253376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910145da0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x5599101589b0 tx=0x7fa3f8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e7616231bb5b52 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.532+0000 7fa3fd200700 1 -- 172.21.15.16:0/2035253376 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8023040 con 0x55991013c5c0 2024-04-05T23:40:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.532+0000 7fa3fd200700 1 -- 172.21.15.16:0/2035253376 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3f8004500 con 0x55991013c5c0 2024-04-05T23:40:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.532+0000 7fa3fd200700 1 -- 172.21.15.16:0/2035253376 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f801ed30 con 0x55991013c5c0 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 -- 172.21.15.16:0/2035253376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 msgr2=0x559910145da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2035253376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910145da0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x5599101589b0 tx=0x7fa3f8014000 comp rx=0 tx=0).stop 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 -- 172.21.15.16:0/2035253376 shutdown_connections 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2035253376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x559910145da0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2035253376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5599101424a0 0x559910151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 --2- 172.21.15.16:0/2035253376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559910141b80 0x55991014cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 -- 172.21.15.16:0/2035253376 >> 172.21.15.16:0/2035253376 conn(0x5599100b01c0 msgr2=0x559910151300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 -- 172.21.15.16:0/2035253376 shutdown_connections 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 -- 172.21.15.16:0/2035253376 wait complete. 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.533+0000 7fa42186bb80 1 -- start start 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.534+0000 7fa42186bb80 1 Processor -- start 2024-04-05T23:40:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.534+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.534+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559910141b80 0x55991013ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.534+0000 7fa42186bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599101424a0 0x55991013d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.534+0000 7fa42186bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55991013dc20 con 0x55991013c5c0 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.534+0000 7fa42186bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55991013dd90 con 0x559910141b80 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.534+0000 7fa42186bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55991013df50 con 0x5599101424a0 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.534+0000 7fa42186bb80 10 client.0 _ll_register_callbacks cb 0x7fa400009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa406a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa406a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57080/0 (socket says 172.21.15.16:57080) 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa3ffe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559910141b80 0x55991013ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa406a00700 1 -- 172.21.15.16:0/3407904674 learned_addr learned my addr 172.21.15.16:0/3407904674 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa407400700 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599101424a0 0x55991013d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa406a00700 1 -- 172.21.15.16:0/3407904674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599101424a0 msgr2=0x55991013d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa406a00700 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599101424a0 0x55991013d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa406a00700 1 -- 172.21.15.16:0/3407904674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559910141b80 msgr2=0x55991013ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa406a00700 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559910141b80 0x55991013ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa406a00700 1 -- 172.21.15.16:0/3407904674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f8017070 con 0x55991013c5c0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559910141b80 0x55991013ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa406a00700 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013e920 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa3f801e820 tx=0x7fa3f801e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc663fecc852553 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa3f4c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.535+0000 7fa3f4c00700 1 -- 172.21.15.16:0/3407904674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8023050 con 0x55991013c5c0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa3f4c00700 1 -- 172.21.15.16:0/3407904674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fa3f8038070 con 0x55991013c5c0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa42186bb80 1 -- 172.21.15.16:0/3407904674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa3f009b0a0 con 0x55991013c5c0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa3f4c00700 1 -- 172.21.15.16:0/3407904674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f8004240 con 0x55991013c5c0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa42186bb80 10 client.16371 fetch_fsmap learned FSMap version 52 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa42186bb80 1 -- 172.21.15.16:0/3407904674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) v3 -- 0x55990ff12c70 con 0x55991013c5c0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa3f4c00700 1 -- 172.21.15.16:0/3407904674 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3f009b0a0 con 0x55991013c5c0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa3f4c00700 1 -- 172.21.15.16:0/3407904674 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 52) v1 ==== 7498+0+0 (secure 0 0 0) 0x7fa3f8003c00 con 0x55991013c5c0 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa42186bb80 10 client.16371 fetch_fsmap finished waiting for FSMap version 52 2024-04-05T23:40:33.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa42186bb80 10 client.16371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa42186bb80 20 client.16371 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa42186bb80 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5599101aab10 0x5599101acf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa42186bb80 4 client.16371 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:40:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.536+0000 7fa42186bb80 1 -- 172.21.15.16:0/3407904674 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55990fedbc90 con 0x5599101aab10 2024-04-05T23:40:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.537+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5599101aab10 0x5599101acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.537+0000 7fa3ffe00700 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5599101aab10 0x5599101acf30 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=730d1a0f7621353d server_cookie=e04c5821672e14bf in_seq=0 out_seq=0 2024-04-05T23:40:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.537+0000 7fa3f4c00700 10 client.16371 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.566+0000 7fa3f4c00700 1 -- 172.21.15.16:0/3407904674 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55990fedbc90 con 0x5599101aab10 2024-04-05T23:40:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa3f4c00700 10 client.16371 handle_command_reply: tid=0 2024-04-05T23:40:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 client.16371 shutdown 2024-04-05T23:40:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 -- 172.21.15.16:0/3407904674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 msgr2=0x55991013e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013e920 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa3f801e820 tx=0x7fa3f801e850 comp rx=0 tx=0).stop 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 -- 172.21.15.16:0/3407904674 shutdown_connections 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5599101aab10 0x5599101acf30 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55991013c5c0 0x55991013e920 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599101424a0 0x55991013d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 --2- 172.21.15.16:0/3407904674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559910141b80 0x55991013ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 -- 172.21.15.16:0/3407904674 >> 172.21.15.16:0/3407904674 conn(0x5599100b01c0 msgr2=0x559910151b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa3f4c00700 0 client.16371 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 -- 172.21.15.16:0/3407904674 shutdown_connections 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.567+0000 7fa42186bb80 1 -- 172.21.15.16:0/3407904674 wait complete. 2024-04-05T23:40:33.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa42186bb80 20 client.16371 trim_cache size 0 max 16384 2024-04-05T23:40:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:40:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:40:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:40:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:40:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa40de00700 1 -- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa3ec0a1310 msgr2=0x7fa3ec0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa40de00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa3ec0a1310 0x7fa3ec0a37d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009c60 tx=0x7fa3f8009260 comp rx=0 tx=0).stop 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa40de00700 1 -- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 msgr2=0x7fa4081a3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa40de00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 0x7fa4081a3bf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0000c00 tx=0x7fa3f0005740 comp rx=0 tx=0).stop 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa40de00700 1 -- 172.21.15.16:0/4144864360 shutdown_connections 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa40de00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa3ec0a1310 0x7fa3ec0a37d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa40de00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa408111dd0 0x7fa40819d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa40de00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa408073280 0x7fa4081a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.568+0000 7fa40de00700 1 --2- 172.21.15.16:0/4144864360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408073ba0 0x7fa4081a3bf0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.569+0000 7fa40de00700 1 -- 172.21.15.16:0/4144864360 >> 172.21.15.16:0/4144864360 conn(0x7fa4080fc8b0 msgr2=0x7fa4080fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.569+0000 7fa40de00700 1 -- 172.21.15.16:0/4144864360 shutdown_connections 2024-04-05T23:40:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:33.569+0000 7fa40de00700 1 -- 172.21.15.16:0/4144864360 wait complete. 2024-04-05T23:40:34.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 5.93 MB/sec execute 233 sec latency 10688.882 ms 2024-04-05T23:40:34.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:34 smithi016 ceph-mon[38248]: pgmap v217: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 803 KiB/s wr, 56 op/s 2024-04-05T23:40:34.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:34 smithi130 ceph-mon[79417]: pgmap v217: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 803 KiB/s wr, 56 op/s 2024-04-05T23:40:34.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:34 smithi027 ceph-mon[78597]: pgmap v217: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 803 KiB/s wr, 56 op/s 2024-04-05T23:40:35.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 5.90 MB/sec execute 234 sec latency 11689.022 ms 2024-04-05T23:40:36.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376576 5.88 MB/sec execute 235 sec latency 12689.098 ms 2024-04-05T23:40:36.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:36 smithi016 ceph-mon[38248]: pgmap v218: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 803 KiB/s wr, 56 op/s 2024-04-05T23:40:36.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:36 smithi130 ceph-mon[79417]: pgmap v218: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 803 KiB/s wr, 56 op/s 2024-04-05T23:40:36.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:36 smithi027 ceph-mon[78597]: pgmap v218: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 803 KiB/s wr, 56 op/s 2024-04-05T23:40:37.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376964 5.86 MB/sec execute 236 sec latency 13408.314 ms 2024-04-05T23:40:38.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 377666 5.85 MB/sec execute 237 sec latency 574.399 ms 2024-04-05T23:40:38.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:38 smithi016 ceph-mon[38248]: pgmap v219: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-04-05T23:40:38.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:38 smithi130 ceph-mon[79417]: pgmap v219: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-04-05T23:40:38.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:38 smithi027 ceph-mon[78597]: pgmap v219: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-04-05T23:40:39.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 377666 5.83 MB/sec execute 238 sec latency 1574.548 ms 2024-04-05T23:40:39.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:39 smithi027 ceph-mon[78597]: pgmap v220: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 544 KiB/s wr, 41 op/s 2024-04-05T23:40:40.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 377666 5.80 MB/sec execute 239 sec latency 2574.683 ms 2024-04-05T23:40:40.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:39 smithi016 ceph-mon[38248]: pgmap v220: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 544 KiB/s wr, 41 op/s 2024-04-05T23:40:40.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:39 smithi130 ceph-mon[79417]: pgmap v220: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 544 KiB/s wr, 41 op/s 2024-04-05T23:40:41.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 379637 5.83 MB/sec execute 240 sec latency 2670.669 ms 2024-04-05T23:40:42.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 380747 5.82 MB/sec execute 241 sec latency 18.147 ms 2024-04-05T23:40:42.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:41 smithi016 ceph-mon[38248]: pgmap v221: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1003 KiB/s wr, 82 op/s 2024-04-05T23:40:42.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:41 smithi130 ceph-mon[79417]: pgmap v221: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1003 KiB/s wr, 82 op/s 2024-04-05T23:40:42.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:41 smithi027 ceph-mon[78597]: pgmap v221: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1003 KiB/s wr, 82 op/s 2024-04-05T23:40:43.009 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381255 5.81 MB/sec execute 242 sec latency 655.150 ms 2024-04-05T23:40:44.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381255 5.79 MB/sec execute 243 sec latency 1655.200 ms 2024-04-05T23:40:44.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:43 smithi016 ceph-mon[38248]: pgmap v222: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 746 KiB/s wr, 65 op/s 2024-04-05T23:40:44.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:43 smithi130 ceph-mon[79417]: pgmap v222: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 746 KiB/s wr, 65 op/s 2024-04-05T23:40:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:43 smithi027 ceph-mon[78597]: pgmap v222: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 746 KiB/s wr, 65 op/s 2024-04-05T23:40:45.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381255 5.76 MB/sec execute 244 sec latency 2655.271 ms 2024-04-05T23:40:46.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381479 5.74 MB/sec execute 245 sec latency 3540.576 ms 2024-04-05T23:40:46.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:45 smithi016 ceph-mon[38248]: pgmap v223: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 746 KiB/s wr, 65 op/s 2024-04-05T23:40:46.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:45 smithi130 ceph-mon[79417]: pgmap v223: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 746 KiB/s wr, 65 op/s 2024-04-05T23:40:46.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:45 smithi027 ceph-mon[78597]: pgmap v223: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 746 KiB/s wr, 65 op/s 2024-04-05T23:40:46.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.727+0000 7fb5e3c00700 1 -- 172.21.15.16:0/2832363573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc101030 msgr2=0x7fb5dc101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:46.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.727+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/2832363573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc101030 0x7fb5dc101410 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009990 tx=0x7fb5d0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.727+0000 7fb5e3c00700 1 -- 172.21.15.16:0/2832363573 shutdown_connections 2024-04-05T23:40:46.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.727+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/2832363573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5dc106150 0x7fb5dc10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.727+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/2832363573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc101030 0x7fb5dc101410 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.727+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/2832363573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5dc1019e0 0x7fb5dc105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.727+0000 7fb5e3c00700 1 -- 172.21.15.16:0/2832363573 >> 172.21.15.16:0/2832363573 conn(0x7fb5dc0fc8b0 msgr2=0x7fb5dc0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:46.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.727+0000 7fb5e3c00700 1 -- 172.21.15.16:0/2832363573 shutdown_connections 2024-04-05T23:40:46.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.728+0000 7fb5e3c00700 1 -- 172.21.15.16:0/2832363573 wait complete. 2024-04-05T23:40:46.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.729+0000 7fb5e3c00700 1 Processor -- start 2024-04-05T23:40:46.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.729+0000 7fb5e3c00700 1 -- start start 2024-04-05T23:40:46.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.729+0000 7fb5e3c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5dc101030 0x7fb5dc1994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.729+0000 7fb5e3c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc106150 0x7fb5dc1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.729+0000 7fb5e3c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5dc19dd80 0x7fb5dc19e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5e3c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc19e800 con 0x7fb5dc101030 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5e3c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc19e970 con 0x7fb5dc106150 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5e3c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc19eb30 con 0x7fb5dc19dd80 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5dbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc106150 0x7fb5dc1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5dbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc106150 0x7fb5dc1999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59458/0 (socket says 172.21.15.16:59458) 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5dbe00700 1 -- 172.21.15.16:0/1800626321 learned_addr learned my addr 172.21.15.16:0/1800626321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5e1200700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5dc101030 0x7fb5dc1994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5dbe00700 1 -- 172.21.15.16:0/1800626321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5dc19dd80 msgr2=0x7fb5dc19e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5dc19dd80 0x7fb5dc19e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5dc19dd80 0x7fb5dc19e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5dbe00700 1 -- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5dc101030 msgr2=0x7fb5dc1994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5dc101030 0x7fb5dc1994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5dbe00700 1 -- 172.21.15.16:0/1800626321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d00095f0 con 0x7fb5dc106150 2024-04-05T23:40:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5e1200700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5dc101030 0x7fb5dc1994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.730+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5dc19dd80 0x7fb5dc19e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:40:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.731+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc106150 0x7fb5dc1999e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800d950 tx=0x7fb5c800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.731+0000 7fb5d9600700 1 -- 172.21.15.16:0/1800626321 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8004220 con 0x7fb5dc106150 2024-04-05T23:40:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.731+0000 7fb5d9600700 1 -- 172.21.15.16:0/1800626321 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5c8007620 con 0x7fb5dc106150 2024-04-05T23:40:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.731+0000 7fb5d9600700 1 -- 172.21.15.16:0/1800626321 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8005690 con 0x7fb5dc106150 2024-04-05T23:40:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.731+0000 7fb5e3c00700 1 -- 172.21.15.16:0/1800626321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc1a6090 con 0x7fb5dc106150 2024-04-05T23:40:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.731+0000 7fb5e3c00700 1 -- 172.21.15.16:0/1800626321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc1a6520 con 0x7fb5dc106150 2024-04-05T23:40:46.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.735+0000 7fb5d9600700 1 -- 172.21.15.16:0/1800626321 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fb5c8004750 con 0x7fb5dc106150 2024-04-05T23:40:46.728 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.737+0000 7fb5f77e5b80 1 Processor -- start 2024-04-05T23:40:46.728 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.737+0000 7fb5f77e5b80 1 -- start start 2024-04-05T23:40:46.728 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.737+0000 7fb5d9600700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c40a1310 0x7fb5c40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.728 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3caf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.728 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5d9600700 1 -- 172.21.15.16:0/1800626321 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fb5c80cca70 con 0x7fb5dc106150 2024-04-05T23:40:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3c5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c32bb80 con 0x556e6c3cab80 2024-04-05T23:40:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c31c930 con 0x556e6c3c55c0 2024-04-05T23:40:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c31a2f0 con 0x556e6c3cb4a0 2024-04-05T23:40:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c40a1310 0x7fb5c40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5e1200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5e1200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3caf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54168/0 (socket says 172.21.15.16:54168) 2024-04-05T23:40:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5e1200700 1 -- 172.21.15.16:0/387158931 learned_addr learned my addr 172.21.15.16:0/387158931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5dbe00700 1 -- 172.21.15.16:0/387158931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 msgr2=0x556e6c3c5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3c5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5dbe00700 1 -- 172.21.15.16:0/387158931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 msgr2=0x556e6c3caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.738+0000 7fb5dbe00700 1 -- 172.21.15.16:0/387158931 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e6c197350 con 0x556e6c3cb4a0 2024-04-05T23:40:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.739+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3c5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:40:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.739+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3c5030 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80ca140 tx=0x7fb5c80994e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b78e8916ff65e73 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.739+0000 7fb5ce000700 1 -- 172.21.15.16:0/387158931 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c809d930 con 0x556e6c3cb4a0 2024-04-05T23:40:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.739+0000 7fb5ce000700 1 -- 172.21.15.16:0/387158931 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5c80a1f80 con 0x556e6c3cb4a0 2024-04-05T23:40:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.739+0000 7fb5ce000700 1 -- 172.21.15.16:0/387158931 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8099230 con 0x556e6c3cb4a0 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/387158931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 msgr2=0x556e6c3c5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c40a1310 0x7fb5c40a37d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x556e6c3c17f0 tx=0x7fb5d4009920 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3c5030 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80ca140 tx=0x7fb5c80994e0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/387158931 shutdown_connections 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3c5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/387158931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3c5030 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/387158931 >> 172.21.15.16:0/387158931 conn(0x556e6c3391c0 msgr2=0x556e6c3bfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/387158931 shutdown_connections 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.740+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/387158931 wait complete. 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5f77e5b80 1 -- start start 2024-04-05T23:40:46.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5f77e5b80 1 Processor -- start 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3c55c0 0x556e6c3ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3cb4a0 0x556e6c3cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c315be0 con 0x556e6c3cab80 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c2b1d20 con 0x556e6c3cb4a0 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c2b19c0 con 0x556e6c3c55c0 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5f77e5b80 10 client.0 _ll_register_callbacks cb 0x556e6c11d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5e1200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3c55c0 0x556e6c3ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.741+0000 7fb5e1c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3cb4a0 0x556e6c3cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5e1200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3c55c0 0x556e6c3ce8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:52310/0 (socket says 172.21.15.16:52310) 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5e1200700 1 -- 172.21.15.16:0/2192394044 learned_addr learned my addr 172.21.15.16:0/2192394044 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5dbe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5e1c00700 1 -- 172.21.15.16:0/2192394044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3c55c0 msgr2=0x556e6c3ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:46.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3c55c0 0x556e6c3ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5e1c00700 1 -- 172.21.15.16:0/2192394044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 msgr2=0x556e6c3cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:46.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5e1c00700 1 -- 172.21.15.16:0/2192394044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc1a6520 con 0x556e6c3cb4a0 2024-04-05T23:40:46.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3cb4a0 0x556e6c3cf320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40060b0 tx=0x7fb5d4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=684d8712d2893614 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:46.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5c2a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:40:46.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5c2a00700 1 -- 172.21.15.16:0/2192394044 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d401ce30 con 0x556e6c3cb4a0 2024-04-05T23:40:46.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.742+0000 7fb5c2a00700 1 -- 172.21.15.16:0/2192394044 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fb5d4019630 con 0x556e6c3cb4a0 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/2192394044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556e6c3b81e0 con 0x556e6c3cb4a0 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5c2a00700 1 -- 172.21.15.16:0/2192394044 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5d40197f0 con 0x556e6c3cb4a0 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5f77e5b80 10 client.26432 fetch_fsmap learned FSMap version 52 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/2192394044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) v3 -- 0x556e6c143b00 con 0x556e6c3cb4a0 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5c2a00700 1 -- 172.21.15.16:0/2192394044 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556e6c3b81e0 con 0x556e6c3cb4a0 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5c2a00700 1 -- 172.21.15.16:0/2192394044 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 52) v1 ==== 7498+0+0 (secure 0 0 0) 0x7fb5d4024a60 con 0x556e6c3cb4a0 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5f77e5b80 10 client.26432 fetch_fsmap finished waiting for FSMap version 52 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5f77e5b80 10 client.26432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5f77e5b80 20 client.26432 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e6c3defa0 0x556e6c3e13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5f77e5b80 4 client.26432 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.743+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/2192394044 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556e6c30e6a0 con 0x556e6c3defa0 2024-04-05T23:40:46.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.744+0000 7fb5e1200700 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e6c3defa0 0x556e6c3e13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.744+0000 7fb5e1200700 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e6c3defa0 0x556e6c3e13c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56c32830f1fc10c2 server_cookie=c8f134c303bc67f9 in_seq=0 out_seq=0 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.744+0000 7fb5c2a00700 10 client.26432 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5c2a00700 1 -- 172.21.15.16:0/2192394044 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556e6c30e6a0 con 0x556e6c3defa0 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5c2a00700 10 client.26432 handle_command_reply: tid=0 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 client.26432 shutdown 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/2192394044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3cb4a0 msgr2=0x556e6c3cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3cb4a0 0x556e6c3cf320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40060b0 tx=0x7fb5d4004ad0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/2192394044 shutdown_connections 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e6c3defa0 0x556e6c3e13c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3cb4a0 0x556e6c3cf320 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5c2a00700 0 client.26432 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3c55c0 0x556e6c3ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/2192394044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/2192394044 >> 172.21.15.16:0/2192394044 conn(0x556e6c3391c0 msgr2=0x556e6c3bf280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/2192394044 shutdown_connections 2024-04-05T23:40:46.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.745+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/2192394044 wait complete. 2024-04-05T23:40:46.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.746+0000 7fb5f77e5b80 20 client.26432 trim_cache size 0 max 16384 2024-04-05T23:40:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.761+0000 7fb5f77e5b80 1 Processor -- start 2024-04-05T23:40:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.761+0000 7fb5f77e5b80 1 -- start start 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3c5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c199640 con 0x556e6c3cab80 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c3ca790 con 0x556e6c3c55c0 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c3cce10 con 0x556e6c3cb4a0 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5e1c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5e1200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5e1c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3bfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:52324/0 (socket says 172.21.15.16:52324) 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5e1c00700 1 -- 172.21.15.16:0/894678945 learned_addr learned my addr 172.21.15.16:0/894678945 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/894678945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5e1c00700 1 -- 172.21.15.16:0/894678945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 msgr2=0x556e6c3c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/894678945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5e1c00700 1 -- 172.21.15.16:0/894678945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 msgr2=0x556e6c3ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/894678945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.762+0000 7fb5e1c00700 1 -- 172.21.15.16:0/894678945 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e6c2dbe40 con 0x556e6c3cb4a0 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.763+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/894678945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3bfa00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40092d0 tx=0x7fb5d4009300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eedf5dd9599bb7e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.763+0000 7fb5cea00700 1 -- 172.21.15.16:0/894678945 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4018540 con 0x556e6c3cb4a0 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.763+0000 7fb5cea00700 1 -- 172.21.15.16:0/894678945 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5d4011040 con 0x556e6c3cb4a0 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.763+0000 7fb5cea00700 1 -- 172.21.15.16:0/894678945 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4024b50 con 0x556e6c3cb4a0 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.763+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/894678945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 msgr2=0x556e6c3bfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/894678945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3bfa00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40092d0 tx=0x7fb5d4009300 comp rx=0 tx=0).stop 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/894678945 shutdown_connections 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/894678945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cb4a0 0x556e6c3bfa00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/894678945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/894678945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cab80 0x556e6c3ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/894678945 >> 172.21.15.16:0/894678945 conn(0x556e6c3391c0 msgr2=0x556e6c3c9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/894678945 shutdown_connections 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/894678945 wait complete. 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 -- start start 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.764+0000 7fb5f77e5b80 1 Processor -- start 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3d5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cab80 0x556e6c3d4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5f77e5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cb4a0 0x556e6c3d46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c3d4c90 con 0x556e6c3cb4a0 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c3d4e00 con 0x556e6c3c55c0 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5f77e5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6c3d4fc0 con 0x556e6c3cab80 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5f77e5b80 10 client.0 _ll_register_callbacks cb 0x556e6c11d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5e1200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3d5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5e1200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3d5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59494/0 (socket says 172.21.15.16:59494) 2024-04-05T23:40:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5e1200700 1 -- 172.21.15.16:0/4121800447 learned_addr learned my addr 172.21.15.16:0/4121800447 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5dbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cab80 0x556e6c3d4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5e1c00700 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cb4a0 0x556e6c3d46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5e1200700 1 -- 172.21.15.16:0/4121800447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cab80 msgr2=0x556e6c3d4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5e1200700 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cab80 0x556e6c3d4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5e1200700 1 -- 172.21.15.16:0/4121800447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cb4a0 msgr2=0x556e6c3d46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5e1200700 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cb4a0 0x556e6c3d46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.765+0000 7fb5e1200700 1 -- 172.21.15.16:0/4121800447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e6c143b00 con 0x556e6c3c55c0 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5e1200700 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3d5d50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x556e6c3e1cd0 tx=0x7fb5d000bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c2d4588efacebcf server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5c2a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5c2a00700 1 -- 172.21.15.16:0/4121800447 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d001d040 con 0x556e6c3c55c0 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/4121800447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5c80a1f80 con 0x556e6c3c55c0 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5c2a00700 1 -- 172.21.15.16:0/4121800447 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fb5d00290b0 con 0x556e6c3c55c0 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5c2a00700 1 -- 172.21.15.16:0/4121800447 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5d0029270 con 0x556e6c3c55c0 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5f77e5b80 10 client.26438 fetch_fsmap learned FSMap version 52 2024-04-05T23:40:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/4121800447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) v3 -- 0x556e6c19bc70 con 0x556e6c3c55c0 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5c2a00700 1 -- 172.21.15.16:0/4121800447 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5c80a1f80 con 0x556e6c3c55c0 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.766+0000 7fb5c2a00700 1 -- 172.21.15.16:0/4121800447 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 52) v1 ==== 7498+0+0 (secure 0 0 0) 0x7fb5d0004620 con 0x556e6c3c55c0 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.767+0000 7fb5f77e5b80 10 client.26438 fetch_fsmap finished waiting for FSMap version 52 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.767+0000 7fb5f77e5b80 10 client.26438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.767+0000 7fb5f77e5b80 20 client.26438 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.767+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e6c4342e0 0x556e6c436700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.767+0000 7fb5f77e5b80 4 client.26438 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.767+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/4121800447 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556e6c164c90 con 0x556e6c4342e0 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.767+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e6c4342e0 0x556e6c436700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.767+0000 7fb5c2a00700 10 client.26438 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.767+0000 7fb5dbe00700 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e6c4342e0 0x556e6c436700 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c832182880db130c server_cookie=36b95d4fac55a48f in_seq=0 out_seq=0 2024-04-05T23:40:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.796+0000 7fb5c2a00700 1 -- 172.21.15.16:0/4121800447 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556e6c164c90 con 0x556e6c4342e0 2024-04-05T23:40:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.796+0000 7fb5c2a00700 10 client.26438 handle_command_reply: tid=0 2024-04-05T23:40:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.796+0000 7fb5f77e5b80 1 client.26438 shutdown 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.796+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/4121800447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 msgr2=0x556e6c3d5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.796+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3d5d50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x556e6c3e1cd0 tx=0x7fb5d000bd10 comp rx=0 tx=0).stop 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.796+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/4121800447 shutdown_connections 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.797+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e6c4342e0 0x556e6c436700 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.797+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e6c3cb4a0 0x556e6c3d46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.797+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e6c3c55c0 0x556e6c3d5d50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.797+0000 7fb5c2a00700 0 client.26438 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.797+0000 7fb5f77e5b80 1 --2- 172.21.15.16:0/4121800447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e6c3cab80 0x556e6c3d4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.797+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/4121800447 >> 172.21.15.16:0/4121800447 conn(0x556e6c3391c0 msgr2=0x556e6c3cb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.797+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/4121800447 shutdown_connections 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.797+0000 7fb5f77e5b80 1 -- 172.21.15.16:0/4121800447 wait complete. 2024-04-05T23:40:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.797+0000 7fb5f77e5b80 20 client.26438 trim_cache size 0 max 16384 2024-04-05T23:40:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:40:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:40:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:40:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 -- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c40a1310 msgr2=0x7fb5c40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c40a1310 0x7fb5c40a37d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x556e6c3c17f0 tx=0x7fb5d4009920 comp rx=0 tx=0).stop 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 -- 172.21.15.16:0/1800626321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc106150 msgr2=0x7fb5dc1999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc106150 0x7fb5dc1999e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800d950 tx=0x7fb5c800dc60 comp rx=0 tx=0).stop 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 -- 172.21.15.16:0/1800626321 shutdown_connections 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c40a1310 0x7fb5c40a37d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5dc101030 0x7fb5dc1994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5dc19dd80 0x7fb5dc19e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 --2- 172.21.15.16:0/1800626321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5dc106150 0x7fb5dc1999e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 -- 172.21.15.16:0/1800626321 >> 172.21.15.16:0/1800626321 conn(0x7fb5dc0fc8b0 msgr2=0x7fb5dc0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 -- 172.21.15.16:0/1800626321 shutdown_connections 2024-04-05T23:40:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:46.798+0000 7fb5e3c00700 1 -- 172.21.15.16:0/1800626321 wait complete. 2024-04-05T23:40:47.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 383364 5.77 MB/sec execute 246 sec latency 15.716 ms 2024-04-05T23:40:48.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384529 5.77 MB/sec execute 247 sec latency 18.679 ms 2024-04-05T23:40:48.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:47 smithi016 ceph-mon[38248]: pgmap v224: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-04-05T23:40:48.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:47 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:40:48.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:47 smithi130 ceph-mon[79417]: pgmap v224: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-04-05T23:40:48.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:47 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:40:48.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:47 smithi027 ceph-mon[78597]: pgmap v224: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-04-05T23:40:48.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:47 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:40:49.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384796 5.75 MB/sec execute 248 sec latency 785.009 ms 2024-04-05T23:40:49.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:48 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:40:49.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:48 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:40:49.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:48 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:40:49.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:48 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:40:49.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:48 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:40:49.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:48 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:40:49.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:48 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:40:49.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:48 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:40:49.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:48 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:40:50.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384796 5.72 MB/sec execute 249 sec latency 1785.077 ms 2024-04-05T23:40:50.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:50 smithi027 ceph-mon[78597]: pgmap v225: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 951 KiB/s wr, 89 op/s 2024-04-05T23:40:50.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:50 smithi016 ceph-mon[38248]: pgmap v225: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 951 KiB/s wr, 89 op/s 2024-04-05T23:40:50.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:50 smithi130 ceph-mon[79417]: pgmap v225: 97 pgs: 97 active+clean; 141 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 951 KiB/s wr, 89 op/s 2024-04-05T23:40:51.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384796 5.70 MB/sec execute 250 sec latency 2785.217 ms 2024-04-05T23:40:52.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 385329 5.68 MB/sec execute 251 sec latency 3507.848 ms 2024-04-05T23:40:52.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:52 smithi027 ceph-mon[78597]: pgmap v226: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 154 op/s 2024-04-05T23:40:52.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:52 smithi016 ceph-mon[38248]: pgmap v226: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 154 op/s 2024-04-05T23:40:52.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:52 smithi130 ceph-mon[79417]: pgmap v226: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 154 op/s 2024-04-05T23:40:53.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 386995 5.71 MB/sec execute 252 sec latency 11.577 ms 2024-04-05T23:40:54.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388285 5.71 MB/sec execute 253 sec latency 18.978 ms 2024-04-05T23:40:54.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:54 smithi027 ceph-mon[78597]: pgmap v227: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-05T23:40:54.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:54 smithi016 ceph-mon[38248]: pgmap v227: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-05T23:40:54.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:54 smithi130 ceph-mon[79417]: pgmap v227: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-05T23:40:55.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388368 5.69 MB/sec execute 254 sec latency 939.365 ms 2024-04-05T23:40:56.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388368 5.66 MB/sec execute 255 sec latency 1939.453 ms 2024-04-05T23:40:56.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:56 smithi027 ceph-mon[78597]: pgmap v228: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-05T23:40:56.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:56 smithi016 ceph-mon[38248]: pgmap v228: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-05T23:40:56.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:56 smithi130 ceph-mon[79417]: pgmap v228: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-05T23:40:57.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 389065 5.65 MB/sec execute 256 sec latency 2660.273 ms 2024-04-05T23:40:58.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 390623 5.67 MB/sec execute 257 sec latency 11.447 ms 2024-04-05T23:40:58.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:58 smithi027 ceph-mon[78597]: pgmap v229: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-04-05T23:40:58.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:58 smithi016 ceph-mon[38248]: pgmap v229: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-04-05T23:40:58.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:58 smithi130 ceph-mon[79417]: pgmap v229: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-04-05T23:40:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 391738 5.67 MB/sec execute 258 sec latency 19.820 ms 2024-04-05T23:40:59.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:40:59 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:40:59.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:40:59 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:40:59.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:40:59 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:40:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.949+0000 7fbf4ce00700 1 -- 172.21.15.16:0/3533386692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 msgr2=0x7fbf48101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.949+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/3533386692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 0x7fbf48101410 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbf380098f0 tx=0x7fbf38009c00 comp rx=0 tx=0).stop 2024-04-05T23:40:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.949+0000 7fbf4ce00700 1 -- 172.21.15.16:0/3533386692 shutdown_connections 2024-04-05T23:40:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.949+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/3533386692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf48106150 0x7fbf4810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.949+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/3533386692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 0x7fbf48101410 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.949+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/3533386692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf481019e0 0x7fbf48105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.949+0000 7fbf4ce00700 1 -- 172.21.15.16:0/3533386692 >> 172.21.15.16:0/3533386692 conn(0x7fbf480fc8b0 msgr2=0x7fbf480fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:59.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.949+0000 7fbf4ce00700 1 -- 172.21.15.16:0/3533386692 shutdown_connections 2024-04-05T23:40:59.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.949+0000 7fbf4ce00700 1 -- 172.21.15.16:0/3533386692 wait complete. 2024-04-05T23:40:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.950+0000 7fbf4ce00700 1 Processor -- start 2024-04-05T23:40:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.950+0000 7fbf4ce00700 1 -- start start 2024-04-05T23:40:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.950+0000 7fbf4ce00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 0x7fbf48197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf4ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf48106150 0x7fbf481978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf4ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf4819bc40 0x7fbf4819c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf4ce00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4819c6c0 con 0x7fbf48106150 2024-04-05T23:40:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf4ce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4819c830 con 0x7fbf4819bc40 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf4ce00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4819c9f0 con 0x7fbf48101030 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 0x7fbf48197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf48106150 0x7fbf481978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 0x7fbf48197360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44678/0 (socket says 172.21.15.16:44678) 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45e00700 1 -- 172.21.15.16:0/1104285065 learned_addr learned my addr 172.21.15.16:0/1104285065 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf46800700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf4819bc40 0x7fbf4819c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45e00700 1 -- 172.21.15.16:0/1104285065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf4819bc40 msgr2=0x7fbf4819c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45e00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf4819bc40 0x7fbf4819c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45e00700 1 -- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf48106150 msgr2=0x7fbf481978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45e00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf48106150 0x7fbf481978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45e00700 1 -- 172.21.15.16:0/1104285065 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf38009520 con 0x7fbf48101030 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.951+0000 7fbf45400700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf48106150 0x7fbf481978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.952+0000 7fbf45e00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 0x7fbf48197360 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbf38005950 tx=0x7fbf3800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:59.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.952+0000 7fbf3ea00700 1 -- 172.21.15.16:0/1104285065 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3801c070 con 0x7fbf48101030 2024-04-05T23:40:59.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.952+0000 7fbf3ea00700 1 -- 172.21.15.16:0/1104285065 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf3800ba40 con 0x7fbf48101030 2024-04-05T23:40:59.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.952+0000 7fbf4ce00700 1 -- 172.21.15.16:0/1104285065 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf481a3ef0 con 0x7fbf48101030 2024-04-05T23:40:59.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.952+0000 7fbf3ea00700 1 -- 172.21.15.16:0/1104285065 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf380038e0 con 0x7fbf48101030 2024-04-05T23:40:59.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.952+0000 7fbf4ce00700 1 -- 172.21.15.16:0/1104285065 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf481a4310 con 0x7fbf48101030 2024-04-05T23:40:59.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.955+0000 7fbf3ea00700 1 -- 172.21.15.16:0/1104285065 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fbf38020030 con 0x7fbf48101030 2024-04-05T23:40:59.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.956+0000 7fbf608e2b80 1 Processor -- start 2024-04-05T23:40:59.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.956+0000 7fbf608e2b80 1 -- start start 2024-04-05T23:40:59.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.956+0000 7fbf3ea00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbf2c0a1310 0x7fbf2c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf3ea00700 1 -- 172.21.15.16:0/1104285065 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fbf380cdb50 con 0x7fbf48101030 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064ce470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064c8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064c8980 0x5586064c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860642ab80 con 0x5586064ce9b0 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55860641b930 con 0x5586064ce090 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586064192f0 con 0x5586064c8980 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf46800700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbf2c0a1310 0x7fbf2c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064c8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33676/0 (socket says 172.21.15.16:33676) 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45400700 1 -- 172.21.15.16:0/2521779127 learned_addr learned my addr 172.21.15.16:0/2521779127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45e00700 1 -- 172.21.15.16:0/2521779127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064c8980 msgr2=0x5586064c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45e00700 1 --2- 172.21.15.16:0/2521779127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064c8980 0x5586064c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45e00700 1 -- 172.21.15.16:0/2521779127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 msgr2=0x5586064c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45e00700 1 --2- 172.21.15.16:0/2521779127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064c8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45e00700 1 -- 172.21.15.16:0/2521779127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558606296350 con 0x5586064ce090 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.957+0000 7fbf45400700 1 --2- 172.21.15.16:0/2521779127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.958+0000 7fbf46800700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbf2c0a1310 0x7fbf2c0a37d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbf400093a0 tx=0x7fbf400099e0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.958+0000 7fbf45e00700 1 --2- 172.21.15.16:0/2521779127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064ce470 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbf380cb220 tx=0x7fbf380970c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8967066d70503e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:59.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.958+0000 7fbf33400700 1 -- 172.21.15.16:0/2521779127 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf380b2070 con 0x5586064ce090 2024-04-05T23:40:59.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.958+0000 7fbf33400700 1 -- 172.21.15.16:0/2521779127 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf3809ac70 con 0x5586064ce090 2024-04-05T23:40:59.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.958+0000 7fbf33400700 1 -- 172.21.15.16:0/2521779127 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38097470 con 0x5586064ce090 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2521779127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 msgr2=0x5586064ce470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2521779127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064ce470 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbf380cb220 tx=0x7fbf380970c0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2521779127 shutdown_connections 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2521779127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064c8980 0x5586064c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2521779127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064ce470 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2521779127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2521779127 >> 172.21.15.16:0/2521779127 conn(0x5586064381c0 msgr2=0x5586064bebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2521779127 shutdown_connections 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2521779127 wait complete. 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.959+0000 7fbf608e2b80 1 -- start start 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf608e2b80 1 Processor -- start 2024-04-05T23:40:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064c8980 0x5586064d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce9b0 0x5586064d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558606414be0 con 0x5586064c8980 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586063b0d20 con 0x5586064ce090 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586063b09c0 con 0x5586064ce9b0 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf608e2b80 10 client.0 _ll_register_callbacks cb 0x55860621c340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf45400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf45400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:57124/0 (socket says 172.21.15.16:57124) 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf45400700 1 -- 172.21.15.16:0/3209905345 learned_addr learned my addr 172.21.15.16:0/3209905345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf46800700 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce9b0 0x5586064d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.960+0000 7fbf45e00700 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064c8980 0x5586064d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf46800700 1 -- 172.21.15.16:0/3209905345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 msgr2=0x5586064d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf46800700 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf46800700 1 -- 172.21.15.16:0/3209905345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064c8980 msgr2=0x5586064d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf46800700 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064c8980 0x5586064d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf46800700 1 -- 172.21.15.16:0/3209905345 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf34009d10 con 0x5586064ce9b0 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf45400700 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf45e00700 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064c8980 0x5586064d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf46800700 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce9b0 0x5586064d2280 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009260 tx=0x7fbf40004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=338d891d54147335 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf27e00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf27e00700 1 -- 172.21.15.16:0/3209905345 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4001d070 con 0x5586064ce9b0 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf27e00700 1 -- 172.21.15.16:0/3209905345 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fbf40016ac0 con 0x5586064ce9b0 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf608e2b80 1 -- 172.21.15.16:0/3209905345 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5586064b71e0 con 0x5586064ce9b0 2024-04-05T23:40:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.961+0000 7fbf27e00700 1 -- 172.21.15.16:0/3209905345 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf4000e050 con 0x5586064ce9b0 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf608e2b80 10 client.26479 fetch_fsmap learned FSMap version 53 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf608e2b80 1 -- 172.21.15.16:0/3209905345 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) v3 -- 0x7fbf481a4310 con 0x5586064ce9b0 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf27e00700 1 -- 172.21.15.16:0/3209905345 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf4002e020 con 0x5586064ce9b0 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf27e00700 1 -- 172.21.15.16:0/3209905345 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 53) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fbf40021cb0 con 0x5586064ce9b0 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf608e2b80 10 client.26479 fetch_fsmap finished waiting for FSMap version 53 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf608e2b80 10 client.26479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf608e2b80 20 client.26479 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5586064e20b0 0x5586064e44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf608e2b80 4 client.26479 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf608e2b80 1 -- 172.21.15.16:0/3209905345 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55860640d6a0 con 0x5586064e20b0 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.962+0000 7fbf45e00700 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5586064e20b0 0x5586064e44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.963+0000 7fbf45e00700 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5586064e20b0 0x5586064e44d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c5de84200ffaa4c server_cookie=ac13128a0002b796 in_seq=0 out_seq=0 2024-04-05T23:40:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.963+0000 7fbf27e00700 10 client.26479 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.963+0000 7fbf27e00700 1 -- 172.21.15.16:0/3209905345 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55860640d6a0 con 0x5586064e20b0 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.963+0000 7fbf27e00700 10 client.26479 handle_command_reply: tid=0 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.963+0000 7fbf608e2b80 1 client.26479 shutdown 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 -- 172.21.15.16:0/3209905345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce9b0 msgr2=0x5586064d2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce9b0 0x5586064d2280 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009260 tx=0x7fbf40004b40 comp rx=0 tx=0).stop 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 -- 172.21.15.16:0/3209905345 shutdown_connections 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5586064e20b0 0x5586064e44d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064c8980 0x5586064d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf27e00700 0 client.26479 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce9b0 0x5586064d2280 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/3209905345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce090 0x5586064d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 -- 172.21.15.16:0/3209905345 >> 172.21.15.16:0/3209905345 conn(0x5586064381c0 msgr2=0x5586064be280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 -- 172.21.15.16:0/3209905345 shutdown_connections 2024-04-05T23:40:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 1 -- 172.21.15.16:0/3209905345 wait complete. 2024-04-05T23:40:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.964+0000 7fbf608e2b80 20 client.26479 trim_cache size 0 max 16384 2024-04-05T23:40:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.979+0000 7fbf608e2b80 1 Processor -- start 2024-04-05T23:40:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.979+0000 7fbf608e2b80 1 -- start start 2024-04-05T23:40:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.979+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064c8980 0x5586064bea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.979+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce090 0x5586064d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.979+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064d09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.979+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558606298640 con 0x5586064ce9b0 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.979+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586064bbaa0 con 0x5586064c8980 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.979+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586064c0820 con 0x5586064ce090 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf46800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064d09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce090 0x5586064d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce090 0x5586064d04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44706/0 (socket says 172.21.15.16:44706) 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064c8980 0x5586064bea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45400700 1 -- 172.21.15.16:0/193712607 learned_addr learned my addr 172.21.15.16:0/193712607 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf46800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064d09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33692/0 (socket says 172.21.15.16:33692) 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45400700 1 -- 172.21.15.16:0/193712607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064c8980 msgr2=0x5586064bea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45400700 1 --2- 172.21.15.16:0/193712607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064c8980 0x5586064bea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45400700 1 -- 172.21.15.16:0/193712607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 msgr2=0x5586064d09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45400700 1 --2- 172.21.15.16:0/193712607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064d09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45400700 1 -- 172.21.15.16:0/193712607 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558606242b00 con 0x5586064ce090 2024-04-05T23:40:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf46800700 1 --2- 172.21.15.16:0/193712607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064d09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.980+0000 7fbf45400700 1 --2- 172.21.15.16:0/193712607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce090 0x5586064d04b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbf34012ce0 tx=0x7fbf34009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1168a1bd9304137 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf33e00700 1 -- 172.21.15.16:0/193712607 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34017070 con 0x5586064ce090 2024-04-05T23:40:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf33e00700 1 -- 172.21.15.16:0/193712607 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf34018040 con 0x5586064ce090 2024-04-05T23:40:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf33e00700 1 -- 172.21.15.16:0/193712607 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340038e0 con 0x5586064ce090 2024-04-05T23:40:59.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf608e2b80 1 -- 172.21.15.16:0/193712607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce090 msgr2=0x5586064d04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/193712607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce090 0x5586064d04b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbf34012ce0 tx=0x7fbf34009260 comp rx=0 tx=0).stop 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf608e2b80 1 -- 172.21.15.16:0/193712607 shutdown_connections 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/193712607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce9b0 0x5586064d09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/193712607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064c8980 0x5586064bea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/193712607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064ce090 0x5586064d04b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf608e2b80 1 -- 172.21.15.16:0/193712607 >> 172.21.15.16:0/193712607 conn(0x5586064381c0 msgr2=0x5586064cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf608e2b80 1 -- 172.21.15.16:0/193712607 shutdown_connections 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.981+0000 7fbf608e2b80 1 -- 172.21.15.16:0/193712607 wait complete. 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.982+0000 7fbf608e2b80 1 -- start start 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.982+0000 7fbf608e2b80 1 Processor -- start 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064c8980 0x5586064d8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce090 0x5586064d5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf608e2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce9b0 0x5586064d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586064d6a50 con 0x5586064ce090 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586064d6bc0 con 0x5586064ce9b0 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf608e2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586064d6d80 con 0x5586064c8980 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf608e2b80 10 client.0 _ll_register_callbacks cb 0x7fbf400187b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf45400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce090 0x5586064d5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf45400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce090 0x5586064d5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33696/0 (socket says 172.21.15.16:33696) 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf45400700 1 -- 172.21.15.16:0/2677912845 learned_addr learned my addr 172.21.15.16:0/2677912845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf46800700 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce9b0 0x5586064d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.983+0000 7fbf45e00700 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064c8980 0x5586064d8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf45400700 1 -- 172.21.15.16:0/2677912845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064c8980 msgr2=0x5586064d8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:59.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf45400700 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064c8980 0x5586064d8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf45400700 1 -- 172.21.15.16:0/2677912845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce9b0 msgr2=0x5586064d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf45400700 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce9b0 0x5586064d64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf45400700 1 -- 172.21.15.16:0/2677912845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf481a4310 con 0x5586064ce090 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf46800700 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce9b0 0x5586064d64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf45400700 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce090 0x5586064d5f90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbf34012cb0 tx=0x7fbf3400fcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181cbaffabff6863 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf27e00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf27e00700 1 -- 172.21.15.16:0/2677912845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34017040 con 0x5586064ce090 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf27e00700 1 -- 172.21.15.16:0/2677912845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fbf34004460 con 0x5586064ce090 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2677912845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf3809ac70 con 0x5586064ce090 2024-04-05T23:40:59.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf27e00700 1 -- 172.21.15.16:0/2677912845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf34012290 con 0x5586064ce090 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf608e2b80 10 client.16407 fetch_fsmap learned FSMap version 53 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.984+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2677912845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) v3 -- 0x7fbf34012990 con 0x5586064ce090 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.985+0000 7fbf27e00700 1 -- 172.21.15.16:0/2677912845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf3809ac70 con 0x5586064ce090 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.985+0000 7fbf27e00700 1 -- 172.21.15.16:0/2677912845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fbf34024070 con 0x5586064ce090 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.985+0000 7fbf608e2b80 10 client.16407 fetch_fsmap finished waiting for FSMap version 53 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.985+0000 7fbf608e2b80 10 client.16407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.985+0000 7fbf608e2b80 20 client.16407 populate_metadata read hostname 'smithi016' 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.985+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x558606537420 0x558606539840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.985+0000 7fbf608e2b80 4 client.16407 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.985+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2677912845 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558606263c90 con 0x558606537420 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.985+0000 7fbf45e00700 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x558606537420 0x558606539840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.986+0000 7fbf45e00700 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x558606537420 0x558606539840 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39924acb80907fad server_cookie=32295bea29fe0664 in_seq=0 out_seq=0 2024-04-05T23:40:59.980 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:40:59.986+0000 7fbf27e00700 10 client.16407 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:00.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf27e00700 1 -- 172.21.15.16:0/2677912845 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558606263c90 con 0x558606537420 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf27e00700 10 client.16407 handle_command_reply: tid=0 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf608e2b80 1 client.16407 shutdown 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2677912845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce090 msgr2=0x5586064d5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce090 0x5586064d5f90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbf34012cb0 tx=0x7fbf3400fcc0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2677912845 shutdown_connections 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x558606537420 0x558606539840 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586064ce9b0 0x5586064d64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586064c8980 0x5586064d8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf27e00700 0 client.16407 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf608e2b80 1 --2- 172.21.15.16:0/2677912845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5586064ce090 0x5586064d5f90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.015+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2677912845 >> 172.21.15.16:0/2677912845 conn(0x5586064381c0 msgr2=0x5586064cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.016+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2677912845 shutdown_connections 2024-04-05T23:41:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.016+0000 7fbf608e2b80 1 -- 172.21.15.16:0/2677912845 wait complete. 2024-04-05T23:41:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.016+0000 7fbf608e2b80 20 client.16407 trim_cache size 0 max 16384 2024-04-05T23:41:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:41:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:41:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:41:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:41:00.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 -- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbf2c0a1310 msgr2=0x7fbf2c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:00.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbf2c0a1310 0x7fbf2c0a37d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbf400093a0 tx=0x7fbf400099e0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 -- 172.21.15.16:0/1104285065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 msgr2=0x7fbf48197360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:00.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 0x7fbf48197360 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbf38005950 tx=0x7fbf3800fa00 comp rx=0 tx=0).stop 2024-04-05T23:41:00.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 -- 172.21.15.16:0/1104285065 shutdown_connections 2024-04-05T23:41:00.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fbf2c0a1310 0x7fbf2c0a37d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf4819bc40 0x7fbf4819c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf48101030 0x7fbf48197360 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 --2- 172.21.15.16:0/1104285065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf48106150 0x7fbf481978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:00.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.017+0000 7fbf4ce00700 1 -- 172.21.15.16:0/1104285065 >> 172.21.15.16:0/1104285065 conn(0x7fbf480fc8b0 msgr2=0x7fbf480fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:00.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.018+0000 7fbf4ce00700 1 -- 172.21.15.16:0/1104285065 shutdown_connections 2024-04-05T23:41:00.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:00.018+0000 7fbf4ce00700 1 -- 172.21.15.16:0/1104285065 wait complete. 2024-04-05T23:41:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: 1 391911 5.65 MB/sec execute 259 sec latency 768.679 ms 2024-04-05T23:41:00.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:00 smithi027 ceph-mon[78597]: pgmap v230: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-05T23:41:00.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:00 smithi016 ceph-mon[38248]: pgmap v230: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-05T23:41:00.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:00 smithi130 ceph-mon[79417]: pgmap v230: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-05T23:41:01.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 393471 5.66 MB/sec execute 260 sec latency 856.698 ms 2024-04-05T23:41:01.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:01 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:01.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:01 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:01.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:01 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:02.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 394317 5.66 MB/sec execute 261 sec latency 26.443 ms 2024-04-05T23:41:02.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:02 smithi027 ceph-mon[78597]: pgmap v231: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-04-05T23:41:02.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:02 smithi016 ceph-mon[38248]: pgmap v231: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-04-05T23:41:02.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:02 smithi130 ceph-mon[79417]: pgmap v231: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-04-05T23:41:03.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395389 5.66 MB/sec execute 262 sec latency 19.612 ms 2024-04-05T23:41:04.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395459 5.64 MB/sec execute 263 sec latency 941.473 ms 2024-04-05T23:41:04.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:04 smithi027 ceph-mon[78597]: pgmap v232: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1005 KiB/s wr, 125 op/s 2024-04-05T23:41:04.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:04 smithi027 ceph-mon[78597]: osdmap e88: 12 total, 12 up, 12 in 2024-04-05T23:41:04.716 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:04 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:04.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:04 smithi016 ceph-mon[38248]: pgmap v232: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1005 KiB/s wr, 125 op/s 2024-04-05T23:41:04.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:04 smithi016 ceph-mon[38248]: osdmap e88: 12 total, 12 up, 12 in 2024-04-05T23:41:04.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:04 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:04.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:04 smithi130 ceph-mon[79417]: pgmap v232: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1005 KiB/s wr, 125 op/s 2024-04-05T23:41:04.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:04 smithi130 ceph-mon[79417]: osdmap e88: 12 total, 12 up, 12 in 2024-04-05T23:41:04.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:04 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:05.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395459 5.61 MB/sec execute 264 sec latency 1941.561 ms 2024-04-05T23:41:06.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395682 5.59 MB/sec execute 265 sec latency 2820.114 ms 2024-04-05T23:41:06.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:06 smithi027 ceph-mon[78597]: pgmap v234: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-04-05T23:41:06.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:06 smithi016 ceph-mon[38248]: pgmap v234: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-04-05T23:41:06.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:06 smithi130 ceph-mon[79417]: pgmap v234: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-04-05T23:41:07.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 397565 5.62 MB/sec execute 266 sec latency 15.919 ms 2024-04-05T23:41:08.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 398734 5.62 MB/sec execute 267 sec latency 19.535 ms 2024-04-05T23:41:08.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:08 smithi027 ceph-mon[78597]: pgmap v235: 97 pgs: 97 active+clean; 143 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1007 KiB/s wr, 177 op/s 2024-04-05T23:41:08.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:08 smithi016 ceph-mon[38248]: pgmap v235: 97 pgs: 97 active+clean; 143 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1007 KiB/s wr, 177 op/s 2024-04-05T23:41:08.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:08 smithi130 ceph-mon[79417]: pgmap v235: 97 pgs: 97 active+clean; 143 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1007 KiB/s wr, 177 op/s 2024-04-05T23:41:09.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 399001 5.60 MB/sec execute 268 sec latency 584.800 ms 2024-04-05T23:41:10.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 399001 5.58 MB/sec execute 269 sec latency 1584.870 ms 2024-04-05T23:41:10.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:10 smithi016 ceph-mon[38248]: pgmap v236: 97 pgs: 97 active+clean; 143 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1007 KiB/s wr, 177 op/s 2024-04-05T23:41:10.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:10 smithi016 ceph-mon[38248]: osdmap e89: 12 total, 12 up, 12 in 2024-04-05T23:41:10.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:10 smithi130 ceph-mon[79417]: pgmap v236: 97 pgs: 97 active+clean; 143 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1007 KiB/s wr, 177 op/s 2024-04-05T23:41:10.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:10 smithi130 ceph-mon[79417]: osdmap e89: 12 total, 12 up, 12 in 2024-04-05T23:41:10.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:10 smithi027 ceph-mon[78597]: pgmap v236: 97 pgs: 97 active+clean; 143 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1007 KiB/s wr, 177 op/s 2024-04-05T23:41:10.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:10 smithi027 ceph-mon[78597]: osdmap e89: 12 total, 12 up, 12 in 2024-04-05T23:41:11.012 INFO:tasks.workunit.client.0.smithi016.stdout: 1 399001 5.56 MB/sec execute 270 sec latency 2584.978 ms 2024-04-05T23:41:11.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:11 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:11.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:11 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:11.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:11 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:12.011 INFO:tasks.workunit.client.0.smithi016.stdout: 1 399705 5.55 MB/sec execute 271 sec latency 3305.220 ms 2024-04-05T23:41:12.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:12 smithi016 ceph-mon[38248]: pgmap v238: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 194 op/s 2024-04-05T23:41:12.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:12 smithi130 ceph-mon[79417]: pgmap v238: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 194 op/s 2024-04-05T23:41:12.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:12 smithi027 ceph-mon[78597]: pgmap v238: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 194 op/s 2024-04-05T23:41:13.012 INFO:tasks.workunit.client.0.smithi016.stdout: 1 400582 5.55 MB/sec execute 272 sec latency 532.813 ms 2024-04-05T23:41:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.174+0000 7fe350200700 1 -- 172.21.15.16:0/2434093737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348101030 msgr2=0x7fe348101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.174+0000 7fe350200700 1 --2- 172.21.15.16:0/2434093737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348101030 0x7fe348101410 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009990 tx=0x7fe33c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.174+0000 7fe350200700 1 -- 172.21.15.16:0/2434093737 shutdown_connections 2024-04-05T23:41:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.174+0000 7fe350200700 1 --2- 172.21.15.16:0/2434093737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348101030 0x7fe348101410 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.174+0000 7fe350200700 1 --2- 172.21.15.16:0/2434093737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348106150 0x7fe34810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.174+0000 7fe350200700 1 --2- 172.21.15.16:0/2434093737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3481019e0 0x7fe348105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.174+0000 7fe350200700 1 -- 172.21.15.16:0/2434093737 >> 172.21.15.16:0/2434093737 conn(0x7fe3480fc8b0 msgr2=0x7fe3480fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.174+0000 7fe350200700 1 -- 172.21.15.16:0/2434093737 shutdown_connections 2024-04-05T23:41:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.174+0000 7fe350200700 1 -- 172.21.15.16:0/2434093737 wait complete. 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.175+0000 7fe350200700 1 Processor -- start 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.175+0000 7fe350200700 1 -- start start 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe350200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101030 0x7fe348199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe350200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348106150 0x7fe348199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe350200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe34819de30 0x7fe34819e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe350200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34819e8b0 con 0x7fe348106150 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe350200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34819ea20 con 0x7fe348101030 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe350200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34819ebe0 con 0x7fe34819de30 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348106150 0x7fe348199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348106150 0x7fe348199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53414/0 (socket says 172.21.15.16:53414) 2024-04-05T23:41:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34ce00700 1 -- 172.21.15.16:0/138980078 learned_addr learned my addr 172.21.15.16:0/138980078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34d800700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101030 0x7fe348199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34e200700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe34819de30 0x7fe34819e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34ce00700 1 -- 172.21.15.16:0/138980078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe34819de30 msgr2=0x7fe34819e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34ce00700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe34819de30 0x7fe34819e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34ce00700 1 -- 172.21.15.16:0/138980078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101030 msgr2=0x7fe348199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34ce00700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101030 0x7fe348199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.176+0000 7fe34ce00700 1 -- 172.21.15.16:0/138980078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c0095f0 con 0x7fe348106150 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.177+0000 7fe34ce00700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348106150 0x7fe348199a90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe33400d950 tx=0x7fe33400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.177+0000 7fe346000700 1 -- 172.21.15.16:0/138980078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334004220 con 0x7fe348106150 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.177+0000 7fe346000700 1 -- 172.21.15.16:0/138980078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe334007620 con 0x7fe348106150 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.177+0000 7fe346000700 1 -- 172.21.15.16:0/138980078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334005690 con 0x7fe348106150 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.177+0000 7fe350200700 1 -- 172.21.15.16:0/138980078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3481a6140 con 0x7fe348106150 2024-04-05T23:41:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.177+0000 7fe350200700 1 -- 172.21.15.16:0/138980078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3481a65d0 con 0x7fe348106150 2024-04-05T23:41:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.180+0000 7fe346000700 1 -- 172.21.15.16:0/138980078 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fe334004750 con 0x7fe348106150 2024-04-05T23:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.181+0000 7fe363c7ab80 1 Processor -- start 2024-04-05T23:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.181+0000 7fe346000700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3300a1310 0x7fe3300a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.181+0000 7fe363c7ab80 1 -- start start 2024-04-05T23:41:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe346000700 1 -- 172.21.15.16:0/138980078 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe3340cca70 con 0x7fe348106150 2024-04-05T23:41:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dc3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe34e200700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3300a1310 0x7fe3300a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc44a0 0x55a956dbe030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dbe5c0 0x55a956dbea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956d24b80 con 0x55a956dc3b80 2024-04-05T23:41:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956d15930 con 0x55a956dbe5c0 2024-04-05T23:41:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956d132f0 con 0x55a956dc44a0 2024-04-05T23:41:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe34d800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dc3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe34d800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dc3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53430/0 (socket says 172.21.15.16:53430) 2024-04-05T23:41:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe34e200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dbe5c0 0x55a956dbea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe34d800700 1 -- 172.21.15.16:0/3628314158 learned_addr learned my addr 172.21.15.16:0/3628314158 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.182+0000 7fe34ce00700 1 --2- 172.21.15.16:0/3628314158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc44a0 0x55a956dbe030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe34d800700 1 -- 172.21.15.16:0/3628314158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc44a0 msgr2=0x55a956dbe030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe34d800700 1 --2- 172.21.15.16:0/3628314158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc44a0 0x55a956dbe030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe34d800700 1 -- 172.21.15.16:0/3628314158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dbe5c0 msgr2=0x55a956dbea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe34d800700 1 --2- 172.21.15.16:0/3628314158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dbe5c0 0x55a956dbea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe34d800700 1 -- 172.21.15.16:0/3628314158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a956b90350 con 0x55a956dc3b80 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe34e200700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3300a1310 0x7fe3300a37d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe3400076f0 tx=0x7fe340007ab0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe34d800700 1 --2- 172.21.15.16:0/3628314158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dc3f60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00b5b0 tx=0x7fe33c00b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc6800222e56c31 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe33aa00700 1 -- 172.21.15.16:0/3628314158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c01d070 con 0x55a956dc3b80 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe33aa00700 1 -- 172.21.15.16:0/3628314158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe33c01e040 con 0x55a956dc3b80 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.183+0000 7fe33aa00700 1 -- 172.21.15.16:0/3628314158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c004760 con 0x55a956dc3b80 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.184+0000 7fe363c7ab80 1 -- 172.21.15.16:0/3628314158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 msgr2=0x55a956dc3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.184+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/3628314158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dc3f60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00b5b0 tx=0x7fe33c00b5e0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.184+0000 7fe363c7ab80 1 -- 172.21.15.16:0/3628314158 shutdown_connections 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.184+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/3628314158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dc3f60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.184+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/3628314158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dbe5c0 0x55a956dbea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.184+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/3628314158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc44a0 0x55a956dbe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.184+0000 7fe363c7ab80 1 -- 172.21.15.16:0/3628314158 >> 172.21.15.16:0/3628314158 conn(0x55a956d321c0 msgr2=0x55a956db8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.184+0000 7fe363c7ab80 1 -- 172.21.15.16:0/3628314158 shutdown_connections 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.184+0000 7fe363c7ab80 1 -- 172.21.15.16:0/3628314158 wait complete. 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.185+0000 7fe363c7ab80 1 -- start start 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.185+0000 7fe363c7ab80 1 Processor -- start 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.185+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dbe5c0 0x55a956dc77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.185+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc3b80 0x55a956dc7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.185+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dc8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.185+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956d0ebe0 con 0x55a956dbe5c0 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.185+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956caad20 con 0x55a956dc44a0 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.185+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956caa9c0 con 0x55a956dc3b80 2024-04-05T23:41:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.185+0000 7fe363c7ab80 10 client.0 _ll_register_callbacks cb 0x55a956b16340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34d800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dbe5c0 0x55a956dc77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34d800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dbe5c0 0x55a956dc77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53446/0 (socket says 172.21.15.16:53446) 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34d800700 1 -- 172.21.15.16:0/622881751 learned_addr learned my addr 172.21.15.16:0/622881751 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34ce00700 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc3b80 0x55a956dc7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34e200700 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dc8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34d800700 1 -- 172.21.15.16:0/622881751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc3b80 msgr2=0x55a956dc7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34d800700 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc3b80 0x55a956dc7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34d800700 1 -- 172.21.15.16:0/622881751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 msgr2=0x55a956dc8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34d800700 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dc8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34d800700 1 -- 172.21.15.16:0/622881751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34810e540 con 0x55a956dbe5c0 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe34d800700 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dbe5c0 0x55a956dc77f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00b690 tx=0x7fe33c005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec36d4c456e1b99 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.186+0000 7fe32f400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:41:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.187+0000 7fe32f400700 1 -- 172.21.15.16:0/622881751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c01d030 con 0x55a956dbe5c0 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.187+0000 7fe32f400700 1 -- 172.21.15.16:0/622881751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe33c004500 con 0x55a956dbe5c0 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.187+0000 7fe363c7ab80 1 -- 172.21.15.16:0/622881751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a956db11e0 con 0x55a956dbe5c0 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.187+0000 7fe32f400700 1 -- 172.21.15.16:0/622881751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe33c00bbc0 con 0x55a956dbe5c0 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.187+0000 7fe363c7ab80 10 client.16419 fetch_fsmap learned FSMap version 56 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.187+0000 7fe363c7ab80 1 -- 172.21.15.16:0/622881751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) v3 -- 0x55a956b3cb00 con 0x55a956dbe5c0 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.187+0000 7fe32f400700 1 -- 172.21.15.16:0/622881751 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a956db11e0 con 0x55a956dbe5c0 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.187+0000 7fe32f400700 1 -- 172.21.15.16:0/622881751 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 56) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fe33c00f800 con 0x55a956dbe5c0 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.188+0000 7fe363c7ab80 10 client.16419 fetch_fsmap finished waiting for FSMap version 56 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.188+0000 7fe363c7ab80 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.188+0000 7fe363c7ab80 20 client.16419 populate_metadata read hostname 'smithi016' 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.188+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a956dd7a50 0x55a956dd9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.188+0000 7fe363c7ab80 4 client.16419 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:41:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.188+0000 7fe363c7ab80 1 -- 172.21.15.16:0/622881751 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a956d076a0 con 0x55a956dd7a50 2024-04-05T23:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.188+0000 7fe34ce00700 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a956dd7a50 0x55a956dd9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.189+0000 7fe34ce00700 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a956dd7a50 0x55a956dd9e40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bae43e4751db6410 server_cookie=89b4cdfd53ae5aed in_seq=0 out_seq=0 2024-04-05T23:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.189+0000 7fe32f400700 10 client.16419 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.189+0000 7fe32f400700 1 -- 172.21.15.16:0/622881751 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a956d076a0 con 0x55a956dd7a50 2024-04-05T23:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.189+0000 7fe32f400700 10 client.16419 handle_command_reply: tid=0 2024-04-05T23:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.189+0000 7fe363c7ab80 1 client.16419 shutdown 2024-04-05T23:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.189+0000 7fe363c7ab80 1 -- 172.21.15.16:0/622881751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dbe5c0 msgr2=0x55a956dc77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.189+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dbe5c0 0x55a956dc77f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00b690 tx=0x7fe33c005c30 comp rx=0 tx=0).stop 2024-04-05T23:41:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe363c7ab80 1 -- 172.21.15.16:0/622881751 shutdown_connections 2024-04-05T23:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a956dd7a50 0x55a956dd9e40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dbe5c0 0x55a956dc77f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dc8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe32f400700 0 client.16419 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/622881751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dc3b80 0x55a956dc7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe363c7ab80 1 -- 172.21.15.16:0/622881751 >> 172.21.15.16:0/622881751 conn(0x55a956d321c0 msgr2=0x55a956db8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe363c7ab80 1 -- 172.21.15.16:0/622881751 shutdown_connections 2024-04-05T23:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe363c7ab80 1 -- 172.21.15.16:0/622881751 wait complete. 2024-04-05T23:41:13.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.190+0000 7fe363c7ab80 20 client.16419 trim_cache size 0 max 16384 2024-04-05T23:41:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.205+0000 7fe363c7ab80 1 Processor -- start 2024-04-05T23:41:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.205+0000 7fe363c7ab80 1 -- start start 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956db8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dc7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956b92640 con 0x55a956dc3b80 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956db0de0 con 0x55a956dc44a0 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956dcf4b0 con 0x55a956dbe5c0 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe34ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956db8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe34ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956db8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53458/0 (socket says 172.21.15.16:53458) 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe34ce00700 1 -- 172.21.15.16:0/1856551593 learned_addr learned my addr 172.21.15.16:0/1856551593 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe34d800700 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe34e200700 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dc7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.206+0000 7fe34ce00700 1 -- 172.21.15.16:0/1856551593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 msgr2=0x55a956dceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.207+0000 7fe34ce00700 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.207+0000 7fe34ce00700 1 -- 172.21.15.16:0/1856551593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 msgr2=0x55a956dc7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.207+0000 7fe34ce00700 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dc7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.207+0000 7fe34ce00700 1 -- 172.21.15.16:0/1856551593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a956cd4e40 con 0x55a956dc3b80 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.207+0000 7fe34d800700 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.207+0000 7fe34ce00700 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956db8a00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe334099a90 tx=0x7fe33400df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00de5ba2095eb41 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.207+0000 7fe33b400700 1 -- 172.21.15.16:0/1856551593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334099b00 con 0x55a956dc3b80 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.207+0000 7fe33b400700 1 -- 172.21.15.16:0/1856551593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe33409d930 con 0x55a956dc3b80 2024-04-05T23:41:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.207+0000 7fe33b400700 1 -- 172.21.15.16:0/1856551593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3340b24b0 con 0x55a956dc3b80 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1856551593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 msgr2=0x55a956db8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956db8a00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe334099a90 tx=0x7fe33400df90 comp rx=0 tx=0).stop 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1856551593 shutdown_connections 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dc7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1856551593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956db8a00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1856551593 >> 172.21.15.16:0/1856551593 conn(0x55a956d321c0 msgr2=0x55a956dc60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1856551593 shutdown_connections 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1856551593 wait complete. 2024-04-05T23:41:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 -- start start 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.208+0000 7fe363c7ab80 1 Processor -- start 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dcea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe363c7ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dcd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956dcd960 con 0x55a956dc3b80 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956dcdad0 con 0x55a956dc44a0 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe363c7ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a956dcdc90 con 0x55a956dbe5c0 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe363c7ab80 10 client.0 _ll_register_callbacks cb 0x55a956b16340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe34ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe34e200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dcd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe34ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53464/0 (socket says 172.21.15.16:53464) 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe34d800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dcea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.209+0000 7fe34ce00700 1 -- 172.21.15.16:0/1681489861 learned_addr learned my addr 172.21.15.16:0/1681489861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe34e200700 1 -- 172.21.15.16:0/1681489861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 msgr2=0x55a956dcea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe34e200700 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dcea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe34e200700 1 -- 172.21.15.16:0/1681489861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 msgr2=0x55a956dccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe34e200700 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe34e200700 1 -- 172.21.15.16:0/1681489861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a956b3cb00 con 0x55a956dc44a0 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe34d800700 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dcea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe34ce00700 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe34e200700 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dcd420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe3400094b0 tx=0x7fe340004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1883f8161e1994ce server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe32f400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe32f400700 1 -- 172.21.15.16:0/1681489861 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34000e050 con 0x55a956dc44a0 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1681489861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe33c01e040 con 0x55a956dc44a0 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.210+0000 7fe32f400700 1 -- 172.21.15.16:0/1681489861 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe340025450 con 0x55a956dc44a0 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe32f400700 1 -- 172.21.15.16:0/1681489861 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe340025610 con 0x55a956dc44a0 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe363c7ab80 10 client.26474 fetch_fsmap learned FSMap version 56 2024-04-05T23:41:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1681489861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) v3 -- 0x7fe3481a65d0 con 0x55a956dc44a0 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe32f400700 1 -- 172.21.15.16:0/1681489861 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe33c01e040 con 0x55a956dc44a0 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe32f400700 1 -- 172.21.15.16:0/1681489861 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 56) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fe340024650 con 0x55a956dc44a0 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe363c7ab80 10 client.26474 fetch_fsmap finished waiting for FSMap version 56 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe363c7ab80 10 client.26474 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe363c7ab80 20 client.26474 populate_metadata read hostname 'smithi016' 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a956e2cc20 0x55a956e2f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe363c7ab80 4 client.26474 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1681489861 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a956b5dc90 con 0x55a956e2cc20 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.211+0000 7fe34d800700 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a956e2cc20 0x55a956e2f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.212+0000 7fe34d800700 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a956e2cc20 0x55a956e2f040 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4221d5186dd42b5e server_cookie=cf13a959eb5b66ea in_seq=0 out_seq=0 2024-04-05T23:41:13.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.212+0000 7fe32f400700 10 client.26474 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe32f400700 1 -- 172.21.15.16:0/1681489861 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a956b5dc90 con 0x55a956e2cc20 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe32f400700 10 client.26474 handle_command_reply: tid=0 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe363c7ab80 1 client.26474 shutdown 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1681489861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 msgr2=0x55a956dcd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dcd420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe3400094b0 tx=0x7fe340004b40 comp rx=0 tx=0).stop 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1681489861 shutdown_connections 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a956e2cc20 0x55a956e2f040 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a956dc44a0 0x55a956dcd420 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a956dbe5c0 0x55a956dcea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe32f400700 0 client.26474 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe363c7ab80 1 --2- 172.21.15.16:0/1681489861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a956dc3b80 0x55a956dccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.240+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1681489861 >> 172.21.15.16:0/1681489861 conn(0x55a956d321c0 msgr2=0x55a956dc4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.241+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1681489861 shutdown_connections 2024-04-05T23:41:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.241+0000 7fe363c7ab80 1 -- 172.21.15.16:0/1681489861 wait complete. 2024-04-05T23:41:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.241+0000 7fe363c7ab80 20 client.26474 trim_cache size 0 max 16384 2024-04-05T23:41:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:41:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:41:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:41:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:41:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.241+0000 7fe350200700 1 -- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3300a1310 msgr2=0x7fe3300a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.241+0000 7fe350200700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3300a1310 0x7fe3300a37d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe3400076f0 tx=0x7fe340007ab0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.241+0000 7fe350200700 1 -- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348106150 msgr2=0x7fe348199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.241+0000 7fe350200700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348106150 0x7fe348199a90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe33400d950 tx=0x7fe33400dc60 comp rx=0 tx=0).stop 2024-04-05T23:41:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.242+0000 7fe350200700 1 -- 172.21.15.16:0/138980078 shutdown_connections 2024-04-05T23:41:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.242+0000 7fe350200700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3300a1310 0x7fe3300a37d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.242+0000 7fe350200700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe34819de30 0x7fe34819e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.242+0000 7fe350200700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe348101030 0x7fe348199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.242+0000 7fe350200700 1 --2- 172.21.15.16:0/138980078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe348106150 0x7fe348199a90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.242+0000 7fe350200700 1 -- 172.21.15.16:0/138980078 >> 172.21.15.16:0/138980078 conn(0x7fe3480fc8b0 msgr2=0x7fe3480fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.242+0000 7fe350200700 1 -- 172.21.15.16:0/138980078 shutdown_connections 2024-04-05T23:41:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:13.242+0000 7fe350200700 1 -- 172.21.15.16:0/138980078 wait complete. 2024-04-05T23:41:14.012 INFO:tasks.workunit.client.0.smithi016.stdout: 1 400582 5.53 MB/sec execute 273 sec latency 1532.936 ms 2024-04-05T23:41:14.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:14 smithi016 ceph-mon[38248]: pgmap v239: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 167 op/s 2024-04-05T23:41:14.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:14 smithi130 ceph-mon[79417]: pgmap v239: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 167 op/s 2024-04-05T23:41:14.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:14 smithi027 ceph-mon[78597]: pgmap v239: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 167 op/s 2024-04-05T23:41:15.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 400582 5.51 MB/sec execute 274 sec latency 2533.052 ms 2024-04-05T23:41:16.012 INFO:tasks.workunit.client.0.smithi016.stdout: 1 400582 5.49 MB/sec execute 275 sec latency 3533.162 ms 2024-04-05T23:41:16.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:16 smithi016 ceph-mon[38248]: pgmap v240: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 155 op/s 2024-04-05T23:41:16.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:16 smithi130 ceph-mon[79417]: pgmap v240: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 155 op/s 2024-04-05T23:41:16.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:16 smithi027 ceph-mon[78597]: pgmap v240: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 155 op/s 2024-04-05T23:41:17.012 INFO:tasks.workunit.client.0.smithi016.stdout: 1 400582 5.47 MB/sec execute 276 sec latency 4533.304 ms 2024-04-05T23:41:18.012 INFO:tasks.workunit.client.0.smithi016.stdout: 1 400582 5.45 MB/sec execute 277 sec latency 5533.418 ms 2024-04-05T23:41:18.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:18 smithi016 ceph-mon[38248]: pgmap v241: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-05T23:41:18.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:18 smithi130 ceph-mon[79417]: pgmap v241: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-05T23:41:18.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:18 smithi027 ceph-mon[78597]: pgmap v241: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-05T23:41:19.012 INFO:tasks.workunit.client.0.smithi016.stdout: 1 400582 5.43 MB/sec execute 278 sec latency 6533.554 ms 2024-04-05T23:41:20.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 400582 5.41 MB/sec execute 279 sec latency 7533.674 ms 2024-04-05T23:41:20.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:20 smithi016 ceph-mon[38248]: pgmap v242: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-05T23:41:20.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:20 smithi130 ceph-mon[79417]: pgmap v242: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-05T23:41:20.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:20 smithi027 ceph-mon[78597]: pgmap v242: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-05T23:41:21.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 401591 5.41 MB/sec execute 280 sec latency 7623.034 ms 2024-04-05T23:41:22.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 403260 5.42 MB/sec execute 281 sec latency 109.296 ms 2024-04-05T23:41:22.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:22 smithi016 ceph-mon[38248]: pgmap v243: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-04-05T23:41:22.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:22 smithi130 ceph-mon[79417]: pgmap v243: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-04-05T23:41:22.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:22 smithi027 ceph-mon[78597]: pgmap v243: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.4 MiB/s wr, 120 op/s 2024-04-05T23:41:23.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 404806 5.44 MB/sec execute 282 sec latency 10.884 ms 2024-04-05T23:41:24.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 405934 5.44 MB/sec execute 283 sec latency 18.989 ms 2024-04-05T23:41:24.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:24 smithi016 ceph-mon[38248]: pgmap v244: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 915 KiB/s wr, 61 op/s 2024-04-05T23:41:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:24 smithi130 ceph-mon[79417]: pgmap v244: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 915 KiB/s wr, 61 op/s 2024-04-05T23:41:24.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:24 smithi027 ceph-mon[78597]: pgmap v244: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 915 KiB/s wr, 61 op/s 2024-04-05T23:41:25.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 406092 5.42 MB/sec execute 284 sec latency 815.589 ms 2024-04-05T23:41:26.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 406092 5.40 MB/sec execute 285 sec latency 1815.721 ms 2024-04-05T23:41:26.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.410+0000 7f9e55600700 1 -- 172.21.15.16:0/2670170390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50101030 msgr2=0x7f9e50101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:26.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.410+0000 7f9e55600700 1 --2- 172.21.15.16:0/2670170390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50101030 0x7f9e50101410 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009990 tx=0x7f9e40009ca0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.410+0000 7f9e55600700 1 -- 172.21.15.16:0/2670170390 shutdown_connections 2024-04-05T23:41:26.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.410+0000 7f9e55600700 1 --2- 172.21.15.16:0/2670170390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50101030 0x7f9e50101410 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.410+0000 7f9e55600700 1 --2- 172.21.15.16:0/2670170390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e50106150 0x7f9e5010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.410+0000 7f9e55600700 1 --2- 172.21.15.16:0/2670170390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e501019e0 0x7f9e50105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.410+0000 7f9e55600700 1 -- 172.21.15.16:0/2670170390 >> 172.21.15.16:0/2670170390 conn(0x7f9e500fc8b0 msgr2=0x7f9e500fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.410+0000 7f9e55600700 1 -- 172.21.15.16:0/2670170390 shutdown_connections 2024-04-05T23:41:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.410+0000 7f9e55600700 1 -- 172.21.15.16:0/2670170390 wait complete. 2024-04-05T23:41:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.411+0000 7f9e55600700 1 Processor -- start 2024-04-05T23:41:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.411+0000 7f9e55600700 1 -- start start 2024-04-05T23:41:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e55600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e50101030 0x7f9e50197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e55600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50106150 0x7f9e50197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e55600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e5019bb90 0x7f9e5019c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e55600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5019c610 con 0x7f9e50106150 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e55600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5019c780 con 0x7f9e5019bb90 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e55600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5019c940 con 0x7f9e50101030 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50106150 0x7f9e50197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50106150 0x7f9e50197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51160/0 (socket says 172.21.15.16:51160) 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4e000700 1 -- 172.21.15.16:0/1732471591 learned_addr learned my addr 172.21.15.16:0/1732471591 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e50101030 0x7f9e50197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4f400700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e5019bb90 0x7f9e5019c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4e000700 1 -- 172.21.15.16:0/1732471591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e50101030 msgr2=0x7f9e50197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4e000700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e50101030 0x7f9e50197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4e000700 1 -- 172.21.15.16:0/1732471591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e5019bb90 msgr2=0x7f9e5019c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4e000700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e5019bb90 0x7f9e5019c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4e000700 1 -- 172.21.15.16:0/1732471591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e400095f0 con 0x7f9e50106150 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.412+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e50101030 0x7f9e50197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.413+0000 7f9e4e000700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50106150 0x7f9e50197840 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00e970 tx=0x7f9e3c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.413+0000 7f9e47400700 1 -- 172.21.15.16:0/1732471591 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c009790 con 0x7f9e50106150 2024-04-05T23:41:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.413+0000 7f9e47400700 1 -- 172.21.15.16:0/1732471591 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e3c00c7e0 con 0x7f9e50106150 2024-04-05T23:41:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.413+0000 7f9e55600700 1 -- 172.21.15.16:0/1732471591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e501a3ea0 con 0x7f9e50106150 2024-04-05T23:41:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.413+0000 7f9e47400700 1 -- 172.21.15.16:0/1732471591 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c0054e0 con 0x7f9e50106150 2024-04-05T23:41:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.413+0000 7f9e55600700 1 -- 172.21.15.16:0/1732471591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e501a4370 con 0x7f9e50106150 2024-04-05T23:41:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.417+0000 7f9e47400700 1 -- 172.21.15.16:0/1732471591 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f9e3c00c950 con 0x7f9e50106150 2024-04-05T23:41:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e69097b80 1 Processor -- start 2024-04-05T23:41:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e47400700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e340a1310 0x7f9e340a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e69097b80 1 -- start start 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e47400700 1 -- 172.21.15.16:0/1732471591 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f9e3c014070 con 0x7f9e50106150 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e4f400700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e340a1310 0x7f9e340a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177d8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 0x5645177d3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e69097b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x564517739b80 con 0x5645177d35c0 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e69097b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56451772a930 con 0x5645177d8b80 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.418+0000 7f9e69097b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645177282f0 con 0x5645177d94a0 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177d8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177d8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:57458/0 (socket says 172.21.15.16:57458) 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4ea00700 1 -- 172.21.15.16:0/3583036070 learned_addr learned my addr 172.21.15.16:0/3583036070 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4f400700 1 --2- 172.21.15.16:0/3583036070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4e000700 1 --2- 172.21.15.16:0/3583036070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 0x5645177d3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4ea00700 1 -- 172.21.15.16:0/3583036070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 msgr2=0x5645177d3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:26.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/3583036070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 0x5645177d3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4ea00700 1 -- 172.21.15.16:0/3583036070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 msgr2=0x5645177d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:26.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/3583036070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4ea00700 1 -- 172.21.15.16:0/3583036070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645175a5350 con 0x5645177d8b80 2024-04-05T23:41:26.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4f400700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e340a1310 0x7f9e340a37d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9e48003fd0 tx=0x7f9e48004040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.419+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/3583036070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177d8f60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000b5b0 tx=0x7f9e40005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d70babfd227f401 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.420+0000 7f9e3b400700 1 -- 172.21.15.16:0/3583036070 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4001d050 con 0x5645177d8b80 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.420+0000 7f9e3b400700 1 -- 172.21.15.16:0/3583036070 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e4001e040 con 0x5645177d8b80 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.420+0000 7f9e3b400700 1 -- 172.21.15.16:0/3583036070 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e40005450 con 0x5645177d8b80 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.420+0000 7f9e69097b80 1 -- 172.21.15.16:0/3583036070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 msgr2=0x5645177d8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.420+0000 7f9e69097b80 1 --2- 172.21.15.16:0/3583036070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177d8f60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000b5b0 tx=0x7f9e40005bc0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.421+0000 7f9e69097b80 1 -- 172.21.15.16:0/3583036070 shutdown_connections 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.421+0000 7f9e69097b80 1 --2- 172.21.15.16:0/3583036070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.421+0000 7f9e69097b80 1 --2- 172.21.15.16:0/3583036070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177d8f60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.421+0000 7f9e69097b80 1 --2- 172.21.15.16:0/3583036070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 0x5645177d3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.421+0000 7f9e69097b80 1 -- 172.21.15.16:0/3583036070 >> 172.21.15.16:0/3583036070 conn(0x5645177471c0 msgr2=0x5645177cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.421+0000 7f9e69097b80 1 -- 172.21.15.16:0/3583036070 shutdown_connections 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.421+0000 7f9e69097b80 1 -- 172.21.15.16:0/3583036070 wait complete. 2024-04-05T23:41:26.414 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.421+0000 7f9e69097b80 1 -- start start 2024-04-05T23:41:26.415 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.421+0000 7f9e69097b80 1 Processor -- start 2024-04-05T23:41:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177df570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177dc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 0x5645177dcee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e69097b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x564517723be0 con 0x5645177d35c0 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e69097b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645176bfd20 con 0x5645177d8b80 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e69097b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645176bf9c0 con 0x5645177d94a0 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e69097b80 10 client.0 _ll_register_callbacks cb 0x56451752b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e4ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177df570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e4e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177dc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e4e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177dc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:57464/0 (socket says 172.21.15.16:57464) 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e4e000700 1 -- 172.21.15.16:0/1663646566 learned_addr learned my addr 172.21.15.16:0/1663646566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.422+0000 7f9e4f400700 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 0x5645177dcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e4e000700 1 -- 172.21.15.16:0/1663646566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 msgr2=0x5645177dcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e4e000700 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 0x5645177dcee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e4e000700 1 -- 172.21.15.16:0/1663646566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 msgr2=0x5645177df570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e4e000700 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177df570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e4e000700 1 -- 172.21.15.16:0/1663646566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5010e540 con 0x5645177d8b80 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e4e000700 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177dc7f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c096480 tx=0x7f9e3c09a790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6042bb58f92a1c10 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e2fe00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e2fe00700 1 -- 172.21.15.16:0/1663646566 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c0a2400 con 0x5645177d8b80 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e2fe00700 1 -- 172.21.15.16:0/1663646566 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f9e3c016dd0 con 0x5645177d8b80 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e69097b80 1 -- 172.21.15.16:0/1663646566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5645177c61e0 con 0x5645177d8b80 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e2fe00700 1 -- 172.21.15.16:0/1663646566 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e3c0b8250 con 0x5645177d8b80 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e69097b80 10 client.26486 fetch_fsmap learned FSMap version 57 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.423+0000 7f9e69097b80 1 -- 172.21.15.16:0/1663646566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) v3 -- 0x564517551b00 con 0x5645177d8b80 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.424+0000 7f9e2fe00700 1 -- 172.21.15.16:0/1663646566 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5645177c61e0 con 0x5645177d8b80 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.424+0000 7f9e2fe00700 1 -- 172.21.15.16:0/1663646566 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 57) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f9e3c099470 con 0x5645177d8b80 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.424+0000 7f9e69097b80 10 client.26486 fetch_fsmap finished waiting for FSMap version 57 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.424+0000 7f9e69097b80 10 client.26486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:41:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.424+0000 7f9e69097b80 20 client.26486 populate_metadata read hostname 'smithi016' 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.424+0000 7f9e69097b80 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5645177ece70 0x5645177ef290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.424+0000 7f9e69097b80 4 client.26486 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.424+0000 7f9e69097b80 1 -- 172.21.15.16:0/1663646566 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56451771c6a0 con 0x5645177ece70 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.424+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5645177ece70 0x5645177ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.425+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5645177ece70 0x5645177ef290 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da10e24bacbc836 server_cookie=cd74a4c6140ee70 in_seq=0 out_seq=0 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.425+0000 7f9e2fe00700 10 client.26486 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e2fe00700 1 -- 172.21.15.16:0/1663646566 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56451771c6a0 con 0x5645177ece70 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e2fe00700 10 client.26486 handle_command_reply: tid=0 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 client.26486 shutdown 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 -- 172.21.15.16:0/1663646566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 msgr2=0x5645177dc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177dc7f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c096480 tx=0x7f9e3c09a790 comp rx=0 tx=0).stop 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 -- 172.21.15.16:0/1663646566 shutdown_connections 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5645177ece70 0x5645177ef290 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177df570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d94a0 0x5645177dcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 --2- 172.21.15.16:0/1663646566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d8b80 0x5645177dc7f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e2fe00700 0 client.26486 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 -- 172.21.15.16:0/1663646566 >> 172.21.15.16:0/1663646566 conn(0x5645177471c0 msgr2=0x5645177cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 -- 172.21.15.16:0/1663646566 shutdown_connections 2024-04-05T23:41:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 1 -- 172.21.15.16:0/1663646566 wait complete. 2024-04-05T23:41:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.426+0000 7f9e69097b80 20 client.26486 trim_cache size 0 max 16384 2024-04-05T23:41:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.442+0000 7f9e69097b80 1 Processor -- start 2024-04-05T23:41:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.442+0000 7f9e69097b80 1 -- start start 2024-04-05T23:41:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177e3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d8b80 0x5645177e8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 0x5645177cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e69097b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645175a7640 con 0x5645177d35c0 2024-04-05T23:41:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e69097b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645177cf400 con 0x5645177d94a0 2024-04-05T23:41:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e69097b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645177e4460 con 0x5645177d8b80 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4e000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d8b80 0x5645177e8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177e3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4e000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d8b80 0x5645177e8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:38710/0 (socket says 172.21.15.16:38710) 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4e000700 1 -- 172.21.15.16:0/401602592 learned_addr learned my addr 172.21.15.16:0/401602592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4f400700 1 --2- 172.21.15.16:0/401602592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 0x5645177cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4e000700 1 -- 172.21.15.16:0/401602592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 msgr2=0x5645177cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4e000700 1 --2- 172.21.15.16:0/401602592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 0x5645177cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4e000700 1 -- 172.21.15.16:0/401602592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 msgr2=0x5645177e3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4e000700 1 --2- 172.21.15.16:0/401602592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177e3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4e000700 1 -- 172.21.15.16:0/401602592 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645176e9e40 con 0x5645177d8b80 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.443+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/401602592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177e3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.444+0000 7f9e4e000700 1 --2- 172.21.15.16:0/401602592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d8b80 0x5645177e8d60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c099c80 tx=0x7f9e3c098530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74e39db943834444 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.444+0000 7f9e3be00700 1 -- 172.21.15.16:0/401602592 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c0a2090 con 0x5645177d8b80 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.444+0000 7f9e3be00700 1 -- 172.21.15.16:0/401602592 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e3c0a2400 con 0x5645177d8b80 2024-04-05T23:41:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.444+0000 7f9e3be00700 1 -- 172.21.15.16:0/401602592 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c0b8350 con 0x5645177d8b80 2024-04-05T23:41:26.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.444+0000 7f9e69097b80 1 -- 172.21.15.16:0/401602592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d8b80 msgr2=0x5645177e8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:26.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.444+0000 7f9e69097b80 1 --2- 172.21.15.16:0/401602592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d8b80 0x5645177e8d60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c099c80 tx=0x7f9e3c098530 comp rx=0 tx=0).stop 2024-04-05T23:41:26.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.445+0000 7f9e69097b80 1 -- 172.21.15.16:0/401602592 shutdown_connections 2024-04-05T23:41:26.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.445+0000 7f9e69097b80 1 --2- 172.21.15.16:0/401602592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d35c0 0x5645177e3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.445+0000 7f9e69097b80 1 --2- 172.21.15.16:0/401602592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 0x5645177cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.445+0000 7f9e69097b80 1 --2- 172.21.15.16:0/401602592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d8b80 0x5645177e8d60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.445+0000 7f9e69097b80 1 -- 172.21.15.16:0/401602592 >> 172.21.15.16:0/401602592 conn(0x5645177471c0 msgr2=0x5645177e8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:26.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.445+0000 7f9e69097b80 1 -- 172.21.15.16:0/401602592 shutdown_connections 2024-04-05T23:41:26.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.445+0000 7f9e69097b80 1 -- 172.21.15.16:0/401602592 wait complete. 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.445+0000 7f9e69097b80 1 -- start start 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.445+0000 7f9e69097b80 1 Processor -- start 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d35c0 0x5645177dfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d8b80 0x5645177e0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e69097b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 0x5645177e0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e69097b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x564517844270 con 0x5645177d8b80 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e69097b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645178443e0 con 0x5645177d94a0 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e69097b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645178445a0 con 0x5645177d35c0 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e69097b80 10 client.0 _ll_register_callbacks cb 0x7f9e3c00cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:41:26.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e4e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d8b80 0x5645177e0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e4e000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d8b80 0x5645177e0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51200/0 (socket says 172.21.15.16:51200) 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e4e000700 1 -- 172.21.15.16:0/310298208 learned_addr learned my addr 172.21.15.16:0/310298208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d35c0 0x5645177dfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.446+0000 7f9e4f400700 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 0x5645177e0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e4e000700 1 -- 172.21.15.16:0/310298208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d35c0 msgr2=0x5645177dfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e4e000700 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d35c0 0x5645177dfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e4e000700 1 -- 172.21.15.16:0/310298208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 msgr2=0x5645177e0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e4e000700 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 0x5645177e0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e4e000700 1 -- 172.21.15.16:0/310298208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645175a5350 con 0x5645177d8b80 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e4e000700 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d8b80 0x5645177e0410 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00cc00 tx=0x7f9e3c099b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2255fd0bed8bb67 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e2fe00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e2fe00700 1 -- 172.21.15.16:0/310298208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c099f80 con 0x5645177d8b80 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e2fe00700 1 -- 172.21.15.16:0/310298208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f9e3c0a2090 con 0x5645177d8b80 2024-04-05T23:41:26.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e69097b80 1 -- 172.21.15.16:0/310298208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e4001e040 con 0x5645177d8b80 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e2fe00700 1 -- 172.21.15.16:0/310298208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e3c0b8250 con 0x5645177d8b80 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e69097b80 10 client.16443 fetch_fsmap learned FSMap version 57 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.447+0000 7f9e69097b80 1 -- 172.21.15.16:0/310298208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) v3 -- 0x564517551b00 con 0x5645177d8b80 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.448+0000 7f9e2fe00700 1 -- 172.21.15.16:0/310298208 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e4001e040 con 0x5645177d8b80 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.448+0000 7f9e2fe00700 1 -- 172.21.15.16:0/310298208 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 57) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f9e3c094b90 con 0x5645177d8b80 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.448+0000 7f9e69097b80 10 client.16443 fetch_fsmap finished waiting for FSMap version 57 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.448+0000 7f9e69097b80 10 client.16443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.448+0000 7f9e69097b80 20 client.16443 populate_metadata read hostname 'smithi016' 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.448+0000 7f9e69097b80 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5645178458a0 0x564517847c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.448+0000 7f9e69097b80 4 client.16443 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.448+0000 7f9e69097b80 1 -- 172.21.15.16:0/310298208 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564517572c90 con 0x5645178458a0 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.448+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5645178458a0 0x564517847c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.449+0000 7f9e4ea00700 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5645178458a0 0x564517847c90 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ced7f2144b4f173 server_cookie=f289c21a9a29762 in_seq=0 out_seq=0 2024-04-05T23:41:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.449+0000 7f9e2fe00700 10 client.16443 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e2fe00700 1 -- 172.21.15.16:0/310298208 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564517572c90 con 0x5645178458a0 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e2fe00700 10 client.16443 handle_command_reply: tid=0 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 client.16443 shutdown 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 -- 172.21.15.16:0/310298208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d8b80 msgr2=0x5645177e0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d8b80 0x5645177e0410 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00cc00 tx=0x7f9e3c099b80 comp rx=0 tx=0).stop 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 -- 172.21.15.16:0/310298208 shutdown_connections 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5645178458a0 0x564517847c90 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5645177d94a0 0x5645177e0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645177d35c0 0x5645177dfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e2fe00700 0 client.16443 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 --2- 172.21.15.16:0/310298208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5645177d8b80 0x5645177e0410 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 -- 172.21.15.16:0/310298208 >> 172.21.15.16:0/310298208 conn(0x5645177471c0 msgr2=0x5645177d9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 -- 172.21.15.16:0/310298208 shutdown_connections 2024-04-05T23:41:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 1 -- 172.21.15.16:0/310298208 wait complete. 2024-04-05T23:41:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e69097b80 20 client.16443 trim_cache size 0 max 16384 2024-04-05T23:41:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:41:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:41:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:41:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:41:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e55600700 1 -- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e340a1310 msgr2=0x7f9e340a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e55600700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e340a1310 0x7f9e340a37d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9e48003fd0 tx=0x7f9e48004040 comp rx=0 tx=0).stop 2024-04-05T23:41:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e55600700 1 -- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50106150 msgr2=0x7f9e50197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.475+0000 7f9e55600700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50106150 0x7f9e50197840 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00e970 tx=0x7f9e3c00ec80 comp rx=0 tx=0).stop 2024-04-05T23:41:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.476+0000 7f9e55600700 1 -- 172.21.15.16:0/1732471591 shutdown_connections 2024-04-05T23:41:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.476+0000 7f9e55600700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9e340a1310 0x7f9e340a37d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.476+0000 7f9e55600700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e5019bb90 0x7f9e5019c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.476+0000 7f9e55600700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e50101030 0x7f9e50197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.476+0000 7f9e55600700 1 --2- 172.21.15.16:0/1732471591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e50106150 0x7f9e50197840 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.476+0000 7f9e55600700 1 -- 172.21.15.16:0/1732471591 >> 172.21.15.16:0/1732471591 conn(0x7f9e500fc8b0 msgr2=0x7f9e500fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.476+0000 7f9e55600700 1 -- 172.21.15.16:0/1732471591 shutdown_connections 2024-04-05T23:41:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:26.476+0000 7f9e55600700 1 -- 172.21.15.16:0/1732471591 wait complete. 2024-04-05T23:41:26.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:26 smithi016 ceph-mon[38248]: pgmap v245: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 915 KiB/s wr, 61 op/s 2024-04-05T23:41:26.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:26 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:26.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:26 smithi130 ceph-mon[79417]: pgmap v245: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 915 KiB/s wr, 61 op/s 2024-04-05T23:41:26.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:26 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:26.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:26 smithi027 ceph-mon[78597]: pgmap v245: 97 pgs: 97 active+clean; 148 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 915 KiB/s wr, 61 op/s 2024-04-05T23:41:26.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:26 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:27.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 406796 5.39 MB/sec execute 286 sec latency 2534.708 ms 2024-04-05T23:41:28.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 407763 5.40 MB/sec execute 287 sec latency 495.466 ms 2024-04-05T23:41:28.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:28 smithi016 ceph-mon[38248]: pgmap v246: 97 pgs: 97 active+clean; 151 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-04-05T23:41:28.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:28 smithi130 ceph-mon[79417]: pgmap v246: 97 pgs: 97 active+clean; 151 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-04-05T23:41:28.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:28 smithi027 ceph-mon[78597]: pgmap v246: 97 pgs: 97 active+clean; 151 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-04-05T23:41:29.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 407763 5.38 MB/sec execute 288 sec latency 1495.538 ms 2024-04-05T23:41:30.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 407763 5.36 MB/sec execute 289 sec latency 2495.609 ms 2024-04-05T23:41:30.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:30 smithi016 ceph-mon[38248]: pgmap v247: 97 pgs: 97 active+clean; 151 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-05T23:41:30.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:30 smithi130 ceph-mon[79417]: pgmap v247: 97 pgs: 97 active+clean; 151 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-05T23:41:30.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:30 smithi027 ceph-mon[78597]: pgmap v247: 97 pgs: 97 active+clean; 151 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-05T23:41:31.013 INFO:tasks.workunit.client.0.smithi016.stdout: 1 407763 5.34 MB/sec execute 290 sec latency 3495.681 ms 2024-04-05T23:41:32.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 407763 5.32 MB/sec execute 291 sec latency 4495.735 ms 2024-04-05T23:41:32.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:32 smithi016 ceph-mon[38248]: pgmap v248: 97 pgs: 97 active+clean; 149 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-04-05T23:41:32.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:32 smithi130 ceph-mon[79417]: pgmap v248: 97 pgs: 97 active+clean; 149 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-04-05T23:41:32.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:32 smithi027 ceph-mon[78597]: pgmap v248: 97 pgs: 97 active+clean; 149 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-04-05T23:41:33.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 408540 5.32 MB/sec execute 292 sec latency 4698.334 ms 2024-04-05T23:41:33.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:33 smithi027 ceph-mon[78597]: pgmap v249: 97 pgs: 97 active+clean; 149 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 110 op/s 2024-04-05T23:41:34.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 409641 5.32 MB/sec execute 293 sec latency 104.023 ms 2024-04-05T23:41:34.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:33 smithi016 ceph-mon[38248]: pgmap v249: 97 pgs: 97 active+clean; 149 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 110 op/s 2024-04-05T23:41:34.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:33 smithi130 ceph-mon[79417]: pgmap v249: 97 pgs: 97 active+clean; 149 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 110 op/s 2024-04-05T23:41:35.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 409641 5.30 MB/sec execute 294 sec latency 1104.099 ms 2024-04-05T23:41:36.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 411554 5.33 MB/sec execute 295 sec latency 1190.813 ms 2024-04-05T23:41:36.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:35 smithi016 ceph-mon[38248]: pgmap v250: 97 pgs: 97 active+clean; 149 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 110 op/s 2024-04-05T23:41:36.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:35 smithi130 ceph-mon[79417]: pgmap v250: 97 pgs: 97 active+clean; 149 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 110 op/s 2024-04-05T23:41:36.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:35 smithi027 ceph-mon[78597]: pgmap v250: 97 pgs: 97 active+clean; 149 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 110 op/s 2024-04-05T23:41:37.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 412500 5.32 MB/sec execute 296 sec latency 19.903 ms 2024-04-05T23:41:38.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413183 5.31 MB/sec execute 297 sec latency 550.766 ms 2024-04-05T23:41:38.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:37 smithi016 ceph-mon[38248]: pgmap v251: 97 pgs: 97 active+clean; 152 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-04-05T23:41:38.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:37 smithi130 ceph-mon[79417]: pgmap v251: 97 pgs: 97 active+clean; 152 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-04-05T23:41:38.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:37 smithi027 ceph-mon[78597]: pgmap v251: 97 pgs: 97 active+clean; 152 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-04-05T23:41:39.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413183 5.29 MB/sec execute 298 sec latency 1550.876 ms 2024-04-05T23:41:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.637+0000 7f2364200700 1 -- 172.21.15.16:0/3234331443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c1019e0 msgr2=0x7f235c105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.637+0000 7f2364200700 1 --2- 172.21.15.16:0/3234331443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c1019e0 0x7f235c105a20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2348009990 tx=0x7f2348009ca0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.637+0000 7f2364200700 1 -- 172.21.15.16:0/3234331443 shutdown_connections 2024-04-05T23:41:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.637+0000 7f2364200700 1 --2- 172.21.15.16:0/3234331443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c106150 0x7f235c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.637+0000 7f2364200700 1 --2- 172.21.15.16:0/3234331443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235c101030 0x7f235c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.637+0000 7f2364200700 1 --2- 172.21.15.16:0/3234331443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c1019e0 0x7f235c105a20 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.637+0000 7f2364200700 1 -- 172.21.15.16:0/3234331443 >> 172.21.15.16:0/3234331443 conn(0x7f235c0fc8b0 msgr2=0x7f235c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.637+0000 7f2364200700 1 -- 172.21.15.16:0/3234331443 shutdown_connections 2024-04-05T23:41:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.637+0000 7f2364200700 1 -- 172.21.15.16:0/3234331443 wait complete. 2024-04-05T23:41:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.638+0000 7f2364200700 1 Processor -- start 2024-04-05T23:41:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.638+0000 7f2364200700 1 -- start start 2024-04-05T23:41:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2364200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c101030 0x7f235c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2364200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c106150 0x7f235c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2364200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235c074ea0 0x7f235c073ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2364200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235c1a9f60 con 0x7f235c101030 2024-04-05T23:41:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2364200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235c1aa0d0 con 0x7f235c074ea0 2024-04-05T23:41:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2364200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235c1aa290 con 0x7f235c106150 2024-04-05T23:41:39.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2361800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c101030 0x7f235c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2361800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c101030 0x7f235c0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38258/0 (socket says 172.21.15.16:38258) 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2361800700 1 -- 172.21.15.16:0/3735466019 learned_addr learned my addr 172.21.15.16:0/3735466019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2362200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235c074ea0 0x7f235c073ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2360e00700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c106150 0x7f235c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2361800700 1 -- 172.21.15.16:0/3735466019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c106150 msgr2=0x7f235c073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2361800700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c106150 0x7f235c073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2361800700 1 -- 172.21.15.16:0/3735466019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235c074ea0 msgr2=0x7f235c073ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2361800700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235c074ea0 0x7f235c073ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.639+0000 7f2361800700 1 -- 172.21.15.16:0/3735466019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23480095f0 con 0x7f235c101030 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.640+0000 7f2361800700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c101030 0x7f235c0731e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f235000e9a0 tx=0x7f235000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.640+0000 7f235a000700 1 -- 172.21.15.16:0/3735466019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2350009790 con 0x7f235c101030 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.640+0000 7f2364200700 1 -- 172.21.15.16:0/3735466019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235c1aa570 con 0x7f235c101030 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.640+0000 7f235a000700 1 -- 172.21.15.16:0/3735466019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2350010430 con 0x7f235c101030 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.640+0000 7f2364200700 1 -- 172.21.15.16:0/3735466019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235c1aaa50 con 0x7f235c101030 2024-04-05T23:41:39.633 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.640+0000 7f235a000700 1 -- 172.21.15.16:0/3735466019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2350018620 con 0x7f235c101030 2024-04-05T23:41:39.636 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.645+0000 7f235a000700 1 -- 172.21.15.16:0/3735466019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f23500105a0 con 0x7f235c101030 2024-04-05T23:41:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f2377c95b80 1 Processor -- start 2024-04-05T23:41:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f2377c95b80 1 -- start start 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f235a000700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f23440a1290 0x7f23440a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134daf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134db4a0 0x55a5134d5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134d5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f2377c95b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a51343bb80 con 0x55a5134d55c0 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f2377c95b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a51342c930 con 0x55a5134db4a0 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f2377c95b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a51342a2f0 con 0x55a5134dab80 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.646+0000 7f235a000700 1 -- 172.21.15.16:0/3735466019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f2350014070 con 0x7f235c101030 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2361800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134daf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2361800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134daf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:46048/0 (socket says 172.21.15.16:46048) 2024-04-05T23:41:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2361800700 1 -- 172.21.15.16:0/1360530521 learned_addr learned my addr 172.21.15.16:0/1360530521 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2362200700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f23440a1290 0x7f23440a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2360e00700 1 --2- 172.21.15.16:0/1360530521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134db4a0 0x55a5134d5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2362200700 1 --2- 172.21.15.16:0/1360530521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134d5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2361800700 1 -- 172.21.15.16:0/1360530521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134db4a0 msgr2=0x55a5134d5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2361800700 1 --2- 172.21.15.16:0/1360530521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134db4a0 0x55a5134d5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2361800700 1 -- 172.21.15.16:0/1360530521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 msgr2=0x55a5134d5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2361800700 1 --2- 172.21.15.16:0/1360530521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134d5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2361800700 1 -- 172.21.15.16:0/1360530521 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5132a7350 con 0x55a5134dab80 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2361800700 1 --2- 172.21.15.16:0/1360530521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134daf60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2350096470 tx=0x7f23500998d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1a2ffe1a2121544 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.647+0000 7f2362200700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f23440a1290 0x7f23440a3750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55a5134d17f0 tx=0x7f2354009920 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f234ea00700 1 -- 172.21.15.16:0/1360530521 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23500b2b20 con 0x55a5134dab80 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f234ea00700 1 -- 172.21.15.16:0/1360530521 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f235009a670 con 0x55a5134dab80 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f234ea00700 1 -- 172.21.15.16:0/1360530521 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23500b1020 con 0x55a5134dab80 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f2377c95b80 1 -- 172.21.15.16:0/1360530521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 msgr2=0x55a5134daf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1360530521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134daf60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2350096470 tx=0x7f23500998d0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f2377c95b80 1 -- 172.21.15.16:0/1360530521 shutdown_connections 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1360530521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134d5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1360530521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134daf60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1360530521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134db4a0 0x55a5134d5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.648+0000 7f2377c95b80 1 -- 172.21.15.16:0/1360530521 >> 172.21.15.16:0/1360530521 conn(0x55a5134491c0 msgr2=0x55a5134cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.649+0000 7f2377c95b80 1 -- 172.21.15.16:0/1360530521 shutdown_connections 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.649+0000 7f2377c95b80 1 -- 172.21.15.16:0/1360530521 wait complete. 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.649+0000 7f2377c95b80 1 -- start start 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.649+0000 7f2377c95b80 1 Processor -- start 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134d55c0 0x55a5134de870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 0x55a5134dedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134df2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2377c95b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a513425be0 con 0x55a5134db4a0 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2377c95b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5133c1d20 con 0x55a5134dab80 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2377c95b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5133c19c0 con 0x55a5134d55c0 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2377c95b80 10 client.0 _ll_register_callbacks cb 0x55a51322d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2362200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134df2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2361800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134d55c0 0x55a5134de870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2362200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134df2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38274/0 (socket says 172.21.15.16:38274) 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2362200700 1 -- 172.21.15.16:0/2454035839 learned_addr learned my addr 172.21.15.16:0/2454035839 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.650+0000 7f2360e00700 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 0x55a5134dedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2362200700 1 -- 172.21.15.16:0/2454035839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134d55c0 msgr2=0x55a5134de870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2362200700 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134d55c0 0x55a5134de870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2362200700 1 -- 172.21.15.16:0/2454035839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 msgr2=0x55a5134dedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2362200700 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 0x55a5134dedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2362200700 1 -- 172.21.15.16:0/2454035839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235c1aaa50 con 0x55a5134db4a0 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2360e00700 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 0x55a5134dedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2361800700 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134d55c0 0x55a5134de870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2362200700 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134df2f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2354006070 tx=0x7f2354004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbecbac6f604a07a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2343400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2343400700 1 -- 172.21.15.16:0/2454035839 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354020070 con 0x55a5134db4a0 2024-04-05T23:41:39.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2343400700 1 -- 172.21.15.16:0/2454035839 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f235401a680 con 0x55a5134db4a0 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2377c95b80 1 -- 172.21.15.16:0/2454035839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5134c81e0 con 0x55a5134db4a0 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2343400700 1 -- 172.21.15.16:0/2454035839 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f235400e070 con 0x55a5134db4a0 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2377c95b80 10 client.16455 fetch_fsmap learned FSMap version 57 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.651+0000 7f2377c95b80 1 -- 172.21.15.16:0/2454035839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) v3 -- 0x55a513253b00 con 0x55a5134db4a0 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.652+0000 7f2343400700 1 -- 172.21.15.16:0/2454035839 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a5134c81e0 con 0x55a5134db4a0 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.652+0000 7f2343400700 1 -- 172.21.15.16:0/2454035839 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 57) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f23540040b0 con 0x55a5134db4a0 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.652+0000 7f2377c95b80 10 client.16455 fetch_fsmap finished waiting for FSMap version 57 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.652+0000 7f2377c95b80 10 client.16455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.652+0000 7f2377c95b80 20 client.16455 populate_metadata read hostname 'smithi016' 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.652+0000 7f2377c95b80 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5134ef020 0x55a5134f1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.652+0000 7f2377c95b80 4 client.16455 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.652+0000 7f2377c95b80 1 -- 172.21.15.16:0/2454035839 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a51341e6a0 con 0x55a5134ef020 2024-04-05T23:41:39.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.652+0000 7f2361800700 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5134ef020 0x55a5134f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.653+0000 7f2343400700 10 client.16455 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.653+0000 7f2361800700 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5134ef020 0x55a5134f1440 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca145fbad6e37b2e server_cookie=8a15666e5d9c80a0 in_seq=0 out_seq=0 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.653+0000 7f2343400700 1 -- 172.21.15.16:0/2454035839 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a51341e6a0 con 0x55a5134ef020 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.653+0000 7f2343400700 10 client.16455 handle_command_reply: tid=0 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.653+0000 7f2377c95b80 1 client.16455 shutdown 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 -- 172.21.15.16:0/2454035839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 msgr2=0x55a5134df2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134df2f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2354006070 tx=0x7f2354004b40 comp rx=0 tx=0).stop 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 -- 172.21.15.16:0/2454035839 shutdown_connections 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a5134ef020 0x55a5134f1440 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134df2f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2343400700 0 client.16455 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134d55c0 0x55a5134de870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 --2- 172.21.15.16:0/2454035839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 0x55a5134dedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.650 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 -- 172.21.15.16:0/2454035839 >> 172.21.15.16:0/2454035839 conn(0x55a5134491c0 msgr2=0x55a5134cf280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:39.650 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 -- 172.21.15.16:0/2454035839 shutdown_connections 2024-04-05T23:41:39.650 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 1 -- 172.21.15.16:0/2454035839 wait complete. 2024-04-05T23:41:39.650 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.654+0000 7f2377c95b80 20 client.16455 trim_cache size 0 max 16384 2024-04-05T23:41:39.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.670+0000 7f2377c95b80 1 Processor -- start 2024-04-05T23:41:39.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.670+0000 7f2377c95b80 1 -- start start 2024-04-05T23:41:39.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.670+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134d55c0 0x55a5134d5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.670+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134e5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.670+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.670+0000 7f2377c95b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5132a9640 con 0x55a5134db4a0 2024-04-05T23:41:39.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.670+0000 7f2377c95b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5134ec6d0 con 0x55a5134d55c0 2024-04-05T23:41:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.670+0000 7f2377c95b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5134e6490 con 0x55a5134dab80 2024-04-05T23:41:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2362200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2362200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38284/0 (socket says 172.21.15.16:38284) 2024-04-05T23:41:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2362200700 1 -- 172.21.15.16:0/1157359955 learned_addr learned my addr 172.21.15.16:0/1157359955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2360e00700 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134e5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2361800700 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134d55c0 0x55a5134d5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2362200700 1 -- 172.21.15.16:0/1157359955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 msgr2=0x55a5134e5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2362200700 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134e5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2362200700 1 -- 172.21.15.16:0/1157359955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134d55c0 msgr2=0x55a5134d5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2362200700 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134d55c0 0x55a5134d5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2362200700 1 -- 172.21.15.16:0/1157359955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5133ebe40 con 0x55a5134db4a0 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2361800700 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134d55c0 0x55a5134d5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.671+0000 7f2362200700 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134cfa00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2354008730 tx=0x7f23540045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509cca1a88bf5472 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f234f400700 1 -- 172.21.15.16:0/1157359955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354020070 con 0x55a5134db4a0 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f234f400700 1 -- 172.21.15.16:0/1157359955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f235400f040 con 0x55a5134db4a0 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f234f400700 1 -- 172.21.15.16:0/1157359955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354003af0 con 0x55a5134db4a0 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f2377c95b80 1 -- 172.21.15.16:0/1157359955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 msgr2=0x55a5134cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134cfa00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2354008730 tx=0x7f23540045e0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f2377c95b80 1 -- 172.21.15.16:0/1157359955 shutdown_connections 2024-04-05T23:41:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134db4a0 0x55a5134cfa00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134d55c0 0x55a5134d5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1157359955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134dab80 0x55a5134e5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f2377c95b80 1 -- 172.21.15.16:0/1157359955 >> 172.21.15.16:0/1157359955 conn(0x55a5134491c0 msgr2=0x55a5134dd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f2377c95b80 1 -- 172.21.15.16:0/1157359955 shutdown_connections 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.672+0000 7f2377c95b80 1 -- 172.21.15.16:0/1157359955 wait complete. 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.673+0000 7f2377c95b80 1 -- start start 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.673+0000 7f2377c95b80 1 Processor -- start 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.673+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134e5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.673+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 0x55a5134e58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.673+0000 7f2377c95b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134db4a0 0x55a513541e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.673+0000 7f2377c95b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a513542370 con 0x55a5134d55c0 2024-04-05T23:41:39.667 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.673+0000 7f2377c95b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5135424b0 con 0x55a5134dab80 2024-04-05T23:41:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.673+0000 7f2377c95b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a513542670 con 0x55a5134db4a0 2024-04-05T23:41:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.673+0000 7f2377c95b80 10 client.0 _ll_register_callbacks cb 0x7f2354008d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:41:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2361800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134e5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2361800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134e5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38294/0 (socket says 172.21.15.16:38294) 2024-04-05T23:41:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2361800700 1 -- 172.21.15.16:0/1869459620 learned_addr learned my addr 172.21.15.16:0/1869459620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2360e00700 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 0x55a5134e58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2362200700 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134db4a0 0x55a513541e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2361800700 1 -- 172.21.15.16:0/1869459620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134db4a0 msgr2=0x55a513541e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2361800700 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134db4a0 0x55a513541e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2361800700 1 -- 172.21.15.16:0/1869459620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 msgr2=0x55a5134e58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2361800700 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 0x55a5134e58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2361800700 1 -- 172.21.15.16:0/1869459620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a513253b00 con 0x55a5134d55c0 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2362200700 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134db4a0 0x55a513541e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2361800700 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134e5370 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f23500ca660 tx=0x7f2350099d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1afac7e420f5b85 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.674+0000 7f2343400700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2343400700 1 -- 172.21.15.16:0/1869459620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23500998d0 con 0x55a5134d55c0 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2343400700 1 -- 172.21.15.16:0/1869459620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f23500ade40 con 0x55a5134d55c0 2024-04-05T23:41:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2377c95b80 1 -- 172.21.15.16:0/1869459620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f235009a670 con 0x55a5134d55c0 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2343400700 1 -- 172.21.15.16:0/1869459620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f235009dda0 con 0x55a5134d55c0 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2377c95b80 10 client.16461 fetch_fsmap learned FSMap version 57 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2377c95b80 1 -- 172.21.15.16:0/1869459620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) v3 -- 0x55a5132abc70 con 0x55a5134d55c0 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2343400700 1 -- 172.21.15.16:0/1869459620 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f235009a670 con 0x55a5134d55c0 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2343400700 1 -- 172.21.15.16:0/1869459620 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 57) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f23500b4070 con 0x55a5134d55c0 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2377c95b80 10 client.16461 fetch_fsmap finished waiting for FSMap version 57 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2377c95b80 10 client.16461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2377c95b80 20 client.16461 populate_metadata read hostname 'smithi016' 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a513543b10 0x55a513545f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2377c95b80 4 client.16461 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.675+0000 7f2377c95b80 1 -- 172.21.15.16:0/1869459620 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a513274c90 con 0x55a513543b10 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.676+0000 7f2360e00700 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a513543b10 0x55a513545f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.676+0000 7f2360e00700 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a513543b10 0x55a513545f30 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18bdbe6bbf4aaa40 server_cookie=d0cca081114bcef4 in_seq=0 out_seq=0 2024-04-05T23:41:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.676+0000 7f2343400700 10 client.16461 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:39.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.715+0000 7f2343400700 1 -- 172.21.15.16:0/1869459620 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a513274c90 con 0x55a513543b10 2024-04-05T23:41:39.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.715+0000 7f2343400700 10 client.16461 handle_command_reply: tid=0 2024-04-05T23:41:39.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 client.16461 shutdown 2024-04-05T23:41:39.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 -- 172.21.15.16:0/1869459620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 msgr2=0x55a5134e5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:39.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134e5370 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f23500ca660 tx=0x7f2350099d40 comp rx=0 tx=0).stop 2024-04-05T23:41:39.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 -- 172.21.15.16:0/1869459620 shutdown_connections 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a513543b10 0x55a513545f30 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5134d55c0 0x55a5134e5370 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5134db4a0 0x55a513541e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2343400700 0 client.16461 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 --2- 172.21.15.16:0/1869459620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a5134dab80 0x55a5134e58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 -- 172.21.15.16:0/1869459620 >> 172.21.15.16:0/1869459620 conn(0x55a5134491c0 msgr2=0x55a5134db880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 -- 172.21.15.16:0/1869459620 shutdown_connections 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 1 -- 172.21.15.16:0/1869459620 wait complete. 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.716+0000 7f2377c95b80 20 client.16461 trim_cache size 0 max 16384 2024-04-05T23:41:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:41:39.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:41:39.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:41:39.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:41:39.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 -- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f23440a1290 msgr2=0x7f23440a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:39.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f23440a1290 0x7f23440a3750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55a5134d17f0 tx=0x7f2354009920 comp rx=0 tx=0).stop 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 -- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c101030 msgr2=0x7f235c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c101030 0x7f235c0731e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f235000e9a0 tx=0x7f235000ecb0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 -- 172.21.15.16:0/3735466019 shutdown_connections 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f23440a1290 0x7f23440a3750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235c101030 0x7f235c0731e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235c074ea0 0x7f235c073ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 --2- 172.21.15.16:0/3735466019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c106150 0x7f235c073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 -- 172.21.15.16:0/3735466019 >> 172.21.15.16:0/3735466019 conn(0x7f235c0fc8b0 msgr2=0x7f235c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 -- 172.21.15.16:0/3735466019 shutdown_connections 2024-04-05T23:41:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:39.717+0000 7f2364200700 1 -- 172.21.15.16:0/3735466019 wait complete. 2024-04-05T23:41:40.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413183 5.28 MB/sec execute 299 sec latency 2550.993 ms 2024-04-05T23:41:40.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:40 smithi016 ceph-mon[38248]: pgmap v252: 97 pgs: 97 active+clean; 152 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 736 KiB/s wr, 90 op/s 2024-04-05T23:41:40.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:40 smithi130 ceph-mon[79417]: pgmap v252: 97 pgs: 97 active+clean; 152 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 736 KiB/s wr, 90 op/s 2024-04-05T23:41:40.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:40 smithi027 ceph-mon[78597]: pgmap v252: 97 pgs: 97 active+clean; 152 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 736 KiB/s wr, 90 op/s 2024-04-05T23:41:41.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413366 5.26 MB/sec execute 300 sec latency 3428.919 ms 2024-04-05T23:41:42.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 415278 5.29 MB/sec execute 301 sec latency 14.636 ms 2024-04-05T23:41:42.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:42 smithi016 ceph-mon[38248]: pgmap v253: 97 pgs: 97 active+clean; 150 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 141 op/s 2024-04-05T23:41:42.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:42 smithi130 ceph-mon[79417]: pgmap v253: 97 pgs: 97 active+clean; 150 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 141 op/s 2024-04-05T23:41:42.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:42 smithi027 ceph-mon[78597]: pgmap v253: 97 pgs: 97 active+clean; 150 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 141 op/s 2024-04-05T23:41:43.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416475 5.29 MB/sec execute 302 sec latency 25.844 ms 2024-04-05T23:41:43.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:43 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:43.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:43 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:43.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:43 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:44.014 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416732 5.27 MB/sec execute 303 sec latency 811.419 ms 2024-04-05T23:41:44.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:44 smithi016 ceph-mon[38248]: pgmap v254: 97 pgs: 97 active+clean; 150 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 887 KiB/s wr, 107 op/s 2024-04-05T23:41:44.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:44 smithi130 ceph-mon[79417]: pgmap v254: 97 pgs: 97 active+clean; 150 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 887 KiB/s wr, 107 op/s 2024-04-05T23:41:44.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:44 smithi027 ceph-mon[78597]: pgmap v254: 97 pgs: 97 active+clean; 150 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 887 KiB/s wr, 107 op/s 2024-04-05T23:41:45.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416732 5.25 MB/sec execute 304 sec latency 1811.572 ms 2024-04-05T23:41:46.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416732 5.24 MB/sec execute 305 sec latency 2811.712 ms 2024-04-05T23:41:46.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:46 smithi016 ceph-mon[38248]: pgmap v255: 97 pgs: 97 active+clean; 150 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 887 KiB/s wr, 107 op/s 2024-04-05T23:41:46.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:46 smithi130 ceph-mon[79417]: pgmap v255: 97 pgs: 97 active+clean; 150 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 887 KiB/s wr, 107 op/s 2024-04-05T23:41:46.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:46 smithi027 ceph-mon[78597]: pgmap v255: 97 pgs: 97 active+clean; 150 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 887 KiB/s wr, 107 op/s 2024-04-05T23:41:47.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 417436 5.22 MB/sec execute 306 sec latency 3529.717 ms 2024-04-05T23:41:48.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 418962 5.25 MB/sec execute 307 sec latency 48.257 ms 2024-04-05T23:41:48.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:48 smithi016 ceph-mon[38248]: pgmap v256: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-05T23:41:48.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:48 smithi130 ceph-mon[79417]: pgmap v256: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-05T23:41:48.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:48 smithi027 ceph-mon[78597]: pgmap v256: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-05T23:41:49.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420177 5.25 MB/sec execute 308 sec latency 44.584 ms 2024-04-05T23:41:49.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:49 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:41:49.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:49 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:41:49.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:49 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:41:49.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:49 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:41:49.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:49 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:41:49.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:49 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:41:49.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:49 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:41:49.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:49 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:41:49.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:49 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:41:49.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:49 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:41:49.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:49 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:41:49.966 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:49 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:41:50.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420275 5.23 MB/sec execute 309 sec latency 922.725 ms 2024-04-05T23:41:50.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:50 smithi016 ceph-mon[38248]: pgmap v257: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 914 KiB/s wr, 110 op/s 2024-04-05T23:41:50.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:50 smithi130 ceph-mon[79417]: pgmap v257: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 914 KiB/s wr, 110 op/s 2024-04-05T23:41:50.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:50 smithi027 ceph-mon[78597]: pgmap v257: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 914 KiB/s wr, 110 op/s 2024-04-05T23:41:51.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420275 5.21 MB/sec execute 310 sec latency 1922.886 ms 2024-04-05T23:41:52.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420973 5.20 MB/sec execute 311 sec latency 2642.995 ms 2024-04-05T23:41:52.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:52 smithi016 ceph-mon[38248]: pgmap v258: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 162 op/s 2024-04-05T23:41:52.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:52 smithi130 ceph-mon[79417]: pgmap v258: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 162 op/s 2024-04-05T23:41:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.882+0000 7fce2d600700 1 -- 172.21.15.16:0/981588296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce281019e0 msgr2=0x7fce28105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.882+0000 7fce2d600700 1 --2- 172.21.15.16:0/981588296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce281019e0 0x7fce28105a20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fce14009990 tx=0x7fce14009ca0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.883+0000 7fce2d600700 1 -- 172.21.15.16:0/981588296 shutdown_connections 2024-04-05T23:41:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.883+0000 7fce2d600700 1 --2- 172.21.15.16:0/981588296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce28101030 0x7fce28101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.883+0000 7fce2d600700 1 --2- 172.21.15.16:0/981588296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce28106150 0x7fce2810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.883+0000 7fce2d600700 1 --2- 172.21.15.16:0/981588296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce281019e0 0x7fce28105a20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.883+0000 7fce2d600700 1 -- 172.21.15.16:0/981588296 >> 172.21.15.16:0/981588296 conn(0x7fce280fc8b0 msgr2=0x7fce280fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.883+0000 7fce2d600700 1 -- 172.21.15.16:0/981588296 shutdown_connections 2024-04-05T23:41:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.883+0000 7fce2d600700 1 -- 172.21.15.16:0/981588296 wait complete. 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.884+0000 7fce2d600700 1 Processor -- start 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.884+0000 7fce2d600700 1 -- start start 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.884+0000 7fce2d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce28101030 0x7fce28199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.884+0000 7fce2d600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce28106150 0x7fce28199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.884+0000 7fce2d600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce2819dce0 0x7fce2819e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.884+0000 7fce2d600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2819e760 con 0x7fce28106150 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.884+0000 7fce2d600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2819e8d0 con 0x7fce28101030 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.884+0000 7fce2d600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2819ea90 con 0x7fce2819dce0 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.884+0000 7fce26a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce28101030 0x7fce28199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce26a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce28101030 0x7fce28199400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33956/0 (socket says 172.21.15.16:33956) 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce26a00700 1 -- 172.21.15.16:0/2947886005 learned_addr learned my addr 172.21.15.16:0/2947886005 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce27400700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce2819dce0 0x7fce2819e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce26000700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce28106150 0x7fce28199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce26a00700 1 -- 172.21.15.16:0/2947886005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce2819dce0 msgr2=0x7fce2819e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce26a00700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce2819dce0 0x7fce2819e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce26a00700 1 -- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce28106150 msgr2=0x7fce28199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce26a00700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce28106150 0x7fce28199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce26a00700 1 -- 172.21.15.16:0/2947886005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce140095f0 con 0x7fce28101030 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce27400700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce2819dce0 0x7fce2819e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce26a00700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce28101030 0x7fce28199400 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fce1800d980 tx=0x7fce1800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.885+0000 7fce1f400700 1 -- 172.21.15.16:0/2947886005 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce18004220 con 0x7fce28101030 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.886+0000 7fce2d600700 1 -- 172.21.15.16:0/2947886005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce281a5ff0 con 0x7fce28101030 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.886+0000 7fce1f400700 1 -- 172.21.15.16:0/2947886005 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce18007640 con 0x7fce28101030 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.886+0000 7fce1f400700 1 -- 172.21.15.16:0/2947886005 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce18005690 con 0x7fce28101030 2024-04-05T23:41:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.886+0000 7fce2d600700 1 -- 172.21.15.16:0/2947886005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce281a6420 con 0x7fce28101030 2024-04-05T23:41:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.890+0000 7fce1f400700 1 -- 172.21.15.16:0/2947886005 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fce18004750 con 0x7fce28101030 2024-04-05T23:41:52.882 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.891+0000 7fce1f400700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fce0c0a1310 0x7fce0c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.882 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.891+0000 7fce1f400700 1 -- 172.21.15.16:0/2947886005 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fce180cc990 con 0x7fce28101030 2024-04-05T23:41:52.882 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.891+0000 7fce26000700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fce0c0a1310 0x7fce0c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.892+0000 7fce41110b80 1 Processor -- start 2024-04-05T23:41:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.892+0000 7fce26000700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fce0c0a1310 0x7fce0c0a37d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fce1400b2e0 tx=0x7fce14005960 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce41110b80 1 -- start start 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce41110b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f07f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce41110b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066f02030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce41110b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f025c0 0x55b066f02a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce41110b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066e68b80 con 0x55b066f084a0 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce41110b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066e59930 con 0x55b066f025c0 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce41110b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066e572f0 con 0x55b066f07b80 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce26a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f07f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce26a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f07f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42054/0 (socket says 172.21.15.16:42054) 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce26a00700 1 -- 172.21.15.16:0/1299407922 learned_addr learned my addr 172.21.15.16:0/1299407922 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce26000700 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066f02030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.893+0000 7fce27400700 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f025c0 0x55b066f02a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce26a00700 1 -- 172.21.15.16:0/1299407922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f025c0 msgr2=0x55b066f02a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce26a00700 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f025c0 0x55b066f02a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce26a00700 1 -- 172.21.15.16:0/1299407922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 msgr2=0x55b066f02030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce26a00700 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066f02030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce26a00700 1 -- 172.21.15.16:0/1299407922 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b066cd4350 con 0x55b066f07b80 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce27400700 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f025c0 0x55b066f02a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce26000700 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066f02030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce26a00700 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f07f60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fce18099a10 tx=0x7fce1809a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2f64f8ba5eb1bc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce13400700 1 -- 172.21.15.16:0/1299407922 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce180a1f00 con 0x55b066f07b80 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce13400700 1 -- 172.21.15.16:0/1299407922 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce18004380 con 0x55b066f07b80 2024-04-05T23:41:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.894+0000 7fce13400700 1 -- 172.21.15.16:0/1299407922 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce180a1a70 con 0x55b066f07b80 2024-04-05T23:41:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.895+0000 7fce41110b80 1 -- 172.21.15.16:0/1299407922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 msgr2=0x55b066f07f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.895+0000 7fce41110b80 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f07f60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fce18099a10 tx=0x7fce1809a250 comp rx=0 tx=0).stop 2024-04-05T23:41:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.895+0000 7fce41110b80 1 -- 172.21.15.16:0/1299407922 shutdown_connections 2024-04-05T23:41:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.895+0000 7fce41110b80 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f025c0 0x55b066f02a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.895+0000 7fce41110b80 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f07f60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.895+0000 7fce41110b80 1 --2- 172.21.15.16:0/1299407922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066f02030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.895+0000 7fce41110b80 1 -- 172.21.15.16:0/1299407922 >> 172.21.15.16:0/1299407922 conn(0x55b066e761c0 msgr2=0x55b066efcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.895+0000 7fce41110b80 1 -- 172.21.15.16:0/1299407922 shutdown_connections 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.895+0000 7fce41110b80 1 -- 172.21.15.16:0/1299407922 wait complete. 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.896+0000 7fce41110b80 1 -- start start 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.896+0000 7fce41110b80 1 Processor -- start 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.896+0000 7fce41110b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f025c0 0x55b066f0e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.896+0000 7fce41110b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f0b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.896+0000 7fce41110b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f084a0 0x55b066f0bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.896+0000 7fce41110b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066e52be0 con 0x55b066f025c0 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.896+0000 7fce41110b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066deed20 con 0x55b066f084a0 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.896+0000 7fce41110b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066dee9c0 con 0x55b066f07b80 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.896+0000 7fce41110b80 10 client.0 _ll_register_callbacks cb 0x55b066c5a340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce27400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f084a0 0x55b066f0bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce27400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f084a0 0x55b066f0bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33976/0 (socket says 172.21.15.16:33976) 2024-04-05T23:41:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce27400700 1 -- 172.21.15.16:0/2167960736 learned_addr learned my addr 172.21.15.16:0/2167960736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce26000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f0b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce26a00700 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f025c0 0x55b066f0e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce27400700 1 -- 172.21.15.16:0/2167960736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 msgr2=0x55b066f0b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce27400700 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f0b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce27400700 1 -- 172.21.15.16:0/2167960736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f025c0 msgr2=0x55b066f0e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce27400700 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f025c0 0x55b066f0e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce27400700 1 -- 172.21.15.16:0/2167960736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce281a6420 con 0x55b066f084a0 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce26000700 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f0b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce27400700 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f084a0 0x55b066f0bf70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fce200042c0 tx=0x7fce200042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db8e16492c7d7f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.897+0000 7fce07e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce07e00700 1 -- 172.21.15.16:0/2167960736 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce20007500 con 0x55b066f084a0 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce07e00700 1 -- 172.21.15.16:0/2167960736 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fce20007660 con 0x55b066f084a0 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce41110b80 1 -- 172.21.15.16:0/2167960736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b066ef51e0 con 0x55b066f084a0 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce07e00700 1 -- 172.21.15.16:0/2167960736 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce20007820 con 0x55b066f084a0 2024-04-05T23:41:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce41110b80 10 client.26522 fetch_fsmap learned FSMap version 58 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce41110b80 1 -- 172.21.15.16:0/2167960736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) v3 -- 0x55b066c80b00 con 0x55b066f084a0 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce07e00700 1 -- 172.21.15.16:0/2167960736 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b066ef51e0 con 0x55b066f084a0 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce07e00700 1 -- 172.21.15.16:0/2167960736 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 58) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fce2000eb40 con 0x55b066f084a0 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce41110b80 10 client.26522 fetch_fsmap finished waiting for FSMap version 58 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce41110b80 10 client.26522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce41110b80 20 client.26522 populate_metadata read hostname 'smithi016' 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce41110b80 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b066f1bef0 0x55b066f1e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce41110b80 4 client.26522 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.898+0000 7fce41110b80 1 -- 172.21.15.16:0/2167960736 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b066e4b6a0 con 0x55b066f1bef0 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.899+0000 7fce26a00700 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b066f1bef0 0x55b066f1e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.899+0000 7fce07e00700 10 client.26522 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:52.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.899+0000 7fce26a00700 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b066f1bef0 0x55b066f1e310 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b47ac284fd13d50 server_cookie=56dd307b4bc095e6 in_seq=0 out_seq=0 2024-04-05T23:41:52.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce07e00700 1 -- 172.21.15.16:0/2167960736 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b066e4b6a0 con 0x55b066f1bef0 2024-04-05T23:41:52.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce07e00700 10 client.26522 handle_command_reply: tid=0 2024-04-05T23:41:52.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 client.26522 shutdown 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 -- 172.21.15.16:0/2167960736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f084a0 msgr2=0x55b066f0bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f084a0 0x55b066f0bf70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fce200042c0 tx=0x7fce200042f0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 -- 172.21.15.16:0/2167960736 shutdown_connections 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b066f1bef0 0x55b066f1e310 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f025c0 0x55b066f0e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f084a0 0x55b066f0bf70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 --2- 172.21.15.16:0/2167960736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f07b80 0x55b066f0b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce07e00700 0 client.26522 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 -- 172.21.15.16:0/2167960736 >> 172.21.15.16:0/2167960736 conn(0x55b066e761c0 msgr2=0x55b066efc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 -- 172.21.15.16:0/2167960736 shutdown_connections 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.900+0000 7fce41110b80 1 -- 172.21.15.16:0/2167960736 wait complete. 2024-04-05T23:41:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.901+0000 7fce41110b80 20 client.26522 trim_cache size 0 max 16384 2024-04-05T23:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.918+0000 7fce41110b80 1 Processor -- start 2024-04-05T23:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.918+0000 7fce41110b80 1 -- start start 2024-04-05T23:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.918+0000 7fce41110b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066ef8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.918+0000 7fce41110b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 0x55b066f029a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.918+0000 7fce41110b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066efca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.918+0000 7fce41110b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066cd6640 con 0x55b066f084a0 2024-04-05T23:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.918+0000 7fce41110b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066f02410 con 0x55b066f07b80 2024-04-05T23:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.918+0000 7fce41110b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066efd900 con 0x55b066f025c0 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066ef8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066ef8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42066/0 (socket says 172.21.15.16:42066) 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26a00700 1 -- 172.21.15.16:0/257274621 learned_addr learned my addr 172.21.15.16:0/257274621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26000700 1 --2- 172.21.15.16:0/257274621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 0x55b066f029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce27400700 1 --2- 172.21.15.16:0/257274621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066efca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26a00700 1 -- 172.21.15.16:0/257274621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 msgr2=0x55b066f029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26a00700 1 --2- 172.21.15.16:0/257274621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 0x55b066f029a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26a00700 1 -- 172.21.15.16:0/257274621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 msgr2=0x55b066efca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26a00700 1 --2- 172.21.15.16:0/257274621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066efca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26a00700 1 -- 172.21.15.16:0/257274621 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b066e18e40 con 0x55b066f025c0 2024-04-05T23:41:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce26a00700 1 --2- 172.21.15.16:0/257274621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066ef8ef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fce18099780 tx=0x7fce180050b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f88db1bc5aeca714 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce13e00700 1 -- 172.21.15.16:0/257274621 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1807c070 con 0x55b066f025c0 2024-04-05T23:41:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce13e00700 1 -- 172.21.15.16:0/257274621 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce18097e20 con 0x55b066f025c0 2024-04-05T23:41:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.919+0000 7fce13e00700 1 -- 172.21.15.16:0/257274621 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce18098f70 con 0x55b066f025c0 2024-04-05T23:41:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.920+0000 7fce41110b80 1 -- 172.21.15.16:0/257274621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 msgr2=0x55b066ef8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.920+0000 7fce41110b80 1 --2- 172.21.15.16:0/257274621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066ef8ef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fce18099780 tx=0x7fce180050b0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.920+0000 7fce41110b80 1 -- 172.21.15.16:0/257274621 shutdown_connections 2024-04-05T23:41:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.920+0000 7fce41110b80 1 --2- 172.21.15.16:0/257274621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066efca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.920+0000 7fce41110b80 1 --2- 172.21.15.16:0/257274621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066ef8ef0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.920+0000 7fce41110b80 1 --2- 172.21.15.16:0/257274621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 0x55b066f029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.920+0000 7fce41110b80 1 -- 172.21.15.16:0/257274621 >> 172.21.15.16:0/257274621 conn(0x55b066e761c0 msgr2=0x55b066f22c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.920+0000 7fce41110b80 1 -- 172.21.15.16:0/257274621 shutdown_connections 2024-04-05T23:41:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.920+0000 7fce41110b80 1 -- 172.21.15.16:0/257274621 wait complete. 2024-04-05T23:41:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.921+0000 7fce41110b80 1 -- start start 2024-04-05T23:41:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.921+0000 7fce41110b80 1 Processor -- start 2024-04-05T23:41:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce41110b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066f12dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce41110b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 0x55b066f036e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce41110b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066f03c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce41110b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066f04160 con 0x55b066f084a0 2024-04-05T23:41:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce41110b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066f042d0 con 0x55b066f07b80 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce41110b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b066f04490 con 0x55b066f025c0 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce41110b80 10 client.0 _ll_register_callbacks cb 0x7fce20003a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce26a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066f12dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce26a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066f12dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42072/0 (socket says 172.21.15.16:42072) 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce26000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 0x55b066f036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce27400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066f03c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce26a00700 1 -- 172.21.15.16:0/724253550 learned_addr learned my addr 172.21.15.16:0/724253550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce26a00700 1 -- 172.21.15.16:0/724253550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 msgr2=0x55b066f036e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce26a00700 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 0x55b066f036e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce26a00700 1 -- 172.21.15.16:0/724253550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 msgr2=0x55b066f03c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:41:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce26a00700 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066f03c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.922+0000 7fce26a00700 1 -- 172.21.15.16:0/724253550 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b066cd4350 con 0x55b066f025c0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.923+0000 7fce26a00700 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066f12dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fce180999c0 tx=0x7fce1809a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=710a61e97d1c8631 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.923+0000 7fce07e00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.923+0000 7fce07e00700 1 -- 172.21.15.16:0/724253550 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1807c070 con 0x55b066f025c0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.923+0000 7fce07e00700 1 -- 172.21.15.16:0/724253550 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fce18098f70 con 0x55b066f025c0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.923+0000 7fce41110b80 1 -- 172.21.15.16:0/724253550 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce18004380 con 0x55b066f025c0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.923+0000 7fce07e00700 1 -- 172.21.15.16:0/724253550 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce180b2440 con 0x55b066f025c0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.923+0000 7fce41110b80 10 client.26557 fetch_fsmap learned FSMap version 58 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.923+0000 7fce41110b80 1 -- 172.21.15.16:0/724253550 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) v3 -- 0x55b066cd8c70 con 0x55b066f025c0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.923+0000 7fce07e00700 1 -- 172.21.15.16:0/724253550 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce18004380 con 0x55b066f025c0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.924+0000 7fce07e00700 1 -- 172.21.15.16:0/724253550 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 58) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fce18078b60 con 0x55b066f025c0 2024-04-05T23:41:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.924+0000 7fce41110b80 10 client.26557 fetch_fsmap finished waiting for FSMap version 58 2024-04-05T23:41:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.924+0000 7fce41110b80 10 client.26557 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:41:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.924+0000 7fce41110b80 20 client.26557 populate_metadata read hostname 'smithi016' 2024-04-05T23:41:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.924+0000 7fce41110b80 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b066f6fe20 0x55b066f72210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:41:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.924+0000 7fce41110b80 4 client.26557 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:41:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.924+0000 7fce41110b80 1 -- 172.21.15.16:0/724253550 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b066ca1c90 con 0x55b066f6fe20 2024-04-05T23:41:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.924+0000 7fce26000700 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b066f6fe20 0x55b066f72210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:41:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.925+0000 7fce07e00700 10 client.26557 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.925+0000 7fce26000700 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b066f6fe20 0x55b066f72210 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98e5430583939877 server_cookie=796b764496970217 in_seq=0 out_seq=0 2024-04-05T23:41:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce07e00700 1 -- 172.21.15.16:0/724253550 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b066ca1c90 con 0x55b066f6fe20 2024-04-05T23:41:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce07e00700 10 client.26557 handle_command_reply: tid=0 2024-04-05T23:41:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce41110b80 1 client.26557 shutdown 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce41110b80 1 -- 172.21.15.16:0/724253550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 msgr2=0x55b066f12dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce41110b80 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066f12dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fce180999c0 tx=0x7fce1809a180 comp rx=0 tx=0).stop 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce41110b80 1 -- 172.21.15.16:0/724253550 shutdown_connections 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce41110b80 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b066f6fe20 0x55b066f72210 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce41110b80 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b066f084a0 0x55b066f03c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce41110b80 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b066f025c0 0x55b066f12dd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce07e00700 0 client.26557 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce41110b80 1 --2- 172.21.15.16:0/724253550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b066f07b80 0x55b066f036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.951+0000 7fce41110b80 1 -- 172.21.15.16:0/724253550 >> 172.21.15.16:0/724253550 conn(0x55b066e761c0 msgr2=0x55b066f029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.952+0000 7fce41110b80 1 -- 172.21.15.16:0/724253550 shutdown_connections 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.952+0000 7fce41110b80 1 -- 172.21.15.16:0/724253550 wait complete. 2024-04-05T23:41:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.952+0000 7fce41110b80 20 client.26557 trim_cache size 0 max 16384 2024-04-05T23:41:52.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:41:52.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:41:52.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:41:52.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:41:52.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 -- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fce0c0a1310 msgr2=0x7fce0c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fce0c0a1310 0x7fce0c0a37d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fce1400b2e0 tx=0x7fce14005960 comp rx=0 tx=0).stop 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 -- 172.21.15.16:0/2947886005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce28101030 msgr2=0x7fce28199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce28101030 0x7fce28199400 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fce1800d980 tx=0x7fce1800dc90 comp rx=0 tx=0).stop 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 -- 172.21.15.16:0/2947886005 shutdown_connections 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fce0c0a1310 0x7fce0c0a37d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce2819dce0 0x7fce2819e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce28101030 0x7fce28199400 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 --2- 172.21.15.16:0/2947886005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce28106150 0x7fce28199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 -- 172.21.15.16:0/2947886005 >> 172.21.15.16:0/2947886005 conn(0x7fce280fc8b0 msgr2=0x7fce280febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 -- 172.21.15.16:0/2947886005 shutdown_connections 2024-04-05T23:41:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:41:52.953+0000 7fce2d600700 1 -- 172.21.15.16:0/2947886005 wait complete. 2024-04-05T23:41:52.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:52 smithi027 ceph-mon[78597]: pgmap v258: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 162 op/s 2024-04-05T23:41:53.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 422515 5.22 MB/sec execute 312 sec latency 12.897 ms 2024-04-05T23:41:54.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 423824 5.22 MB/sec execute 313 sec latency 27.479 ms 2024-04-05T23:41:54.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:54 smithi016 ceph-mon[38248]: pgmap v259: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-04-05T23:41:54.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:54 smithi130 ceph-mon[79417]: pgmap v259: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-04-05T23:41:54.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:54 smithi027 ceph-mon[78597]: pgmap v259: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-04-05T23:41:55.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 423824 5.21 MB/sec execute 314 sec latency 1027.581 ms 2024-04-05T23:41:56.015 INFO:tasks.workunit.client.0.smithi016.stdout: 1 425782 5.23 MB/sec execute 315 sec latency 1112.860 ms 2024-04-05T23:41:56.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:56 smithi016 ceph-mon[38248]: pgmap v260: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-04-05T23:41:56.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:56 smithi130 ceph-mon[79417]: pgmap v260: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-04-05T23:41:56.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:56 smithi027 ceph-mon[78597]: pgmap v260: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 111 op/s 2024-04-05T23:41:57.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 426905 5.23 MB/sec execute 316 sec latency 28.504 ms 2024-04-05T23:41:58.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427365 5.22 MB/sec execute 317 sec latency 694.889 ms 2024-04-05T23:41:58.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:58 smithi016 ceph-mon[38248]: pgmap v261: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 178 op/s 2024-04-05T23:41:58.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:41:58 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:58.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:58 smithi130 ceph-mon[79417]: pgmap v261: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 178 op/s 2024-04-05T23:41:58.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:41:58 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:58.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:58 smithi027 ceph-mon[78597]: pgmap v261: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 178 op/s 2024-04-05T23:41:58.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:41:58 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:41:59.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427365 5.20 MB/sec execute 318 sec latency 1695.020 ms 2024-04-05T23:42:00.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427365 5.19 MB/sec execute 319 sec latency 2695.152 ms 2024-04-05T23:42:00.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:00 smithi016 ceph-mon[38248]: pgmap v262: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-04-05T23:42:00.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:00 smithi130 ceph-mon[79417]: pgmap v262: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-04-05T23:42:00.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:00 smithi027 ceph-mon[78597]: pgmap v262: 97 pgs: 97 active+clean; 156 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-04-05T23:42:01.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427499 5.17 MB/sec execute 320 sec latency 3573.930 ms 2024-04-05T23:42:02.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 428935 5.18 MB/sec execute 321 sec latency 33.938 ms 2024-04-05T23:42:02.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:02 smithi016 ceph-mon[38248]: pgmap v263: 97 pgs: 97 active+clean; 158 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-05T23:42:02.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:02 smithi130 ceph-mon[79417]: pgmap v263: 97 pgs: 97 active+clean; 158 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-05T23:42:02.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:02 smithi027 ceph-mon[78597]: pgmap v263: 97 pgs: 97 active+clean; 158 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-05T23:42:03.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430049 5.18 MB/sec execute 322 sec latency 15.455 ms 2024-04-05T23:42:03.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:03 smithi027 ceph-mon[78597]: pgmap v264: 97 pgs: 97 active+clean; 158 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 988 KiB/s wr, 118 op/s 2024-04-05T23:42:03.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:03 smithi027 ceph-mon[78597]: osdmap e90: 12 total, 12 up, 12 in 2024-04-05T23:42:04.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430914 5.18 MB/sec execute 323 sec latency 486.719 ms 2024-04-05T23:42:04.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:03 smithi016 ceph-mon[38248]: pgmap v264: 97 pgs: 97 active+clean; 158 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 988 KiB/s wr, 118 op/s 2024-04-05T23:42:04.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:03 smithi016 ceph-mon[38248]: osdmap e90: 12 total, 12 up, 12 in 2024-04-05T23:42:04.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:03 smithi130 ceph-mon[79417]: pgmap v264: 97 pgs: 97 active+clean; 158 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 988 KiB/s wr, 118 op/s 2024-04-05T23:42:04.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:03 smithi130 ceph-mon[79417]: osdmap e90: 12 total, 12 up, 12 in 2024-04-05T23:42:05.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430914 5.17 MB/sec execute 324 sec latency 1486.802 ms 2024-04-05T23:42:06.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430914 5.15 MB/sec execute 325 sec latency 2486.896 ms 2024-04-05T23:42:06.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:05 smithi016 ceph-mon[38248]: pgmap v266: 97 pgs: 97 active+clean; 158 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-04-05T23:42:06.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:05 smithi130 ceph-mon[79417]: pgmap v266: 97 pgs: 97 active+clean; 158 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-04-05T23:42:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.114+0000 7f7398000700 1 -- 172.21.15.16:0/3537153906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390106150 msgr2=0x7f739010dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.114+0000 7f7398000700 1 --2- 172.21.15.16:0/3537153906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390106150 0x7f739010dcd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f738800b0a0 tx=0x7f738800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.114+0000 7f7398000700 1 -- 172.21.15.16:0/3537153906 shutdown_connections 2024-04-05T23:42:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.114+0000 7f7398000700 1 --2- 172.21.15.16:0/3537153906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390106150 0x7f739010dcd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.114+0000 7f7398000700 1 --2- 172.21.15.16:0/3537153906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7390101030 0x7f7390101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.114+0000 7f7398000700 1 --2- 172.21.15.16:0/3537153906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73901019e0 0x7f7390105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.114+0000 7f7398000700 1 -- 172.21.15.16:0/3537153906 >> 172.21.15.16:0/3537153906 conn(0x7f73900fc8b0 msgr2=0x7f73900fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.114+0000 7f7398000700 1 -- 172.21.15.16:0/3537153906 shutdown_connections 2024-04-05T23:42:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.114+0000 7f7398000700 1 -- 172.21.15.16:0/3537153906 wait complete. 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.115+0000 7f7398000700 1 Processor -- start 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.115+0000 7f7398000700 1 -- start start 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7398000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390101030 0x7f7390199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7398000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7390106150 0x7f7390199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7398000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f739019df00 0x7f739019e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7398000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739019e980 con 0x7f7390101030 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7398000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739019eaf0 con 0x7f7390106150 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7398000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739019ecb0 con 0x7f739019df00 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7395600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390101030 0x7f7390199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7395600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390101030 0x7f7390199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41354/0 (socket says 172.21.15.16:41354) 2024-04-05T23:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7395600700 1 -- 172.21.15.16:0/1298540997 learned_addr learned my addr 172.21.15.16:0/1298540997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7396000700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f739019df00 0x7f739019e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7394c00700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7390106150 0x7f7390199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7395600700 1 -- 172.21.15.16:0/1298540997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f739019df00 msgr2=0x7f739019e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7395600700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f739019df00 0x7f739019e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7395600700 1 -- 172.21.15.16:0/1298540997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7390106150 msgr2=0x7f7390199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7395600700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7390106150 0x7f7390199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.116+0000 7f7395600700 1 -- 172.21.15.16:0/1298540997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7388009d00 con 0x7f7390101030 2024-04-05T23:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.117+0000 7f7395600700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390101030 0x7f7390199620 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f738400d950 tx=0x7f738400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.117+0000 7f738e000700 1 -- 172.21.15.16:0/1298540997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738400f890 con 0x7f7390101030 2024-04-05T23:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.117+0000 7f738e000700 1 -- 172.21.15.16:0/1298540997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7384010460 con 0x7f7390101030 2024-04-05T23:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.117+0000 7f738e000700 1 -- 172.21.15.16:0/1298540997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738400e610 con 0x7f7390101030 2024-04-05T23:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.117+0000 7f7398000700 1 -- 172.21.15.16:0/1298540997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73901a6210 con 0x7f7390101030 2024-04-05T23:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.117+0000 7f7398000700 1 -- 172.21.15.16:0/1298540997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73901a66f0 con 0x7f7390101030 2024-04-05T23:42:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.120+0000 7f738e000700 1 -- 172.21.15.16:0/1298540997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f73840108d0 con 0x7f7390101030 2024-04-05T23:42:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.121+0000 7f738e000700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73780a1310 0x7f73780a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f73abbf3b80 1 Processor -- start 2024-04-05T23:42:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f738e000700 1 -- 172.21.15.16:0/1298540997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8600+0+0 (secure 0 0 0) 0x7f73840ccad0 con 0x7f7390101030 2024-04-05T23:42:06.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f7394c00700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73780a1310 0x7f73780a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f73abbf3b80 1 -- start start 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b201475f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 0x55b201470030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201470a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2013d6b80 con 0x55b2014764a0 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2013c7930 con 0x55b2014705c0 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2013c52f0 con 0x55b201475b80 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f7395600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b201475f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f7395600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b201475f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:35174/0 (socket says 172.21.15.16:35174) 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.122+0000 7f7395600700 1 -- 172.21.15.16:0/2871244245 learned_addr learned my addr 172.21.15.16:0/2871244245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7394c00700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73780a1310 0x7f73780a37d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f739019ab70 tx=0x7f737c009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7396000700 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201470a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7394c00700 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 0x55b201470030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7395600700 1 -- 172.21.15.16:0/2871244245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 msgr2=0x55b201470a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7395600700 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201470a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7395600700 1 -- 172.21.15.16:0/2871244245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 msgr2=0x55b201470030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7395600700 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 0x55b201470030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7395600700 1 -- 172.21.15.16:0/2871244245 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b201242350 con 0x55b201475b80 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7394c00700 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 0x55b201470030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7395600700 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b201475f60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f73840a1a80 tx=0x7f738409a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40b62d4e6dd94b85 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7382600700 1 -- 172.21.15.16:0/2871244245 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73840a1fb0 con 0x55b201475b80 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7382600700 1 -- 172.21.15.16:0/2871244245 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7384096240 con 0x55b201475b80 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.123+0000 7f7382600700 1 -- 172.21.15.16:0/2871244245 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384099280 con 0x55b201475b80 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.124+0000 7f73abbf3b80 1 -- 172.21.15.16:0/2871244245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 msgr2=0x55b201475f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.124+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b201475f60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f73840a1a80 tx=0x7f738409a230 comp rx=0 tx=0).stop 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.124+0000 7f73abbf3b80 1 -- 172.21.15.16:0/2871244245 shutdown_connections 2024-04-05T23:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.124+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201470a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.124+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b201475f60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.124+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/2871244245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 0x55b201470030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.124+0000 7f73abbf3b80 1 -- 172.21.15.16:0/2871244245 >> 172.21.15.16:0/2871244245 conn(0x55b2013e41c0 msgr2=0x55b20146abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.124+0000 7f73abbf3b80 1 -- 172.21.15.16:0/2871244245 shutdown_connections 2024-04-05T23:42:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.124+0000 7f73abbf3b80 1 -- 172.21.15.16:0/2871244245 wait complete. 2024-04-05T23:42:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.125+0000 7f73abbf3b80 1 -- start start 2024-04-05T23:42:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.125+0000 7f73abbf3b80 1 Processor -- start 2024-04-05T23:42:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.125+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014705c0 0x55b201479880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.125+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b201475b80 0x55b201479dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.125+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 0x55b20147a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.125+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2013c0be0 con 0x55b2014705c0 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.125+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b20135cd20 con 0x55b201475b80 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.125+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b20135c9c0 con 0x55b2014764a0 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f73abbf3b80 10 client.0 _ll_register_callbacks cb 0x55b2011c8340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7395600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014705c0 0x55b201479880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7395600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014705c0 0x55b201479880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41374/0 (socket says 172.21.15.16:41374) 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7395600700 1 -- 172.21.15.16:0/3386676110 learned_addr learned my addr 172.21.15.16:0/3386676110 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7396000700 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 0x55b20147a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7394c00700 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b201475b80 0x55b201479dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7395600700 1 -- 172.21.15.16:0/3386676110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 msgr2=0x55b20147a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7395600700 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 0x55b20147a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7395600700 1 -- 172.21.15.16:0/3386676110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b201475b80 msgr2=0x55b201479dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7395600700 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b201475b80 0x55b201479dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7395600700 1 -- 172.21.15.16:0/3386676110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73901a66f0 con 0x55b2014705c0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7394c00700 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b201475b80 0x55b201479dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7395600700 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014705c0 0x55b201479880 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f73840ca6f0 tx=0x7f7384099250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef4d78a9d148bb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.126+0000 7f7376a00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f7376a00700 1 -- 172.21.15.16:0/3386676110 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73840b6070 con 0x55b2014705c0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f7376a00700 1 -- 172.21.15.16:0/3386676110 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8600+0+0 (secure 0 0 0) 0x7f73840a1fb0 con 0x55b2014705c0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f73abbf3b80 1 -- 172.21.15.16:0/3386676110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b2014631e0 con 0x55b2014705c0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f7376a00700 1 -- 172.21.15.16:0/3386676110 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7384097ed0 con 0x55b2014705c0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f73abbf3b80 10 client.16491 fetch_fsmap learned FSMap version 59 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f73abbf3b80 1 -- 172.21.15.16:0/3386676110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) v3 -- 0x55b2011eeb00 con 0x55b2014705c0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f7376a00700 1 -- 172.21.15.16:0/3386676110 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b2014631e0 con 0x55b2014705c0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f7376a00700 1 -- 172.21.15.16:0/3386676110 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 59) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f7384095920 con 0x55b2014705c0 2024-04-05T23:42:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f73abbf3b80 10 client.16491 fetch_fsmap finished waiting for FSMap version 59 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f73abbf3b80 10 client.16491 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f73abbf3b80 20 client.16491 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b201489a90 0x55b20148beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f73abbf3b80 4 client.16491 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.127+0000 7f73abbf3b80 1 -- 172.21.15.16:0/3386676110 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b2013b96a0 con 0x55b201489a90 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.128+0000 7f7394c00700 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b201489a90 0x55b20148beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.128+0000 7f7394c00700 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b201489a90 0x55b20148beb0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=153b3fc8f4c4a288 server_cookie=280eca7d25e52b85 in_seq=0 out_seq=0 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.128+0000 7f7376a00700 10 client.16491 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f7376a00700 1 -- 172.21.15.16:0/3386676110 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b2013b96a0 con 0x55b201489a90 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f7376a00700 10 client.16491 handle_command_reply: tid=0 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f73abbf3b80 1 client.16491 shutdown 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f73abbf3b80 1 -- 172.21.15.16:0/3386676110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014705c0 msgr2=0x55b201479880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014705c0 0x55b201479880 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f73840ca6f0 tx=0x7f7384099250 comp rx=0 tx=0).stop 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f73abbf3b80 1 -- 172.21.15.16:0/3386676110 shutdown_connections 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b201489a90 0x55b20148beb0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014705c0 0x55b201479880 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 0x55b20147a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/3386676110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b201475b80 0x55b201479dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f7376a00700 0 client.16491 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.129+0000 7f73abbf3b80 1 -- 172.21.15.16:0/3386676110 >> 172.21.15.16:0/3386676110 conn(0x55b2013e41c0 msgr2=0x55b20146a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.130+0000 7f73abbf3b80 1 -- 172.21.15.16:0/3386676110 shutdown_connections 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.130+0000 7f73abbf3b80 1 -- 172.21.15.16:0/3386676110 wait complete. 2024-04-05T23:42:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.130+0000 7f73abbf3b80 20 client.16491 trim_cache size 0 max 16384 2024-04-05T23:42:06.136 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.145+0000 7f73abbf3b80 1 Processor -- start 2024-04-05T23:42:06.136 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.145+0000 7f73abbf3b80 1 -- start start 2024-04-05T23:42:06.136 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201470030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b201475b80 0x55b201475040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 0x55b20147a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b201244640 con 0x55b201475b80 2024-04-05T23:42:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b201477df0 con 0x55b2014705c0 2024-04-05T23:42:06.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2014742d0 con 0x55b2014764a0 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f7394c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b201475b80 0x55b201475040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f7394c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b201475b80 0x55b201475040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41380/0 (socket says 172.21.15.16:41380) 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f7394c00700 1 -- 172.21.15.16:0/4246723080 learned_addr learned my addr 172.21.15.16:0/4246723080 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f7395600700 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201470030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.146+0000 7f7396000700 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 0x55b20147a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7394c00700 1 -- 172.21.15.16:0/4246723080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 msgr2=0x55b20147a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7394c00700 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 0x55b20147a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7394c00700 1 -- 172.21.15.16:0/4246723080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 msgr2=0x55b201470030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7394c00700 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201470030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7394c00700 1 -- 172.21.15.16:0/4246723080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b201386e40 con 0x55b201475b80 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7395600700 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201470030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:06.139 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7394c00700 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b201475b80 0x55b201475040 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f737c00a010 tx=0x7f737c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fa856a20b37e51 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7383000700 1 -- 172.21.15.16:0/4246723080 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c023040 con 0x55b201475b80 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7383000700 1 -- 172.21.15.16:0/4246723080 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f737c011e90 con 0x55b201475b80 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.147+0000 7f7383000700 1 -- 172.21.15.16:0/4246723080 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c004470 con 0x55b201475b80 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 -- 172.21.15.16:0/4246723080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b201475b80 msgr2=0x55b201475040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b201475b80 0x55b201475040 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f737c00a010 tx=0x7f737c014000 comp rx=0 tx=0).stop 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 -- 172.21.15.16:0/4246723080 shutdown_connections 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b2014764a0 0x55b20147a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201470030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/4246723080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b201475b80 0x55b201475040 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 -- 172.21.15.16:0/4246723080 >> 172.21.15.16:0/4246723080 conn(0x55b2013e41c0 msgr2=0x55b201474930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 -- 172.21.15.16:0/4246723080 shutdown_connections 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 -- 172.21.15.16:0/4246723080 wait complete. 2024-04-05T23:42:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.148+0000 7f73abbf3b80 1 -- start start 2024-04-05T23:42:06.143 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.149+0000 7f73abbf3b80 1 Processor -- start 2024-04-05T23:42:06.143 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.149+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201480e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.143 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.149+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b20147f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.143 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.149+0000 7f73abbf3b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 0x55b20147f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.143 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.149+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b20147fdb0 con 0x55b2014764a0 2024-04-05T23:42:06.143 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.149+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b20147fef0 con 0x55b2014705c0 2024-04-05T23:42:06.143 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.149+0000 7f73abbf3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2014800b0 con 0x55b201475b80 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.149+0000 7f73abbf3b80 10 client.0 _ll_register_callbacks cb 0x7f737c005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7395600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201480e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7395600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201480e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35542/0 (socket says 172.21.15.16:35542) 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7395600700 1 -- 172.21.15.16:0/701511222 learned_addr learned my addr 172.21.15.16:0/701511222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7396000700 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 0x55b20147f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7394c00700 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b20147f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7395600700 1 -- 172.21.15.16:0/701511222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 msgr2=0x55b20147f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7395600700 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b20147f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7395600700 1 -- 172.21.15.16:0/701511222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 msgr2=0x55b20147f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7395600700 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 0x55b20147f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7395600700 1 -- 172.21.15.16:0/701511222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b2011eeb00 con 0x55b2014705c0 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7395600700 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201480e00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f73840051d0 tx=0x7f7384005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e8d9695a8fd5ee3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7376a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7376a00700 1 -- 172.21.15.16:0/701511222 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73840b6070 con 0x55b2014705c0 2024-04-05T23:42:06.144 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f7376a00700 1 -- 172.21.15.16:0/701511222 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8600+0+0 (secure 0 0 0) 0x7f73840963b0 con 0x55b2014705c0 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.150+0000 7f73abbf3b80 1 -- 172.21.15.16:0/701511222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7384096240 con 0x55b2014705c0 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f7376a00700 1 -- 172.21.15.16:0/701511222 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7384083120 con 0x55b2014705c0 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f73abbf3b80 10 client.26546 fetch_fsmap learned FSMap version 59 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f73abbf3b80 1 -- 172.21.15.16:0/701511222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) v3 -- 0x7f737c017070 con 0x55b2014705c0 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f7376a00700 1 -- 172.21.15.16:0/701511222 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7384096240 con 0x55b2014705c0 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f7376a00700 1 -- 172.21.15.16:0/701511222 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 59) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f7384098a90 con 0x55b2014705c0 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f73abbf3b80 10 client.26546 fetch_fsmap finished waiting for FSMap version 59 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f73abbf3b80 10 client.26546 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f73abbf3b80 20 client.26546 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2014dea60 0x55b2014e0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f73abbf3b80 4 client.26546 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f73abbf3b80 1 -- 172.21.15.16:0/701511222 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b20120fc90 con 0x55b2014dea60 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.151+0000 7f7394c00700 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2014dea60 0x55b2014e0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.152+0000 7f7394c00700 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2014dea60 0x55b2014e0e80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bea53d4c2a24ae5c server_cookie=79b850a3008c7d37 in_seq=0 out_seq=0 2024-04-05T23:42:06.145 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.152+0000 7f7376a00700 10 client.26546 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f7376a00700 1 -- 172.21.15.16:0/701511222 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b20120fc90 con 0x55b2014dea60 2024-04-05T23:42:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f7376a00700 10 client.26546 handle_command_reply: tid=0 2024-04-05T23:42:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 client.26546 shutdown 2024-04-05T23:42:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 -- 172.21.15.16:0/701511222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 msgr2=0x55b201480e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201480e00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f73840051d0 tx=0x7f7384005dc0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 -- 172.21.15.16:0/701511222 shutdown_connections 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55b2014dea60 0x55b2014e0e80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f7376a00700 0 client.26546 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b2014764a0 0x55b20147f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2014705c0 0x55b201480e00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 --2- 172.21.15.16:0/701511222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b201475b80 0x55b20147f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 -- 172.21.15.16:0/701511222 >> 172.21.15.16:0/701511222 conn(0x55b2013e41c0 msgr2=0x55b2014751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 -- 172.21.15.16:0/701511222 shutdown_connections 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.182+0000 7f73abbf3b80 1 -- 172.21.15.16:0/701511222 wait complete. 2024-04-05T23:42:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.183+0000 7f73abbf3b80 20 client.26546 trim_cache size 0 max 16384 2024-04-05T23:42:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:42:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:42:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:42:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:42:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.183+0000 7f7398000700 1 -- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73780a1310 msgr2=0x7f73780a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.183+0000 7f7398000700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73780a1310 0x7f73780a37d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f739019ab70 tx=0x7f737c009260 comp rx=0 tx=0).stop 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.183+0000 7f7398000700 1 -- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390101030 msgr2=0x7f7390199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.183+0000 7f7398000700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390101030 0x7f7390199620 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f738400d950 tx=0x7f738400dc60 comp rx=0 tx=0).stop 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.184+0000 7f7398000700 1 -- 172.21.15.16:0/1298540997 shutdown_connections 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.184+0000 7f7398000700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73780a1310 0x7f73780a37d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.184+0000 7f7398000700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7390101030 0x7f7390199620 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.184+0000 7f7398000700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f739019df00 0x7f739019e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.184+0000 7f7398000700 1 --2- 172.21.15.16:0/1298540997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7390106150 0x7f7390199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.184+0000 7f7398000700 1 -- 172.21.15.16:0/1298540997 >> 172.21.15.16:0/1298540997 conn(0x7f73900fc8b0 msgr2=0x7f73900febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.184+0000 7f7398000700 1 -- 172.21.15.16:0/1298540997 shutdown_connections 2024-04-05T23:42:06.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:06.184+0000 7f7398000700 1 -- 172.21.15.16:0/1298540997 wait complete. 2024-04-05T23:42:06.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:05 smithi027 ceph-mon[78597]: pgmap v266: 97 pgs: 97 active+clean; 158 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-04-05T23:42:07.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 431622 5.14 MB/sec execute 326 sec latency 3203.824 ms 2024-04-05T23:42:08.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 433156 5.16 MB/sec execute 327 sec latency 11.437 ms 2024-04-05T23:42:08.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:07 smithi016 ceph-mon[38248]: pgmap v267: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-05T23:42:08.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:07 smithi130 ceph-mon[79417]: pgmap v267: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-05T23:42:08.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:07 smithi027 ceph-mon[78597]: pgmap v267: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-05T23:42:09.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434285 5.16 MB/sec execute 328 sec latency 19.238 ms 2024-04-05T23:42:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434458 5.14 MB/sec execute 329 sec latency 782.450 ms 2024-04-05T23:42:10.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:10 smithi016 ceph-mon[38248]: pgmap v268: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-05T23:42:10.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:10 smithi016 ceph-mon[38248]: osdmap e91: 12 total, 12 up, 12 in 2024-04-05T23:42:10.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:10 smithi130 ceph-mon[79417]: pgmap v268: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-05T23:42:10.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:10 smithi130 ceph-mon[79417]: osdmap e91: 12 total, 12 up, 12 in 2024-04-05T23:42:10.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:10 smithi027 ceph-mon[78597]: pgmap v268: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-04-05T23:42:10.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:10 smithi027 ceph-mon[78597]: osdmap e91: 12 total, 12 up, 12 in 2024-04-05T23:42:11.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434458 5.13 MB/sec execute 330 sec latency 1782.575 ms 2024-04-05T23:42:12.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 435155 5.12 MB/sec execute 331 sec latency 2501.393 ms 2024-04-05T23:42:12.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:12 smithi016 ceph-mon[38248]: pgmap v270: 97 pgs: 97 active+clean; 160 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 241 op/s 2024-04-05T23:42:12.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:12 smithi130 ceph-mon[79417]: pgmap v270: 97 pgs: 97 active+clean; 160 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 241 op/s 2024-04-05T23:42:12.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:12 smithi027 ceph-mon[78597]: pgmap v270: 97 pgs: 97 active+clean; 160 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 241 op/s 2024-04-05T23:42:13.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 436673 5.14 MB/sec execute 332 sec latency 13.601 ms 2024-04-05T23:42:13.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:13 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:13.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:13 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:13.970 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:13 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:14.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 437806 5.14 MB/sec execute 333 sec latency 19.660 ms 2024-04-05T23:42:14.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:14 smithi016 ceph-mon[38248]: pgmap v271: 97 pgs: 97 active+clean; 160 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-04-05T23:42:14.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:14 smithi130 ceph-mon[79417]: pgmap v271: 97 pgs: 97 active+clean; 160 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-04-05T23:42:14.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:14 smithi027 ceph-mon[78597]: pgmap v271: 97 pgs: 97 active+clean; 160 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 211 op/s 2024-04-05T23:42:15.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438005 5.12 MB/sec execute 334 sec latency 705.686 ms 2024-04-05T23:42:16.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 439928 5.15 MB/sec execute 335 sec latency 788.061 ms 2024-04-05T23:42:16.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:16 smithi016 ceph-mon[38248]: pgmap v272: 97 pgs: 97 active+clean; 160 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-05T23:42:16.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:16 smithi130 ceph-mon[79417]: pgmap v272: 97 pgs: 97 active+clean; 160 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-05T23:42:16.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:16 smithi027 ceph-mon[78597]: pgmap v272: 97 pgs: 97 active+clean; 160 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-05T23:42:17.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 440914 5.14 MB/sec execute 336 sec latency 16.101 ms 2024-04-05T23:42:18.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 441547 5.14 MB/sec execute 337 sec latency 584.871 ms 2024-04-05T23:42:18.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:18 smithi016 ceph-mon[38248]: pgmap v273: 97 pgs: 97 active+clean; 164 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-05T23:42:18.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:18 smithi130 ceph-mon[79417]: pgmap v273: 97 pgs: 97 active+clean; 164 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-05T23:42:18.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:18 smithi027 ceph-mon[78597]: pgmap v273: 97 pgs: 97 active+clean; 164 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-05T23:42:19.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 441547 5.12 MB/sec execute 338 sec latency 1584.956 ms 2024-04-05T23:42:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.343+0000 7fa2cda00700 1 -- 172.21.15.16:0/1200066951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8106160 msgr2=0x7fa2c810dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.343+0000 7fa2cda00700 1 --2- 172.21.15.16:0/1200066951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8106160 0x7fa2c810dce0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000b0a0 tx=0x7fa2c000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.344+0000 7fa2cda00700 1 -- 172.21.15.16:0/1200066951 shutdown_connections 2024-04-05T23:42:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.344+0000 7fa2cda00700 1 --2- 172.21.15.16:0/1200066951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8106160 0x7fa2c810dce0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.344+0000 7fa2cda00700 1 --2- 172.21.15.16:0/1200066951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2c8101040 0x7fa2c8101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.344+0000 7fa2cda00700 1 --2- 172.21.15.16:0/1200066951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2c81019f0 0x7fa2c8105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.344+0000 7fa2cda00700 1 -- 172.21.15.16:0/1200066951 >> 172.21.15.16:0/1200066951 conn(0x7fa2c80fc8e0 msgr2=0x7fa2c80fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.344+0000 7fa2cda00700 1 -- 172.21.15.16:0/1200066951 shutdown_connections 2024-04-05T23:42:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.344+0000 7fa2cda00700 1 -- 172.21.15.16:0/1200066951 wait complete. 2024-04-05T23:42:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.345+0000 7fa2cda00700 1 Processor -- start 2024-04-05T23:42:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.345+0000 7fa2cda00700 1 -- start start 2024-04-05T23:42:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.345+0000 7fa2cda00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8101040 0x7fa2c81973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.345+0000 7fa2cda00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2c8106160 0x7fa2c8197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.345+0000 7fa2cda00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2c819bcd0 0x7fa2c819c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.345+0000 7fa2cda00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c819c750 con 0x7fa2c8101040 2024-04-05T23:42:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.345+0000 7fa2cda00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c819c8c0 con 0x7fa2c819bcd0 2024-04-05T23:42:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.345+0000 7fa2cda00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c819ca80 con 0x7fa2c8106160 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8101040 0x7fa2c81973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2c819bcd0 0x7fa2c819c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2c8106160 0x7fa2c8197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2c819bcd0 0x7fa2c819c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37604/0 (socket says 172.21.15.16:37604) 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c7400700 1 -- 172.21.15.16:0/3553560403 learned_addr learned my addr 172.21.15.16:0/3553560403 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c7400700 1 -- 172.21.15.16:0/3553560403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2c8106160 msgr2=0x7fa2c8197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c7400700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2c8106160 0x7fa2c8197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c7400700 1 -- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8101040 msgr2=0x7fa2c81973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c7400700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8101040 0x7fa2c81973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c7400700 1 -- 172.21.15.16:0/3553560403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c0009d00 con 0x7fa2c819bcd0 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c6000700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2c8106160 0x7fa2c8197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:19.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8101040 0x7fa2c81973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2c7400700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2c819bcd0 0x7fa2c819c150 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000b070 tx=0x7fa2c000bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.346+0000 7fa2bf400700 1 -- 172.21.15.16:0/3553560403 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c000e040 con 0x7fa2c819bcd0 2024-04-05T23:42:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.347+0000 7fa2bf400700 1 -- 172.21.15.16:0/3553560403 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2c001d480 con 0x7fa2c819bcd0 2024-04-05T23:42:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.347+0000 7fa2cda00700 1 -- 172.21.15.16:0/3553560403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c81a3f80 con 0x7fa2c819bcd0 2024-04-05T23:42:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.347+0000 7fa2bf400700 1 -- 172.21.15.16:0/3553560403 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c00052a0 con 0x7fa2c819bcd0 2024-04-05T23:42:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.347+0000 7fa2cda00700 1 -- 172.21.15.16:0/3553560403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c81a43a0 con 0x7fa2c819bcd0 2024-04-05T23:42:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.350+0000 7fa2bf400700 1 -- 172.21.15.16:0/3553560403 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fa2c001d5f0 con 0x7fa2c819bcd0 2024-04-05T23:42:19.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.351+0000 7fa2e146bb80 1 Processor -- start 2024-04-05T23:42:19.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.351+0000 7fa2bf400700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa2ac0a1310 0x7fa2ac0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.351+0000 7fa2bf400700 1 -- 172.21.15.16:0/3553560403 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fa2c00cdd10 con 0x7fa2c819bcd0 2024-04-05T23:42:19.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.351+0000 7fa2e146bb80 1 -- start start 2024-04-05T23:42:19.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.351+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa2ac0a1310 0x7fa2ac0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 0x5623470e1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e18b0 0x5623470db440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470dbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56234703db80 con 0x5623470e18b0 2024-04-05T23:42:19.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56234702e930 con 0x5623470db980 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56234702c2f0 con 0x5623470e0f90 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2c6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e18b0 0x5623470db440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2c6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e18b0 0x5623470db440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51776/0 (socket says 172.21.15.16:51776) 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2c6000700 1 -- 172.21.15.16:0/2754740667 learned_addr learned my addr 172.21.15.16:0/2754740667 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2c7400700 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470dbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 0x5623470e1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa2ac0a1310 0x7fa2ac0a37d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8011550 tx=0x7fa2b8009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2c6000700 1 -- 172.21.15.16:0/2754740667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 msgr2=0x5623470e1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.352+0000 7fa2c6000700 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 0x5623470e1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.353+0000 7fa2c6000700 1 -- 172.21.15.16:0/2754740667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 msgr2=0x5623470dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.353+0000 7fa2c6000700 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470dbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.353+0000 7fa2c6000700 1 -- 172.21.15.16:0/2754740667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562346ea9350 con 0x5623470e18b0 2024-04-05T23:42:19.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.353+0000 7fa2c7400700 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470dbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:19.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.353+0000 7fa2c6000700 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e18b0 0x5623470db440 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40114f0 tx=0x7fa2b4011800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c2af848c17deb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:19.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.353+0000 7fa2b3e00700 1 -- 172.21.15.16:0/2754740667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4004a30 con 0x5623470e18b0 2024-04-05T23:42:19.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.353+0000 7fa2b3e00700 1 -- 172.21.15.16:0/2754740667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2b40054e0 con 0x5623470e18b0 2024-04-05T23:42:19.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.353+0000 7fa2b3e00700 1 -- 172.21.15.16:0/2754740667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b400e8f0 con 0x5623470e18b0 2024-04-05T23:42:19.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 -- 172.21.15.16:0/2754740667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e18b0 msgr2=0x5623470db440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:19.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e18b0 0x5623470db440 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40114f0 tx=0x7fa2b4011800 comp rx=0 tx=0).stop 2024-04-05T23:42:19.351 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 -- 172.21.15.16:0/2754740667 shutdown_connections 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470dbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 0x5623470e1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/2754740667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e18b0 0x5623470db440 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 -- 172.21.15.16:0/2754740667 >> 172.21.15.16:0/2754740667 conn(0x56234704b1c0 msgr2=0x5623470d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 -- 172.21.15.16:0/2754740667 shutdown_connections 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 -- 172.21.15.16:0/2754740667 wait complete. 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.354+0000 7fa2e146bb80 1 -- start start 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.355+0000 7fa2e146bb80 1 Processor -- start 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.355+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.355+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470e4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.355+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 0x5623470e5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.355+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562347027be0 con 0x5623470e0f90 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.355+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562346fc3d20 con 0x5623470db980 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.355+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562346fc39c0 con 0x5623470e18b0 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.355+0000 7fa2e146bb80 10 client.0 _ll_register_callbacks cb 0x562346e2f340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470e4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37624/0 (socket says 172.21.15.16:37624) 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6a00700 1 -- 172.21.15.16:0/4119337295 learned_addr learned my addr 172.21.15.16:0/4119337295 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c7400700 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 0x5623470e5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6a00700 1 -- 172.21.15.16:0/4119337295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 msgr2=0x5623470e5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 0x5623470e5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6a00700 1 -- 172.21.15.16:0/4119337295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 msgr2=0x5623470e4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470e4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.353 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6a00700 1 -- 172.21.15.16:0/4119337295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c81a43a0 con 0x5623470db980 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e77b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8003720 tx=0x7fa2b80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d5c96c9bdef0f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.356+0000 7fa2b0c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4119337295 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8022070 con 0x5623470db980 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4119337295 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fa2b801d600 con 0x5623470db980 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4119337295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5623470ca1e0 con 0x5623470db980 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4119337295 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2b8003ea0 con 0x5623470db980 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2e146bb80 10 client.26558 fetch_fsmap learned FSMap version 60 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4119337295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) v3 -- 0x7fa2b4009c90 con 0x5623470db980 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4119337295 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2b8033020 con 0x5623470db980 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4119337295 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 60) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa2b80269e0 con 0x5623470db980 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2e146bb80 10 client.26558 fetch_fsmap finished waiting for FSMap version 60 2024-04-05T23:42:19.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.357+0000 7fa2e146bb80 10 client.26558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.358+0000 7fa2e146bb80 20 client.26558 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.358+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5623470f5350 0x5623470f7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.358+0000 7fa2e146bb80 4 client.26558 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.358+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4119337295 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5623470206a0 con 0x5623470f5350 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.358+0000 7fa2c6000700 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5623470f5350 0x5623470f7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.359+0000 7fa2c6000700 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5623470f5350 0x5623470f7770 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dec9f977410db80f server_cookie=d5b2f4a0d8ec853b in_seq=0 out_seq=0 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.359+0000 7fa2b0c00700 10 client.26558 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.359+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4119337295 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5623470206a0 con 0x5623470f5350 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.359+0000 7fa2b0c00700 10 client.26558 handle_command_reply: tid=0 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.359+0000 7fa2e146bb80 1 client.26558 shutdown 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.359+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4119337295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 msgr2=0x5623470e77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.359+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e77b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8003720 tx=0x7fa2b80047c0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4119337295 shutdown_connections 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5623470f5350 0x5623470f7770 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 0x5623470e5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e77b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4119337295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470e4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4119337295 >> 172.21.15.16:0/4119337295 conn(0x56234704b1c0 msgr2=0x5623470d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2b0c00700 0 client.26558 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4119337295 shutdown_connections 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4119337295 wait complete. 2024-04-05T23:42:19.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.360+0000 7fa2e146bb80 20 client.26558 trim_cache size 0 max 16384 2024-04-05T23:42:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.375+0000 7fa2e146bb80 1 Processor -- start 2024-04-05T23:42:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.375+0000 7fa2e146bb80 1 -- start start 2024-04-05T23:42:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 0x5623470dbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562346eab640 con 0x5623470e0f90 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623470df7f0 con 0x5623470db980 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623470e15c0 con 0x5623470e18b0 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37636/0 (socket says 172.21.15.16:37636) 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6a00700 1 -- 172.21.15.16:0/1517912518 learned_addr learned my addr 172.21.15.16:0/1517912518 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51800/0 (socket says 172.21.15.16:51800) 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c7400700 1 --2- 172.21.15.16:0/1517912518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 0x5623470dbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6a00700 1 -- 172.21.15.16:0/1517912518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 msgr2=0x5623470dbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/1517912518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 0x5623470dbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6a00700 1 -- 172.21.15.16:0/1517912518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 msgr2=0x5623470d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/1517912518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6a00700 1 -- 172.21.15.16:0/1517912518 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562346e55b00 con 0x5623470db980 2024-04-05T23:42:19.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.376+0000 7fa2c6000700 1 --2- 172.21.15.16:0/1517912518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:19.370 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.377+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/1517912518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e2ca0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8026d40 tx=0x7fa2b8009350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c823517ded0066d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.377+0000 7fa2bcc00700 1 -- 172.21.15.16:0/1517912518 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8022070 con 0x5623470db980 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.377+0000 7fa2bcc00700 1 -- 172.21.15.16:0/1517912518 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2b8033030 con 0x5623470db980 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.377+0000 7fa2bcc00700 1 -- 172.21.15.16:0/1517912518 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b801da80 con 0x5623470db980 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 -- 172.21.15.16:0/1517912518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 msgr2=0x5623470e2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/1517912518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e2ca0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8026d40 tx=0x7fa2b8009350 comp rx=0 tx=0).stop 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 -- 172.21.15.16:0/1517912518 shutdown_connections 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/1517912518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e18b0 0x5623470dbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/1517912518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470db980 0x5623470e2ca0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/1517912518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470e0f90 0x5623470d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 -- 172.21.15.16:0/1517912518 >> 172.21.15.16:0/1517912518 conn(0x56234704b1c0 msgr2=0x5623470fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 -- 172.21.15.16:0/1517912518 shutdown_connections 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 -- 172.21.15.16:0/1517912518 wait complete. 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 -- start start 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.378+0000 7fa2e146bb80 1 Processor -- start 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470db980 0x5623470dca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 0x5623470dcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2e146bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470e18b0 0x5623470dd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623470ddb60 con 0x5623470db980 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623470ddcd0 con 0x5623470e18b0 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2e146bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623470dde90 con 0x5623470e0f90 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2e146bb80 10 client.0 _ll_register_callbacks cb 0x7fa2b8004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2c6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470db980 0x5623470dca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2c6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 0x5623470dcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2c6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470db980 0x5623470dca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51806/0 (socket says 172.21.15.16:51806) 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.379+0000 7fa2c6a00700 1 -- 172.21.15.16:0/4113731769 learned_addr learned my addr 172.21.15.16:0/4113731769 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2c7400700 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470e18b0 0x5623470dd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2c6a00700 1 -- 172.21.15.16:0/4113731769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 msgr2=0x5623470dcf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 0x5623470dcf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2c6a00700 1 -- 172.21.15.16:0/4113731769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470e18b0 msgr2=0x5623470dd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470e18b0 0x5623470dd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2c6a00700 1 -- 172.21.15.16:0/4113731769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562346ea9350 con 0x5623470db980 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2c6000700 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 0x5623470dcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2c6a00700 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470db980 0x5623470dca50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8011da0 tx=0x7fa2b80057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51cf10241b1e70eb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2b0c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4113731769 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8022070 con 0x5623470db980 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4113731769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2b40054e0 con 0x5623470db980 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.380+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4113731769 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fa2b8034080 con 0x5623470db980 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4113731769 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2b8026620 con 0x5623470db980 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2e146bb80 10 client.16515 fetch_fsmap learned FSMap version 60 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4113731769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) v3 -- 0x7fa2b4009c90 con 0x5623470db980 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4113731769 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2b8045020 con 0x5623470db980 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4113731769 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 60) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa2b801b660 con 0x5623470db980 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2e146bb80 10 client.16515 fetch_fsmap finished waiting for FSMap version 60 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2e146bb80 10 client.16515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2e146bb80 20 client.16515 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562347149a00 0x56234714be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2e146bb80 4 client.16515 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:42:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4113731769 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562346e76c90 con 0x562347149a00 2024-04-05T23:42:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.381+0000 7fa2c6000700 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562347149a00 0x56234714be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.382+0000 7fa2c6000700 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562347149a00 0x56234714be20 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceb1b3b72e57fc67 server_cookie=cf62f91f181a452d in_seq=0 out_seq=0 2024-04-05T23:42:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.382+0000 7fa2b0c00700 10 client.16515 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2b0c00700 1 -- 172.21.15.16:0/4113731769 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562346e76c90 con 0x562347149a00 2024-04-05T23:42:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2b0c00700 10 client.16515 handle_command_reply: tid=0 2024-04-05T23:42:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 client.16515 shutdown 2024-04-05T23:42:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4113731769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470db980 msgr2=0x5623470dca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470db980 0x5623470dca50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8011da0 tx=0x7fa2b80057e0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4113731769 shutdown_connections 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562347149a00 0x56234714be20 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623470db980 0x5623470dca50 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623470e18b0 0x5623470dd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 --2- 172.21.15.16:0/4113731769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5623470e0f90 0x5623470dcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4113731769 >> 172.21.15.16:0/4113731769 conn(0x56234704b1c0 msgr2=0x5623470e3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2b0c00700 0 client.16515 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4113731769 shutdown_connections 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.418+0000 7fa2e146bb80 1 -- 172.21.15.16:0/4113731769 wait complete. 2024-04-05T23:42:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2e146bb80 20 client.16515 trim_cache size 0 max 16384 2024-04-05T23:42:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:42:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:42:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:42:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:42:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 -- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa2ac0a1310 msgr2=0x7fa2ac0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa2ac0a1310 0x7fa2ac0a37d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8011550 tx=0x7fa2b8009260 comp rx=0 tx=0).stop 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 -- 172.21.15.16:0/3553560403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2c819bcd0 msgr2=0x7fa2c819c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2c819bcd0 0x7fa2c819c150 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000b070 tx=0x7fa2c000bda0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 -- 172.21.15.16:0/3553560403 shutdown_connections 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa2ac0a1310 0x7fa2ac0a37d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2c8101040 0x7fa2c81973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2c819bcd0 0x7fa2c819c150 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 --2- 172.21.15.16:0/3553560403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2c8106160 0x7fa2c8197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 -- 172.21.15.16:0/3553560403 >> 172.21.15.16:0/3553560403 conn(0x7fa2c80fc8e0 msgr2=0x7fa2c80fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 -- 172.21.15.16:0/3553560403 shutdown_connections 2024-04-05T23:42:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:19.419+0000 7fa2cda00700 1 -- 172.21.15.16:0/3553560403 wait complete. 2024-04-05T23:42:20.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 441547 5.11 MB/sec execute 339 sec latency 2585.017 ms 2024-04-05T23:42:20.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:20 smithi016 ceph-mon[38248]: pgmap v274: 97 pgs: 97 active+clean; 164 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-05T23:42:20.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:20 smithi130 ceph-mon[79417]: pgmap v274: 97 pgs: 97 active+clean; 164 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-05T23:42:20.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:20 smithi027 ceph-mon[78597]: pgmap v274: 97 pgs: 97 active+clean; 164 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-05T23:42:21.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 441811 5.09 MB/sec execute 340 sec latency 3461.359 ms 2024-04-05T23:42:22.017 INFO:tasks.workunit.client.0.smithi016.stdout: 1 443649 5.12 MB/sec execute 341 sec latency 25.108 ms 2024-04-05T23:42:22.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:22 smithi016 ceph-mon[38248]: pgmap v275: 97 pgs: 97 active+clean; 159 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-04-05T23:42:22.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:22 smithi130 ceph-mon[79417]: pgmap v275: 97 pgs: 97 active+clean; 159 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-04-05T23:42:22.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:22 smithi027 ceph-mon[78597]: pgmap v275: 97 pgs: 97 active+clean; 159 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-04-05T23:42:23.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 444806 5.12 MB/sec execute 342 sec latency 25.970 ms 2024-04-05T23:42:24.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445096 5.10 MB/sec execute 343 sec latency 609.764 ms 2024-04-05T23:42:24.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:24 smithi016 ceph-mon[38248]: pgmap v276: 97 pgs: 97 active+clean; 159 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 121 op/s 2024-04-05T23:42:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:24 smithi130 ceph-mon[79417]: pgmap v276: 97 pgs: 97 active+clean; 159 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 121 op/s 2024-04-05T23:42:24.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:24 smithi027 ceph-mon[78597]: pgmap v276: 97 pgs: 97 active+clean; 159 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 121 op/s 2024-04-05T23:42:25.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445096 5.09 MB/sec execute 344 sec latency 1609.924 ms 2024-04-05T23:42:26.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445096 5.07 MB/sec execute 345 sec latency 2610.063 ms 2024-04-05T23:42:26.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:26 smithi016 ceph-mon[38248]: pgmap v277: 97 pgs: 97 active+clean; 159 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 121 op/s 2024-04-05T23:42:26.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:26 smithi130 ceph-mon[79417]: pgmap v277: 97 pgs: 97 active+clean; 159 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 121 op/s 2024-04-05T23:42:26.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:26 smithi027 ceph-mon[78597]: pgmap v277: 97 pgs: 97 active+clean; 159 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 121 op/s 2024-04-05T23:42:27.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445806 5.06 MB/sec execute 346 sec latency 3324.300 ms 2024-04-05T23:42:28.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 447274 5.08 MB/sec execute 347 sec latency 20.735 ms 2024-04-05T23:42:28.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:28 smithi016 ceph-mon[38248]: pgmap v278: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 179 op/s 2024-04-05T23:42:28.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:28 smithi130 ceph-mon[79417]: pgmap v278: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 179 op/s 2024-04-05T23:42:28.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:28 smithi027 ceph-mon[78597]: pgmap v278: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 179 op/s 2024-04-05T23:42:29.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 448525 5.08 MB/sec execute 348 sec latency 18.063 ms 2024-04-05T23:42:30.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 448638 5.07 MB/sec execute 349 sec latency 923.225 ms 2024-04-05T23:42:30.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:30 smithi016 ceph-mon[38248]: pgmap v279: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1013 KiB/s wr, 113 op/s 2024-04-05T23:42:30.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:30 smithi130 ceph-mon[79417]: pgmap v279: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1013 KiB/s wr, 113 op/s 2024-04-05T23:42:30.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:30 smithi027 ceph-mon[78597]: pgmap v279: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1013 KiB/s wr, 113 op/s 2024-04-05T23:42:31.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 448638 5.05 MB/sec execute 350 sec latency 1923.341 ms 2024-04-05T23:42:32.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 449342 5.05 MB/sec execute 351 sec latency 2637.440 ms 2024-04-05T23:42:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.588+0000 7f3041c00700 1 -- 172.21.15.16:0/530735614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c1019e0 msgr2=0x7f303c105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.588+0000 7f3041c00700 1 --2- 172.21.15.16:0/530735614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c1019e0 0x7f303c105a20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3028009990 tx=0x7f3028009ca0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.588+0000 7f3041c00700 1 -- 172.21.15.16:0/530735614 shutdown_connections 2024-04-05T23:42:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.588+0000 7f3041c00700 1 --2- 172.21.15.16:0/530735614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303c106150 0x7f303c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.588+0000 7f3041c00700 1 --2- 172.21.15.16:0/530735614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f303c101030 0x7f303c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.588+0000 7f3041c00700 1 --2- 172.21.15.16:0/530735614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c1019e0 0x7f303c105a20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.588+0000 7f3041c00700 1 -- 172.21.15.16:0/530735614 >> 172.21.15.16:0/530735614 conn(0x7f303c0fc8b0 msgr2=0x7f303c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.588+0000 7f3041c00700 1 -- 172.21.15.16:0/530735614 shutdown_connections 2024-04-05T23:42:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.588+0000 7f3041c00700 1 -- 172.21.15.16:0/530735614 wait complete. 2024-04-05T23:42:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.589+0000 7f3041c00700 1 Processor -- start 2024-04-05T23:42:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.589+0000 7f3041c00700 1 -- start start 2024-04-05T23:42:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f3041c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303c101030 0x7f303c199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f3041c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c106150 0x7f303c199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f3041c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f303c19de00 0x7f303c19e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f3041c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c19e880 con 0x7f303c101030 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f3041c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c19e9f0 con 0x7f303c106150 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f3041c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c19ebb0 con 0x7f303c19de00 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c106150 0x7f303c199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c106150 0x7f303c199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35790/0 (socket says 172.21.15.16:35790) 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303a000700 1 -- 172.21.15.16:0/2278041133 learned_addr learned my addr 172.21.15.16:0/2278041133 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303b400700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f303c19de00 0x7f303c19e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303aa00700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303c101030 0x7f303c199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303a000700 1 -- 172.21.15.16:0/2278041133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f303c19de00 msgr2=0x7f303c19e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303a000700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f303c19de00 0x7f303c19e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303a000700 1 -- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303c101030 msgr2=0x7f303c199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303a000700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303c101030 0x7f303c199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303a000700 1 -- 172.21.15.16:0/2278041133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30280095f0 con 0x7f303c106150 2024-04-05T23:42:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.590+0000 7f303b400700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f303c19de00 0x7f303c19e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.591+0000 7f303a000700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c106150 0x7f303c199a60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3028000c00 tx=0x7f3028004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.591+0000 7f3033400700 1 -- 172.21.15.16:0/2278041133 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302801d070 con 0x7f303c106150 2024-04-05T23:42:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.591+0000 7f3033400700 1 -- 172.21.15.16:0/2278041133 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3028021410 con 0x7f303c106150 2024-04-05T23:42:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.591+0000 7f3041c00700 1 -- 172.21.15.16:0/2278041133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c1a60b0 con 0x7f303c106150 2024-04-05T23:42:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.591+0000 7f3033400700 1 -- 172.21.15.16:0/2278041133 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3028017650 con 0x7f303c106150 2024-04-05T23:42:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.591+0000 7f3041c00700 1 -- 172.21.15.16:0/2278041133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c1a6580 con 0x7f303c106150 2024-04-05T23:42:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.594+0000 7f3033400700 1 -- 172.21.15.16:0/2278041133 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f3028021580 con 0x7f303c106150 2024-04-05T23:42:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.596+0000 7f3033400700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f30200a1310 0x7f30200a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.596+0000 7f3033400700 1 -- 172.21.15.16:0/2278041133 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f30280cde00 con 0x7f303c106150 2024-04-05T23:42:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.596+0000 7f303aa00700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f30200a1310 0x7f30200a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.596+0000 7f3055648b80 1 Processor -- start 2024-04-05T23:42:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.596+0000 7f3055648b80 1 -- start start 2024-04-05T23:42:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f303aa00700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f30200a1310 0x7f30200a37d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f302c009c60 tx=0x7f302c009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f3055648b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 0x562a25b21770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f3055648b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b1b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f3055648b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b1c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f3055648b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a25a79b80 con 0x562a25b21390 2024-04-05T23:42:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f3055648b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a25a6a930 con 0x562a25b1bdd0 2024-04-05T23:42:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f3055648b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a25a682f0 con 0x562a25b21cb0 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f303a000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b1b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f303aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 0x562a25b21770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f303a000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b1b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:46696/0 (socket says 172.21.15.16:46696) 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f303a000700 1 -- 172.21.15.16:0/1863611010 learned_addr learned my addr 172.21.15.16:0/1863611010 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.597+0000 7f303b400700 1 --2- 172.21.15.16:0/1863611010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b1c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.598+0000 7f303a000700 1 -- 172.21.15.16:0/1863611010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 msgr2=0x562a25b1c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.598+0000 7f303a000700 1 --2- 172.21.15.16:0/1863611010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b1c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.598+0000 7f303a000700 1 -- 172.21.15.16:0/1863611010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 msgr2=0x562a25b21770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.598+0000 7f303a000700 1 --2- 172.21.15.16:0/1863611010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 0x562a25b21770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.598+0000 7f303a000700 1 -- 172.21.15.16:0/1863611010 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a258e5350 con 0x562a25b21cb0 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.598+0000 7f303a000700 1 --2- 172.21.15.16:0/1863611010 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b1b840 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f30280a2de0 tx=0x7f30280a1120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4fd1ca7b6d7bbf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.598+0000 7f3027e00700 1 -- 172.21.15.16:0/1863611010 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302809a880 con 0x562a25b21cb0 2024-04-05T23:42:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.598+0000 7f3027e00700 1 -- 172.21.15.16:0/1863611010 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f302809b620 con 0x562a25b21cb0 2024-04-05T23:42:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.598+0000 7f3027e00700 1 -- 172.21.15.16:0/1863611010 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302809b180 con 0x562a25b21cb0 2024-04-05T23:42:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.599+0000 7f3055648b80 1 -- 172.21.15.16:0/1863611010 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 msgr2=0x562a25b1b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.599+0000 7f3055648b80 1 --2- 172.21.15.16:0/1863611010 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b1b840 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f30280a2de0 tx=0x7f30280a1120 comp rx=0 tx=0).stop 2024-04-05T23:42:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.599+0000 7f3055648b80 1 -- 172.21.15.16:0/1863611010 shutdown_connections 2024-04-05T23:42:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.599+0000 7f3055648b80 1 --2- 172.21.15.16:0/1863611010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 0x562a25b21770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.599+0000 7f3055648b80 1 --2- 172.21.15.16:0/1863611010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b1c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.599+0000 7f3055648b80 1 --2- 172.21.15.16:0/1863611010 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b1b840 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.599+0000 7f3055648b80 1 -- 172.21.15.16:0/1863611010 >> 172.21.15.16:0/1863611010 conn(0x562a25a871c0 msgr2=0x562a25b0dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.599+0000 7f3055648b80 1 -- 172.21.15.16:0/1863611010 shutdown_connections 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.599+0000 7f3055648b80 1 -- 172.21.15.16:0/1863611010 wait complete. 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.600+0000 7f3055648b80 1 -- start start 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.600+0000 7f3055648b80 1 Processor -- start 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.600+0000 7f3055648b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b27cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.600+0000 7f3055648b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 0x562a25b24ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.600+0000 7f3055648b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b25670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.600+0000 7f3055648b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a25a63be0 con 0x562a25b21390 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.600+0000 7f3055648b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a259ffd20 con 0x562a25b1bdd0 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.600+0000 7f3055648b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a259ff9c0 con 0x562a25b21cb0 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.600+0000 7f3055648b80 10 client.0 _ll_register_callbacks cb 0x562a2586b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b27cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303a000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 0x562a25b24ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b25670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b27cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35814/0 (socket says 172.21.15.16:35814) 2024-04-05T23:42:32.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303aa00700 1 -- 172.21.15.16:0/169193325 learned_addr learned my addr 172.21.15.16:0/169193325 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303aa00700 1 -- 172.21.15.16:0/169193325 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 msgr2=0x562a25b25670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303aa00700 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b25670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303aa00700 1 -- 172.21.15.16:0/169193325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 msgr2=0x562a25b24ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303aa00700 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 0x562a25b24ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303aa00700 1 -- 172.21.15.16:0/169193325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c1a6580 con 0x562a25b1bdd0 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303b400700 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b25670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f303aa00700 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b27cb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f302c003900 tx=0x7f302c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=215908fab3556d6a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.601+0000 7f3024c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3024c00700 1 -- 172.21.15.16:0/169193325 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302c022070 con 0x562a25b1bdd0 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3055648b80 1 -- 172.21.15.16:0/169193325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562a25b061e0 con 0x562a25b1bdd0 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3024c00700 1 -- 172.21.15.16:0/169193325 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f302c027420 con 0x562a25b1bdd0 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3024c00700 1 -- 172.21.15.16:0/169193325 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f302c0275e0 con 0x562a25b1bdd0 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3055648b80 10 client.26576 fetch_fsmap learned FSMap version 60 2024-04-05T23:42:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3055648b80 1 -- 172.21.15.16:0/169193325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) v3 -- 0x562a25891b00 con 0x562a25b1bdd0 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3024c00700 1 -- 172.21.15.16:0/169193325 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562a25b061e0 con 0x562a25b1bdd0 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3024c00700 1 -- 172.21.15.16:0/169193325 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 60) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f302c01d850 con 0x562a25b1bdd0 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3055648b80 10 client.26576 fetch_fsmap finished waiting for FSMap version 60 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3055648b80 10 client.26576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3055648b80 20 client.26576 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3055648b80 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a25b35440 0x562a25b37860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3055648b80 4 client.26576 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.602+0000 7f3055648b80 1 -- 172.21.15.16:0/169193325 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a25a5c6a0 con 0x562a25b35440 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.603+0000 7f303a000700 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a25b35440 0x562a25b37860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.603+0000 7f303a000700 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a25b35440 0x562a25b37860 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba5b5aaa9dd082c7 server_cookie=3fb931eb3014b53d in_seq=0 out_seq=0 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.603+0000 7f3024c00700 10 client.26576 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3024c00700 1 -- 172.21.15.16:0/169193325 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562a25a5c6a0 con 0x562a25b35440 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3024c00700 10 client.26576 handle_command_reply: tid=0 2024-04-05T23:42:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 client.26576 shutdown 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 -- 172.21.15.16:0/169193325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 msgr2=0x562a25b27cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b27cb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f302c003900 tx=0x7f302c0047c0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 -- 172.21.15.16:0/169193325 shutdown_connections 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a25b35440 0x562a25b37860 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b25670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b1bdd0 0x562a25b27cb0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3024c00700 0 client.26576 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 --2- 172.21.15.16:0/169193325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b21390 0x562a25b24ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 -- 172.21.15.16:0/169193325 >> 172.21.15.16:0/169193325 conn(0x562a25a871c0 msgr2=0x562a25b0d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 -- 172.21.15.16:0/169193325 shutdown_connections 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.604+0000 7f3055648b80 1 -- 172.21.15.16:0/169193325 wait complete. 2024-04-05T23:42:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.605+0000 7f3055648b80 20 client.26576 trim_cache size 0 max 16384 2024-04-05T23:42:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.620+0000 7f3055648b80 1 Processor -- start 2024-04-05T23:42:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.620+0000 7f3055648b80 1 -- start start 2024-04-05T23:42:32.613 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f3055648b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 0x562a25b1b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.613 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f3055648b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21390 0x562a25b2c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.613 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f3055648b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21cb0 0x562a25b0da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.613 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f3055648b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a258e7640 con 0x562a25b1bdd0 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f3055648b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a25b1fc90 con 0x562a25b21cb0 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f3055648b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a25b26570 con 0x562a25b21390 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303a000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21390 0x562a25b2c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 0x562a25b1b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21cb0 0x562a25b0da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21cb0 0x562a25b0da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35822/0 (socket says 172.21.15.16:35822) 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303b400700 1 -- 172.21.15.16:0/2826608262 learned_addr learned my addr 172.21.15.16:0/2826608262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303a000700 1 -- 172.21.15.16:0/2826608262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21cb0 msgr2=0x562a25b0da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303a000700 1 --2- 172.21.15.16:0/2826608262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21cb0 0x562a25b0da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303a000700 1 -- 172.21.15.16:0/2826608262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 msgr2=0x562a25b1b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303a000700 1 --2- 172.21.15.16:0/2826608262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 0x562a25b1b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303a000700 1 -- 172.21.15.16:0/2826608262 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a25a29e40 con 0x562a25b21390 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.621+0000 7f303b400700 1 --2- 172.21.15.16:0/2826608262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21cb0 0x562a25b0da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.622+0000 7f303a000700 1 --2- 172.21.15.16:0/2826608262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21390 0x562a25b2c4d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x562a25b22110 tx=0x7f3028021c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da30bc3b826c4bff server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.622+0000 7f3030c00700 1 -- 172.21.15.16:0/2826608262 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302809a880 con 0x562a25b21390 2024-04-05T23:42:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.622+0000 7f3030c00700 1 -- 172.21.15.16:0/2826608262 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3028097260 con 0x562a25b21390 2024-04-05T23:42:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.622+0000 7f3030c00700 1 -- 172.21.15.16:0/2826608262 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302809a410 con 0x562a25b21390 2024-04-05T23:42:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 -- 172.21.15.16:0/2826608262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21390 msgr2=0x562a25b2c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 --2- 172.21.15.16:0/2826608262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21390 0x562a25b2c4d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x562a25b22110 tx=0x7f3028021c30 comp rx=0 tx=0).stop 2024-04-05T23:42:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 -- 172.21.15.16:0/2826608262 shutdown_connections 2024-04-05T23:42:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 --2- 172.21.15.16:0/2826608262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 0x562a25b1b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 --2- 172.21.15.16:0/2826608262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21cb0 0x562a25b0da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 --2- 172.21.15.16:0/2826608262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21390 0x562a25b2c4d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 -- 172.21.15.16:0/2826608262 >> 172.21.15.16:0/2826608262 conn(0x562a25a871c0 msgr2=0x562a25b238c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 -- 172.21.15.16:0/2826608262 shutdown_connections 2024-04-05T23:42:32.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 -- 172.21.15.16:0/2826608262 wait complete. 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 -- start start 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.623+0000 7f3055648b80 1 Processor -- start 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f3055648b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 0x562a25b284b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f3055648b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21390 0x562a25b289f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f3055648b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b88260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f3055648b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a25b29300 con 0x562a25b1bdd0 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f3055648b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a25b887a0 con 0x562a25b21390 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f3055648b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a25b888e0 con 0x562a25b21cb0 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f3055648b80 10 client.0 _ll_register_callbacks cb 0x562a2586b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f303aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 0x562a25b284b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f303a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21390 0x562a25b289f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f303a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21390 0x562a25b289f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35834/0 (socket says 172.21.15.16:35834) 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f303a000700 1 -- 172.21.15.16:0/988677925 learned_addr learned my addr 172.21.15.16:0/988677925 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.624+0000 7f303b400700 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b88260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f303a000700 1 -- 172.21.15.16:0/988677925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 msgr2=0x562a25b88260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f303a000700 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b88260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f303a000700 1 -- 172.21.15.16:0/988677925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 msgr2=0x562a25b284b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f303a000700 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 0x562a25b284b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f303a000700 1 -- 172.21.15.16:0/988677925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303400e740 con 0x562a25b21390 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f303a000700 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21390 0x562a25b289f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3028099270 tx=0x7f30280047a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97cab7301c58eb1e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f3024c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f3024c00700 1 -- 172.21.15.16:0/988677925 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302809a880 con 0x562a25b21390 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f3024c00700 1 -- 172.21.15.16:0/988677925 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f302809af20 con 0x562a25b21390 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f3055648b80 1 -- 172.21.15.16:0/988677925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f302809b620 con 0x562a25b21390 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f3024c00700 1 -- 172.21.15.16:0/988677925 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30280a3310 con 0x562a25b21390 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f3055648b80 10 client.26582 fetch_fsmap learned FSMap version 60 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.625+0000 7f3055648b80 1 -- 172.21.15.16:0/988677925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) v3 -- 0x562a258e5350 con 0x562a25b21390 2024-04-05T23:42:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.626+0000 7f3024c00700 1 -- 172.21.15.16:0/988677925 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f302809b620 con 0x562a25b21390 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.626+0000 7f3024c00700 1 -- 172.21.15.16:0/988677925 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 60) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f302809a410 con 0x562a25b21390 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.626+0000 7f3055648b80 10 client.26582 fetch_fsmap finished waiting for FSMap version 60 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.626+0000 7f3055648b80 10 client.26582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.626+0000 7f3055648b80 20 client.26582 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.626+0000 7f3055648b80 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a25b89cd0 0x562a25b8c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.626+0000 7f3055648b80 4 client.26582 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.626+0000 7f3055648b80 1 -- 172.21.15.16:0/988677925 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562a258b2c90 con 0x562a25b89cd0 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.626+0000 7f303aa00700 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a25b89cd0 0x562a25b8c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.627+0000 7f303aa00700 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a25b89cd0 0x562a25b8c0f0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d5533afb415779f server_cookie=80f30c5b2596731e in_seq=0 out_seq=0 2024-04-05T23:42:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.627+0000 7f3024c00700 10 client.26582 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3024c00700 1 -- 172.21.15.16:0/988677925 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562a258b2c90 con 0x562a25b89cd0 2024-04-05T23:42:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3024c00700 10 client.26582 handle_command_reply: tid=0 2024-04-05T23:42:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 client.26582 shutdown 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 -- 172.21.15.16:0/988677925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21390 msgr2=0x562a25b289f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21390 0x562a25b289f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3028099270 tx=0x7f30280047a0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 -- 172.21.15.16:0/988677925 shutdown_connections 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562a25b89cd0 0x562a25b8c0f0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a25b1bdd0 0x562a25b284b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a25b21cb0 0x562a25b88260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3024c00700 0 client.26582 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 --2- 172.21.15.16:0/988677925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a25b21390 0x562a25b289f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 -- 172.21.15.16:0/988677925 >> 172.21.15.16:0/988677925 conn(0x562a25a871c0 msgr2=0x562a25b30c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 -- 172.21.15.16:0/988677925 shutdown_connections 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 1 -- 172.21.15.16:0/988677925 wait complete. 2024-04-05T23:42:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.652+0000 7f3055648b80 20 client.26582 trim_cache size 0 max 16384 2024-04-05T23:42:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:42:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:42:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:42:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 -- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f30200a1310 msgr2=0x7f30200a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f30200a1310 0x7f30200a37d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f302c009c60 tx=0x7f302c009260 comp rx=0 tx=0).stop 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 -- 172.21.15.16:0/2278041133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c106150 msgr2=0x7f303c199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c106150 0x7f303c199a60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3028000c00 tx=0x7f3028004730 comp rx=0 tx=0).stop 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 -- 172.21.15.16:0/2278041133 shutdown_connections 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f30200a1310 0x7f30200a37d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303c101030 0x7f303c199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f303c19de00 0x7f303c19e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 --2- 172.21.15.16:0/2278041133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f303c106150 0x7f303c199a60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 -- 172.21.15.16:0/2278041133 >> 172.21.15.16:0/2278041133 conn(0x7f303c0fc8b0 msgr2=0x7f303c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 -- 172.21.15.16:0/2278041133 shutdown_connections 2024-04-05T23:42:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:32.653+0000 7f3041c00700 1 -- 172.21.15.16:0/2278041133 wait complete. 2024-04-05T23:42:32.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:32 smithi016 ceph-mon[38248]: pgmap v280: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-04-05T23:42:32.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:32 smithi130 ceph-mon[79417]: pgmap v280: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-04-05T23:42:32.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:32 smithi027 ceph-mon[78597]: pgmap v280: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-04-05T23:42:33.018 INFO:tasks.workunit.client.0.smithi016.stdout: 1 450883 5.07 MB/sec execute 352 sec latency 18.241 ms 2024-04-05T23:42:34.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 452187 5.07 MB/sec execute 353 sec latency 51.900 ms 2024-04-05T23:42:34.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:34 smithi016 ceph-mon[38248]: pgmap v281: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-04-05T23:42:34.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:34 smithi130 ceph-mon[79417]: pgmap v281: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-04-05T23:42:34.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:34 smithi027 ceph-mon[78597]: pgmap v281: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-04-05T23:42:35.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 452187 5.05 MB/sec execute 354 sec latency 1052.055 ms 2024-04-05T23:42:35.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:35 smithi016 ceph-mon[38248]: pgmap v282: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-04-05T23:42:35.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:35 smithi130 ceph-mon[79417]: pgmap v282: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-04-05T23:42:35.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:35 smithi027 ceph-mon[78597]: pgmap v282: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-04-05T23:42:36.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 454145 5.07 MB/sec execute 355 sec latency 1140.116 ms 2024-04-05T23:42:37.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 455106 5.07 MB/sec execute 356 sec latency 26.067 ms 2024-04-05T23:42:38.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 455730 5.06 MB/sec execute 357 sec latency 516.230 ms 2024-04-05T23:42:38.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:37 smithi016 ceph-mon[38248]: pgmap v283: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 188 op/s 2024-04-05T23:42:38.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:37 smithi130 ceph-mon[79417]: pgmap v283: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 188 op/s 2024-04-05T23:42:38.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:37 smithi027 ceph-mon[78597]: pgmap v283: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 188 op/s 2024-04-05T23:42:39.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 455730 5.05 MB/sec execute 358 sec latency 1516.330 ms 2024-04-05T23:42:40.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 455730 5.03 MB/sec execute 359 sec latency 2516.461 ms 2024-04-05T23:42:40.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:39 smithi016 ceph-mon[38248]: pgmap v284: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-04-05T23:42:40.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:39 smithi130 ceph-mon[79417]: pgmap v284: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-04-05T23:42:40.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:39 smithi027 ceph-mon[78597]: pgmap v284: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-04-05T23:42:41.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 456006 5.02 MB/sec execute 360 sec latency 3392.747 ms 2024-04-05T23:42:42.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 457846 5.04 MB/sec execute 361 sec latency 14.174 ms 2024-04-05T23:42:42.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:41 smithi016 ceph-mon[38248]: pgmap v285: 97 pgs: 97 active+clean; 167 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-04-05T23:42:42.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:41 smithi130 ceph-mon[79417]: pgmap v285: 97 pgs: 97 active+clean; 167 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-04-05T23:42:42.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:41 smithi027 ceph-mon[78597]: pgmap v285: 97 pgs: 97 active+clean; 167 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-04-05T23:42:43.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 458567 5.06 MB/sec execute 362 sec latency 47.534 ms 2024-04-05T23:42:44.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 458633 5.05 MB/sec execute 363 sec latency 933.875 ms 2024-04-05T23:42:44.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:43 smithi016 ceph-mon[38248]: pgmap v286: 97 pgs: 97 active+clean; 167 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:42:44.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:43 smithi130 ceph-mon[79417]: pgmap v286: 97 pgs: 97 active+clean; 167 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:42:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:43 smithi027 ceph-mon[78597]: pgmap v286: 97 pgs: 97 active+clean; 167 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:42:45.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 458633 5.03 MB/sec execute 364 sec latency 1934.003 ms 2024-04-05T23:42:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.815+0000 7ff053000700 1 -- 172.21.15.16:0/4000271708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c106160 msgr2=0x7ff04c10dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.815+0000 7ff053000700 1 --2- 172.21.15.16:0/4000271708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c106160 0x7ff04c10dce0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff044009f90 tx=0x7ff04400b330 comp rx=0 tx=0).stop 2024-04-05T23:42:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.815+0000 7ff053000700 1 -- 172.21.15.16:0/4000271708 shutdown_connections 2024-04-05T23:42:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.815+0000 7ff053000700 1 --2- 172.21.15.16:0/4000271708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04c101040 0x7ff04c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.815+0000 7ff053000700 1 --2- 172.21.15.16:0/4000271708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c106160 0x7ff04c10dce0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.815+0000 7ff053000700 1 --2- 172.21.15.16:0/4000271708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff04c1019f0 0x7ff04c105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.815+0000 7ff053000700 1 -- 172.21.15.16:0/4000271708 >> 172.21.15.16:0/4000271708 conn(0x7ff04c0fc8e0 msgr2=0x7ff04c0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.815+0000 7ff053000700 1 -- 172.21.15.16:0/4000271708 shutdown_connections 2024-04-05T23:42:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.815+0000 7ff053000700 1 -- 172.21.15.16:0/4000271708 wait complete. 2024-04-05T23:42:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.817+0000 7ff053000700 1 Processor -- start 2024-04-05T23:42:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.817+0000 7ff053000700 1 -- start start 2024-04-05T23:42:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.817+0000 7ff053000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c101040 0x7ff04c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.817+0000 7ff053000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff04c106160 0x7ff04c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.817+0000 7ff053000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04c19dec0 0x7ff04c19e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.817+0000 7ff053000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c19e940 con 0x7ff04c19dec0 2024-04-05T23:42:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.817+0000 7ff053000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c19eab0 con 0x7ff04c106160 2024-04-05T23:42:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.817+0000 7ff053000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c19ec70 con 0x7ff04c101040 2024-04-05T23:42:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff04c106160 0x7ff04c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c101040 0x7ff04c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c101040 0x7ff04c1995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:39098/0 (socket says 172.21.15.16:39098) 2024-04-05T23:42:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff051000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04c19dec0 0x7ff04c19e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff04c106160 0x7ff04c199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59022/0 (socket says 172.21.15.16:59022) 2024-04-05T23:42:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04be00700 1 -- 172.21.15.16:0/981262689 learned_addr learned my addr 172.21.15.16:0/981262689 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04b400700 1 -- 172.21.15.16:0/981262689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c101040 msgr2=0x7ff04c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04b400700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c101040 0x7ff04c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04b400700 1 -- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04c19dec0 msgr2=0x7ff04c19e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04b400700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04c19dec0 0x7ff04c19e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04b400700 1 -- 172.21.15.16:0/981262689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff044009c40 con 0x7ff04c106160 2024-04-05T23:42:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff051000700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04c19dec0 0x7ff04c19e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.818+0000 7ff04be00700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c101040 0x7ff04c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.819+0000 7ff04b400700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff04c106160 0x7ff04c199b20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff03800e9c0 tx=0x7ff03800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.820+0000 7ff048c00700 1 -- 172.21.15.16:0/981262689 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03800c9c0 con 0x7ff04c106160 2024-04-05T23:42:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.820+0000 7ff048c00700 1 -- 172.21.15.16:0/981262689 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff038010430 con 0x7ff04c106160 2024-04-05T23:42:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.820+0000 7ff053000700 1 -- 172.21.15.16:0/981262689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04c1a61d0 con 0x7ff04c106160 2024-04-05T23:42:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.820+0000 7ff048c00700 1 -- 172.21.15.16:0/981262689 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038018680 con 0x7ff04c106160 2024-04-05T23:42:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.821+0000 7ff053000700 1 -- 172.21.15.16:0/981262689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c1a6700 con 0x7ff04c106160 2024-04-05T23:42:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.823+0000 7ff048c00700 1 -- 172.21.15.16:0/981262689 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7ff03800cb20 con 0x7ff04c106160 2024-04-05T23:42:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.825+0000 7ff048c00700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0300a1310 0x7ff0300a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.825+0000 7ff048c00700 1 -- 172.21.15.16:0/981262689 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7ff038014070 con 0x7ff04c106160 2024-04-05T23:42:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.825+0000 7ff04be00700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0300a1310 0x7ff0300a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.825+0000 7ff066a79b80 1 Processor -- start 2024-04-05T23:42:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.825+0000 7ff066a79b80 1 -- start start 2024-04-05T23:42:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 0x556e93b35f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b30030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b305c0 0x556e93b30a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff066a79b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93a96b80 con 0x556e93b35b80 2024-04-05T23:42:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff066a79b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93a87930 con 0x556e93b364a0 2024-04-05T23:42:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff066a79b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93a852f0 con 0x556e93b305c0 2024-04-05T23:42:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff04be00700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0300a1310 0x7ff0300a37d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0114f0 tx=0x7ff03c0094a0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff04b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b30030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff04b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b30030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59024/0 (socket says 172.21.15.16:59024) 2024-04-05T23:42:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff04b400700 1 -- 172.21.15.16:0/3714871878 learned_addr learned my addr 172.21.15.16:0/3714871878 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff051000700 1 --2- 172.21.15.16:0/3714871878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b305c0 0x556e93b30a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.826+0000 7ff04be00700 1 --2- 172.21.15.16:0/3714871878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 0x556e93b35f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.827+0000 7ff04b400700 1 -- 172.21.15.16:0/3714871878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b305c0 msgr2=0x556e93b30a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.827+0000 7ff04b400700 1 --2- 172.21.15.16:0/3714871878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b305c0 0x556e93b30a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.827+0000 7ff04b400700 1 -- 172.21.15.16:0/3714871878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 msgr2=0x556e93b35f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.827+0000 7ff04b400700 1 --2- 172.21.15.16:0/3714871878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 0x556e93b35f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.827+0000 7ff04b400700 1 -- 172.21.15.16:0/3714871878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e93902350 con 0x556e93b364a0 2024-04-05T23:42:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.827+0000 7ff04b400700 1 --2- 172.21.15.16:0/3714871878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b30030 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff0380a1d80 tx=0x7ff038099a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48083ba688e8c21d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.827+0000 7ff041600700 1 -- 172.21.15.16:0/3714871878 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038099e30 con 0x556e93b364a0 2024-04-05T23:42:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.827+0000 7ff041600700 1 -- 172.21.15.16:0/3714871878 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff03809dc60 con 0x556e93b364a0 2024-04-05T23:42:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.827+0000 7ff041600700 1 -- 172.21.15.16:0/3714871878 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03809a090 con 0x556e93b364a0 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.828+0000 7ff066a79b80 1 -- 172.21.15.16:0/3714871878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 msgr2=0x556e93b30030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.828+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3714871878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b30030 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff0380a1d80 tx=0x7ff038099a70 comp rx=0 tx=0).stop 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.828+0000 7ff066a79b80 1 -- 172.21.15.16:0/3714871878 shutdown_connections 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.828+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3714871878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 0x556e93b35f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.828+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3714871878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b305c0 0x556e93b30a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.828+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3714871878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b30030 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.828+0000 7ff066a79b80 1 -- 172.21.15.16:0/3714871878 >> 172.21.15.16:0/3714871878 conn(0x556e93aa41c0 msgr2=0x556e93b2abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.828+0000 7ff066a79b80 1 -- 172.21.15.16:0/3714871878 shutdown_connections 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.828+0000 7ff066a79b80 1 -- 172.21.15.16:0/3714871878 wait complete. 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.829+0000 7ff066a79b80 1 -- start start 2024-04-05T23:42:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.829+0000 7ff066a79b80 1 Processor -- start 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b39900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b35b80 0x556e93b39e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 0x556e93b3a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff066a79b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93a80be0 con 0x556e93b305c0 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff066a79b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93a1cd20 con 0x556e93b35b80 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff066a79b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93a1c9c0 con 0x556e93b364a0 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff066a79b80 10 client.0 _ll_register_callbacks cb 0x556e93888340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff04b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b35b80 0x556e93b39e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff04be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b39900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff04b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b35b80 0x556e93b39e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59032/0 (socket says 172.21.15.16:59032) 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff04b400700 1 -- 172.21.15.16:0/2576492935 learned_addr learned my addr 172.21.15.16:0/2576492935 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff051000700 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 0x556e93b3a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff04b400700 1 -- 172.21.15.16:0/2576492935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 msgr2=0x556e93b3a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff04b400700 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 0x556e93b3a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff04b400700 1 -- 172.21.15.16:0/2576492935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 msgr2=0x556e93b39900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff04b400700 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b39900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.830+0000 7ff04b400700 1 -- 172.21.15.16:0/2576492935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c1a6700 con 0x556e93b35b80 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff051000700 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 0x556e93b3a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff04be00700 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b39900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff04b400700 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b35b80 0x556e93b39e40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff038099dc0 tx=0x7ff0380a2360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cf4c1313d9c9c93 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff036000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff036000700 1 -- 172.21.15.16:0/2576492935 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0380b6070 con 0x556e93b35b80 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff036000700 1 -- 172.21.15.16:0/2576492935 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7ff03809ae40 con 0x556e93b35b80 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff066a79b80 1 -- 172.21.15.16:0/2576492935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556e93b231e0 con 0x556e93b35b80 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff036000700 1 -- 172.21.15.16:0/2576492935 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0380105a0 con 0x556e93b35b80 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff066a79b80 10 client.26594 fetch_fsmap learned FSMap version 61 2024-04-05T23:42:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.831+0000 7ff066a79b80 1 -- 172.21.15.16:0/2576492935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) v3 -- 0x556e938aeb00 con 0x556e93b35b80 2024-04-05T23:42:45.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.832+0000 7ff036000700 1 -- 172.21.15.16:0/2576492935 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556e93b231e0 con 0x556e93b35b80 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.832+0000 7ff036000700 1 -- 172.21.15.16:0/2576492935 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 61) v1 ==== 7478+0+0 (secure 0 0 0) 0x7ff038081020 con 0x556e93b35b80 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.832+0000 7ff066a79b80 10 client.26594 fetch_fsmap finished waiting for FSMap version 61 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.832+0000 7ff066a79b80 10 client.26594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.832+0000 7ff066a79b80 20 client.26594 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.832+0000 7ff066a79b80 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e93b49b50 0x556e93b4bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.832+0000 7ff066a79b80 4 client.26594 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.832+0000 7ff066a79b80 1 -- 172.21.15.16:0/2576492935 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556e93a796a0 con 0x556e93b49b50 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.832+0000 7ff04be00700 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e93b49b50 0x556e93b4bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.833+0000 7ff04be00700 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e93b49b50 0x556e93b4bf40 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13cfef58a195cf0e server_cookie=351ccfed18bc21f in_seq=0 out_seq=0 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.833+0000 7ff036000700 10 client.26594 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.833+0000 7ff036000700 1 -- 172.21.15.16:0/2576492935 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556e93a796a0 con 0x556e93b49b50 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.833+0000 7ff036000700 10 client.26594 handle_command_reply: tid=0 2024-04-05T23:42:45.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.833+0000 7ff066a79b80 1 client.26594 shutdown 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 -- 172.21.15.16:0/2576492935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b35b80 msgr2=0x556e93b39e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b35b80 0x556e93b39e40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff038099dc0 tx=0x7ff0380a2360 comp rx=0 tx=0).stop 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 -- 172.21.15.16:0/2576492935 shutdown_connections 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e93b49b50 0x556e93b4bf40 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b39900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 0x556e93b3a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 --2- 172.21.15.16:0/2576492935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b35b80 0x556e93b39e40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 -- 172.21.15.16:0/2576492935 >> 172.21.15.16:0/2576492935 conn(0x556e93aa41c0 msgr2=0x556e93b2a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff036000700 0 client.26594 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 -- 172.21.15.16:0/2576492935 shutdown_connections 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 1 -- 172.21.15.16:0/2576492935 wait complete. 2024-04-05T23:42:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.834+0000 7ff066a79b80 20 client.26594 trim_cache size 0 max 16384 2024-04-05T23:42:45.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.851+0000 7ff066a79b80 1 Processor -- start 2024-04-05T23:42:45.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.851+0000 7ff066a79b80 1 -- start start 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b30030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b35b80 0x556e93b2aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b39e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff066a79b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93904640 con 0x556e93b305c0 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff066a79b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93b2c620 con 0x556e93b364a0 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff066a79b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93b3ca30 con 0x556e93b35b80 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff051000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b39e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff051000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b39e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59042/0 (socket says 172.21.15.16:59042) 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff04be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b30030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff051000700 1 -- 172.21.15.16:0/1715227048 learned_addr learned my addr 172.21.15.16:0/1715227048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff04b400700 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b35b80 0x556e93b2aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.846 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff04b400700 1 -- 172.21.15.16:0/1715227048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 msgr2=0x556e93b39e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:45.846 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff04b400700 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b39e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.846 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff04b400700 1 -- 172.21.15.16:0/1715227048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 msgr2=0x556e93b30030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:45.846 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff04b400700 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b30030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.846 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff04b400700 1 -- 172.21.15.16:0/1715227048 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e93a46e40 con 0x556e93b35b80 2024-04-05T23:42:45.846 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff051000700 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b39e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:45.846 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.852+0000 7ff04be00700 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b30030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.853+0000 7ff04b400700 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b35b80 0x556e93b2aa00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff03800f010 tx=0x7ff038096220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94bf759e788a7f70 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.853+0000 7ff042000700 1 -- 172.21.15.16:0/1715227048 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038083420 con 0x556e93b35b80 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.853+0000 7ff042000700 1 -- 172.21.15.16:0/1715227048 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff03807e070 con 0x556e93b35b80 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.853+0000 7ff042000700 1 -- 172.21.15.16:0/1715227048 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038004750 con 0x556e93b35b80 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 -- 172.21.15.16:0/1715227048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b35b80 msgr2=0x556e93b2aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b35b80 0x556e93b2aa00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff03800f010 tx=0x7ff038096220 comp rx=0 tx=0).stop 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 -- 172.21.15.16:0/1715227048 shutdown_connections 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b305c0 0x556e93b30030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b364a0 0x556e93b39e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 --2- 172.21.15.16:0/1715227048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b35b80 0x556e93b2aa00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 -- 172.21.15.16:0/1715227048 >> 172.21.15.16:0/1715227048 conn(0x556e93aa41c0 msgr2=0x556e93b339e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 -- 172.21.15.16:0/1715227048 shutdown_connections 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 -- 172.21.15.16:0/1715227048 wait complete. 2024-04-05T23:42:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 -- start start 2024-04-05T23:42:45.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.854+0000 7ff066a79b80 1 Processor -- start 2024-04-05T23:42:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b305c0 0x556e93b410f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 0x556e93b3eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff066a79b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 0x556e93b3f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff066a79b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93b3fd30 con 0x556e93b35b80 2024-04-05T23:42:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff066a79b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93b9c970 con 0x556e93b305c0 2024-04-05T23:42:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff066a79b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e93b9cab0 con 0x556e93b364a0 2024-04-05T23:42:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff066a79b80 10 client.0 _ll_register_callbacks cb 0x7ff038005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff04be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b305c0 0x556e93b410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff04be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b305c0 0x556e93b410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59044/0 (socket says 172.21.15.16:59044) 2024-04-05T23:42:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff04be00700 1 -- 172.21.15.16:0/3619846759 learned_addr learned my addr 172.21.15.16:0/3619846759 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff04b400700 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 0x556e93b3eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.855+0000 7ff051000700 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 0x556e93b3f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff04be00700 1 -- 172.21.15.16:0/3619846759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 msgr2=0x556e93b3f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff04be00700 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 0x556e93b3f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff04be00700 1 -- 172.21.15.16:0/3619846759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 msgr2=0x556e93b3eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff04be00700 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 0x556e93b3eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff04be00700 1 -- 172.21.15.16:0/3619846759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03c012530 con 0x556e93b305c0 2024-04-05T23:42:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff04b400700 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 0x556e93b3eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:42:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff04be00700 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b305c0 0x556e93b410f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff03c011c40 tx=0x7ff03c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14d542bf26cda0f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff036000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff036000700 1 -- 172.21.15.16:0/3619846759 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c025040 con 0x556e93b305c0 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff036000700 1 -- 172.21.15.16:0/3619846759 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7ff03c018070 con 0x556e93b305c0 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.856+0000 7ff066a79b80 1 -- 172.21.15.16:0/3619846759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff03809dc60 con 0x556e93b305c0 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff036000700 1 -- 172.21.15.16:0/3619846759 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff03c012d30 con 0x556e93b305c0 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff066a79b80 10 client.26600 fetch_fsmap learned FSMap version 61 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff066a79b80 1 -- 172.21.15.16:0/3619846759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) v3 -- 0x7ff04c10e550 con 0x556e93b305c0 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff036000700 1 -- 172.21.15.16:0/3619846759 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff03809dc60 con 0x556e93b305c0 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff036000700 1 -- 172.21.15.16:0/3619846759 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 61) v1 ==== 7478+0+0 (secure 0 0 0) 0x7ff03c003ae0 con 0x556e93b305c0 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff066a79b80 10 client.26600 fetch_fsmap finished waiting for FSMap version 61 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff066a79b80 10 client.26600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff066a79b80 20 client.26600 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:45.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e93b9e270 0x556e93ba0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:45.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff066a79b80 4 client.26600 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:42:45.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff066a79b80 1 -- 172.21.15.16:0/3619846759 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556e938cfc90 con 0x556e93b9e270 2024-04-05T23:42:45.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.857+0000 7ff04b400700 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e93b9e270 0x556e93ba0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:45.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.858+0000 7ff04b400700 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e93b9e270 0x556e93ba0690 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c086991d8745b100 server_cookie=63a8217557ba3949 in_seq=0 out_seq=0 2024-04-05T23:42:45.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.858+0000 7ff036000700 10 client.26600 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.892+0000 7ff036000700 1 -- 172.21.15.16:0/3619846759 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556e938cfc90 con 0x556e93b9e270 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.892+0000 7ff036000700 10 client.26600 handle_command_reply: tid=0 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.892+0000 7ff066a79b80 1 client.26600 shutdown 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 -- 172.21.15.16:0/3619846759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b305c0 msgr2=0x556e93b410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b305c0 0x556e93b410f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff03c011c40 tx=0x7ff03c0047c0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 -- 172.21.15.16:0/3619846759 shutdown_connections 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x556e93b9e270 0x556e93ba0690 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e93b364a0 0x556e93b3f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e93b305c0 0x556e93b410f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 --2- 172.21.15.16:0/3619846759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556e93b35b80 0x556e93b3eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff036000700 0 client.26600 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 -- 172.21.15.16:0/3619846759 >> 172.21.15.16:0/3619846759 conn(0x556e93aa41c0 msgr2=0x556e93b38150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 -- 172.21.15.16:0/3619846759 shutdown_connections 2024-04-05T23:42:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 1 -- 172.21.15.16:0/3619846759 wait complete. 2024-04-05T23:42:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.893+0000 7ff066a79b80 20 client.26600 trim_cache size 0 max 16384 2024-04-05T23:42:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:42:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:42:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:42:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:42:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 -- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0300a1310 msgr2=0x7ff0300a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0300a1310 0x7ff0300a37d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0114f0 tx=0x7ff03c0094a0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 -- 172.21.15.16:0/981262689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff04c106160 msgr2=0x7ff04c199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff04c106160 0x7ff04c199b20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff03800e9c0 tx=0x7ff03800ecd0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 -- 172.21.15.16:0/981262689 shutdown_connections 2024-04-05T23:42:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0300a1310 0x7ff0300a37d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04c19dec0 0x7ff04c19e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff04c101040 0x7ff04c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 --2- 172.21.15.16:0/981262689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff04c106160 0x7ff04c199b20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 -- 172.21.15.16:0/981262689 >> 172.21.15.16:0/981262689 conn(0x7ff04c0fc8e0 msgr2=0x7ff04c0fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 -- 172.21.15.16:0/981262689 shutdown_connections 2024-04-05T23:42:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:45.894+0000 7ff053000700 1 -- 172.21.15.16:0/981262689 wait complete. 2024-04-05T23:42:46.019 INFO:tasks.workunit.client.0.smithi016.stdout: 1 459747 5.06 MB/sec execute 365 sec latency 2013.514 ms 2024-04-05T23:42:46.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:45 smithi016 ceph-mon[38248]: pgmap v287: 97 pgs: 97 active+clean; 167 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:42:46.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:45 smithi130 ceph-mon[79417]: pgmap v287: 97 pgs: 97 active+clean; 167 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:42:46.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:45 smithi027 ceph-mon[78597]: pgmap v287: 97 pgs: 97 active+clean; 167 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:42:47.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460255 5.05 MB/sec execute 366 sec latency 483.798 ms 2024-04-05T23:42:47.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:46 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:47.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:46 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:47.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:46 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:48.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460255 5.04 MB/sec execute 367 sec latency 1172.547 ms 2024-04-05T23:42:48.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:47 smithi016 ceph-mon[38248]: pgmap v288: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-05T23:42:48.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:47 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:48.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:47 smithi130 ceph-mon[79417]: pgmap v288: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-05T23:42:48.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:47 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:48.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:47 smithi027 ceph-mon[78597]: pgmap v288: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-05T23:42:48.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:47 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:49.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460255 5.02 MB/sec execute 368 sec latency 2172.688 ms 2024-04-05T23:42:49.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:48 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:49.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:48 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:49.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:48 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:50.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460255 5.01 MB/sec execute 369 sec latency 3172.755 ms 2024-04-05T23:42:50.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:49 smithi016 ceph-mon[38248]: pgmap v289: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-05T23:42:50.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:49 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:42:50.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:49 smithi130 ceph-mon[79417]: pgmap v289: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-05T23:42:50.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:49 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:42:50.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:49 smithi027 ceph-mon[78597]: pgmap v289: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-05T23:42:50.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:49 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:42:51.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460255 4.99 MB/sec execute 370 sec latency 4172.818 ms 2024-04-05T23:42:51.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:50 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:42:51.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:50 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:42:51.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:50 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:42:51.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:50 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:42:51.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:50 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:42:51.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:50 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:42:51.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:50 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:42:51.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:50 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:42:51.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:50 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:42:52.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460805 4.99 MB/sec execute 371 sec latency 4895.948 ms 2024-04-05T23:42:52.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:52 smithi016 ceph-mon[38248]: pgmap v290: 97 pgs: 97 active+clean; 171 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-05T23:42:52.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:52 smithi130 ceph-mon[79417]: pgmap v290: 97 pgs: 97 active+clean; 171 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-05T23:42:52.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:52 smithi027 ceph-mon[78597]: pgmap v290: 97 pgs: 97 active+clean; 171 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-05T23:42:53.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462112 5.00 MB/sec execute 372 sec latency 220.612 ms 2024-04-05T23:42:53.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:53 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:53.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:53 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:53.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:53 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:54.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462112 4.99 MB/sec execute 373 sec latency 1157.659 ms 2024-04-05T23:42:54.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:54 smithi016 ceph-mon[38248]: pgmap v291: 97 pgs: 97 active+clean; 171 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:42:54.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:54 smithi130 ceph-mon[79417]: pgmap v291: 97 pgs: 97 active+clean; 171 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:42:54.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:54 smithi027 ceph-mon[78597]: pgmap v291: 97 pgs: 97 active+clean; 171 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:42:55.020 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462112 4.98 MB/sec execute 374 sec latency 2157.772 ms 2024-04-05T23:42:56.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462112 4.96 MB/sec execute 375 sec latency 3157.907 ms 2024-04-05T23:42:56.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:56 smithi016 ceph-mon[38248]: pgmap v292: 97 pgs: 97 active+clean; 171 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:42:56.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:56 smithi130 ceph-mon[79417]: pgmap v292: 97 pgs: 97 active+clean; 171 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:42:56.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:56 smithi027 ceph-mon[78597]: pgmap v292: 97 pgs: 97 active+clean; 171 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:42:57.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462112 4.95 MB/sec execute 376 sec latency 4158.064 ms 2024-04-05T23:42:57.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:57 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:57.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:57 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:57.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:57 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:42:58.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462228 4.94 MB/sec execute 377 sec latency 4763.110 ms 2024-04-05T23:42:58.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:42:58 smithi016 ceph-mon[38248]: pgmap v293: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-05T23:42:58.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:42:58 smithi130 ceph-mon[79417]: pgmap v293: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-05T23:42:58.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:42:58 smithi027 ceph-mon[78597]: pgmap v293: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-05T23:42:59.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462228 4.92 MB/sec execute 378 sec latency 1200.493 ms 2024-04-05T23:42:59.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.057+0000 7f44ea800700 1 -- 172.21.15.16:0/52140944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 msgr2=0x7f44e410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:59.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.057+0000 7f44ea800700 1 --2- 172.21.15.16:0/52140944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 0x7f44e410dcd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f44dc009f90 tx=0x7f44dc00b330 comp rx=0 tx=0).stop 2024-04-05T23:42:59.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.057+0000 7f44ea800700 1 -- 172.21.15.16:0/52140944 shutdown_connections 2024-04-05T23:42:59.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.057+0000 7f44ea800700 1 --2- 172.21.15.16:0/52140944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 0x7f44e410dcd0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.057+0000 7f44ea800700 1 --2- 172.21.15.16:0/52140944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44e4101030 0x7f44e4101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.057+0000 7f44ea800700 1 --2- 172.21.15.16:0/52140944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44e41019e0 0x7f44e4105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.057+0000 7f44ea800700 1 -- 172.21.15.16:0/52140944 >> 172.21.15.16:0/52140944 conn(0x7f44e40fc8b0 msgr2=0x7f44e40fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:59.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.057+0000 7f44ea800700 1 -- 172.21.15.16:0/52140944 shutdown_connections 2024-04-05T23:42:59.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.057+0000 7f44ea800700 1 -- 172.21.15.16:0/52140944 wait complete. 2024-04-05T23:42:59.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.058+0000 7f44ea800700 1 Processor -- start 2024-04-05T23:42:59.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.058+0000 7f44ea800700 1 -- start start 2024-04-05T23:42:59.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.058+0000 7f44ea800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44e4101030 0x7f44e40731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.058+0000 7f44ea800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 0x7f44e4073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.058+0000 7f44ea800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44e4073c60 0x7f44e41a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.058+0000 7f44ea800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e41a5c10 con 0x7f44e4106150 2024-04-05T23:42:59.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.058+0000 7f44ea800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e41a5d80 con 0x7f44e4073c60 2024-04-05T23:42:59.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.058+0000 7f44ea800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e41a5f40 con 0x7f44e4101030 2024-04-05T23:42:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 0x7f44e4073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 0x7f44e4073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47262/0 (socket says 172.21.15.16:47262) 2024-04-05T23:42:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e2a00700 1 -- 172.21.15.16:0/2651326190 learned_addr learned my addr 172.21.15.16:0/2651326190 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e3400700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44e4101030 0x7f44e40731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e3e00700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44e4073c60 0x7f44e41a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e2a00700 1 -- 172.21.15.16:0/2651326190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44e4101030 msgr2=0x7f44e40731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:59.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e2a00700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44e4101030 0x7f44e40731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e2a00700 1 -- 172.21.15.16:0/2651326190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44e4073c60 msgr2=0x7f44e41a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:59.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e2a00700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44e4073c60 0x7f44e41a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e2a00700 1 -- 172.21.15.16:0/2651326190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc009c40 con 0x7f44e4106150 2024-04-05T23:42:59.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e3e00700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44e4073c60 0x7f44e41a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:42:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.059+0000 7f44e2a00700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 0x7f44e4073720 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f44d000e9f0 tx=0x7f44d000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.060+0000 7f44dbe00700 1 -- 172.21.15.16:0/2651326190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0009690 con 0x7f44e4106150 2024-04-05T23:42:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.060+0000 7f44dbe00700 1 -- 172.21.15.16:0/2651326190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44d000ce80 con 0x7f44e4106150 2024-04-05T23:42:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.060+0000 7f44ea800700 1 -- 172.21.15.16:0/2651326190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e41a6220 con 0x7f44e4106150 2024-04-05T23:42:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.060+0000 7f44dbe00700 1 -- 172.21.15.16:0/2651326190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0018680 con 0x7f44e4106150 2024-04-05T23:42:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.060+0000 7f44ea800700 1 -- 172.21.15.16:0/2651326190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e41a6730 con 0x7f44e4106150 2024-04-05T23:42:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.063+0000 7f44dbe00700 1 -- 172.21.15.16:0/2651326190 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f44d0010430 con 0x7f44e4106150 2024-04-05T23:42:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.064+0000 7f44dbe00700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f44c80a1290 0x7f44c80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.064+0000 7f44fe37cb80 1 Processor -- start 2024-04-05T23:42:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.064+0000 7f44dbe00700 1 -- 172.21.15.16:0/2651326190 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f44d0014070 con 0x7f44e4106150 2024-04-05T23:42:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.064+0000 7f44fe37cb80 1 -- start start 2024-04-05T23:42:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44e3400700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f44c80a1290 0x7f44c80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f68259d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f682597840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6824f5b80 con 0x55f682597d80 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6824e6930 con 0x55f68259dc90 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6824e42f0 con 0x55f68259d370 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f682597840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44e3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f682597840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42962/0 (socket says 172.21.15.16:42962) 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44e2a00700 1 -- 172.21.15.16:0/1715440216 learned_addr learned my addr 172.21.15.16:0/1715440216 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.065+0000 7f44e3400700 1 --2- 172.21.15.16:0/1715440216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f68259d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44e2a00700 1 -- 172.21.15.16:0/1715440216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 msgr2=0x55f68259d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44e2a00700 1 --2- 172.21.15.16:0/1715440216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f68259d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44e3400700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f44c80a1290 0x7f44c80a3750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009cb0 tx=0x7f44d4009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44e2a00700 1 -- 172.21.15.16:0/1715440216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 msgr2=0x55f6825981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44e2a00700 1 --2- 172.21.15.16:0/1715440216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44e2a00700 1 -- 172.21.15.16:0/1715440216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f682361350 con 0x55f68259dc90 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44e2a00700 1 --2- 172.21.15.16:0/1715440216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f682597840 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f44d0096680 tx=0x7f44d0099bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49528b0a6f03e258 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44d8c00700 1 -- 172.21.15.16:0/1715440216 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d009dfb0 con 0x55f68259dc90 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44d8c00700 1 -- 172.21.15.16:0/1715440216 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44d00a2600 con 0x55f68259dc90 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.066+0000 7f44d8c00700 1 -- 172.21.15.16:0/1715440216 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d00998d0 con 0x55f68259dc90 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.067+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1715440216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 msgr2=0x55f682597840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.067+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1715440216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f682597840 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f44d0096680 tx=0x7f44d0099bb0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.067+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1715440216 shutdown_connections 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.067+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1715440216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.067+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1715440216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f68259d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.067+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1715440216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f682597840 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.067+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1715440216 >> 172.21.15.16:0/1715440216 conn(0x55f6825031c0 msgr2=0x55f682589be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.067+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1715440216 shutdown_connections 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.067+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1715440216 wait complete. 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.068+0000 7f44fe37cb80 1 -- start start 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.068+0000 7f44fe37cb80 1 Processor -- start 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.068+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.068+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f6825a1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.068+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f6825a1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.068+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6824dfbe0 con 0x55f682597d80 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.068+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f68247bd20 con 0x55f68259dc90 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.068+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f68247b9c0 con 0x55f68259d370 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.068+0000 7f44fe37cb80 10 client.0 _ll_register_callbacks cb 0x55f6822e7340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:59.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47282/0 (socket says 172.21.15.16:47282) 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3400700 1 -- 172.21.15.16:0/1825750726 learned_addr learned my addr 172.21.15.16:0/1825750726 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e2a00700 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f6825a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3e00700 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f6825a1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3400700 1 -- 172.21.15.16:0/1825750726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 msgr2=0x55f6825a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3400700 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f6825a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3400700 1 -- 172.21.15.16:0/1825750726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 msgr2=0x55f6825a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3400700 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f6825a1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3400700 1 -- 172.21.15.16:0/1825750726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e41a6730 con 0x55f682597d80 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44e3400700 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a1010 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f44d401c970 tx=0x7f44d40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dceadf7e0c458784 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.069+0000 7f44cd600700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:42:59.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44cd600700 1 -- 172.21.15.16:0/1825750726 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4021070 con 0x55f682597d80 2024-04-05T23:42:59.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1825750726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f6825821e0 con 0x55f682597d80 2024-04-05T23:42:59.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44cd600700 1 -- 172.21.15.16:0/1825750726 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f44d401b9f0 con 0x55f682597d80 2024-04-05T23:42:59.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44cd600700 1 -- 172.21.15.16:0/1825750726 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44d401bbb0 con 0x55f682597d80 2024-04-05T23:42:59.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44fe37cb80 10 client.16563 fetch_fsmap learned FSMap version 65 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1825750726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) v3 -- 0x55f68230db00 con 0x55f682597d80 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44cd600700 1 -- 172.21.15.16:0/1825750726 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44d4032020 con 0x55f682597d80 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44cd600700 1 -- 172.21.15.16:0/1825750726 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 65) v1 ==== 7514+0+0 (secure 0 0 0) 0x7f44d400f3c0 con 0x55f682597d80 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44fe37cb80 10 client.16563 fetch_fsmap finished waiting for FSMap version 65 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44fe37cb80 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44fe37cb80 20 client.16563 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f6825b17a0 0x55f6825b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44fe37cb80 4 client.16563 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.070+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1825750726 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f6824d86a0 con 0x55f6825b17a0 2024-04-05T23:42:59.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.071+0000 7f44e2a00700 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f6825b17a0 0x55f6825b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.071+0000 7f44e2a00700 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f6825b17a0 0x55f6825b3bc0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=433e425e4bf694be server_cookie=f7ba4250ebca6ea8 in_seq=0 out_seq=0 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.071+0000 7f44cd600700 10 client.16563 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.072+0000 7f44cd600700 1 -- 172.21.15.16:0/1825750726 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f6824d86a0 con 0x55f6825b17a0 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.072+0000 7f44cd600700 10 client.16563 handle_command_reply: tid=0 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.072+0000 7f44fe37cb80 1 client.16563 shutdown 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.072+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1825750726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 msgr2=0x55f6825a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.072+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a1010 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f44d401c970 tx=0x7f44d40047c0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1825750726 shutdown_connections 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f6825b17a0 0x55f6825b3bc0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a1010 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f68259dc90 0x55f6825a1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/1825750726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f6825a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1825750726 >> 172.21.15.16:0/1825750726 conn(0x55f6825031c0 msgr2=0x55f682589280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44cd600700 0 client.16563 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1825750726 shutdown_connections 2024-04-05T23:42:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44fe37cb80 1 -- 172.21.15.16:0/1825750726 wait complete. 2024-04-05T23:42:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.073+0000 7f44fe37cb80 20 client.16563 trim_cache size 0 max 16384 2024-04-05T23:42:59.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.088+0000 7f44fe37cb80 1 Processor -- start 2024-04-05T23:42:59.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44fe37cb80 1 -- start start 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6825031c0 0x55f6825a8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f682597d80 0x55f6825a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f682589a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f682363640 con 0x55f6825031c0 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6825a2680 con 0x55f682597d80 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f682586aa0 con 0x55f68259d370 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44e3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6825031c0 0x55f6825a8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44e3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6825031c0 0x55f6825a8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47298/0 (socket says 172.21.15.16:47298) 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44e3400700 1 -- 172.21.15.16:0/2647774318 learned_addr learned my addr 172.21.15.16:0/2647774318 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.089+0000 7f44e2a00700 1 --2- 172.21.15.16:0/2647774318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f682597d80 0x55f6825a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44e3e00700 1 --2- 172.21.15.16:0/2647774318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f682589a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44e3400700 1 -- 172.21.15.16:0/2647774318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 msgr2=0x55f682589a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44e3400700 1 --2- 172.21.15.16:0/2647774318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f682589a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44e3400700 1 -- 172.21.15.16:0/2647774318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f682597d80 msgr2=0x55f6825a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44e3400700 1 --2- 172.21.15.16:0/2647774318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f682597d80 0x55f6825a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44e3400700 1 -- 172.21.15.16:0/2647774318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6825861c0 con 0x55f6825031c0 2024-04-05T23:42:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44e3400700 1 --2- 172.21.15.16:0/2647774318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6825031c0 0x55f6825a8700 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f44d4005a80 tx=0x7f44d401b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4561d3e3202e18 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44d9600700 1 -- 172.21.15.16:0/2647774318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4021030 con 0x55f6825031c0 2024-04-05T23:42:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44d9600700 1 -- 172.21.15.16:0/2647774318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44d4032040 con 0x55f6825031c0 2024-04-05T23:42:59.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.090+0000 7f44d9600700 1 -- 172.21.15.16:0/2647774318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4004280 con 0x55f6825031c0 2024-04-05T23:42:59.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 -- 172.21.15.16:0/2647774318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6825031c0 msgr2=0x55f6825a8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:59.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/2647774318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6825031c0 0x55f6825a8700 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f44d4005a80 tx=0x7f44d401b6b0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 -- 172.21.15.16:0/2647774318 shutdown_connections 2024-04-05T23:42:59.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/2647774318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6825031c0 0x55f6825a8700 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/2647774318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f682589a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/2647774318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f682597d80 0x55f6825a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 -- 172.21.15.16:0/2647774318 >> 172.21.15.16:0/2647774318 conn(0x55f6825984c0 msgr2=0x55f68259b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 -- 172.21.15.16:0/2647774318 shutdown_connections 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 -- 172.21.15.16:0/2647774318 wait complete. 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.091+0000 7f44fe37cb80 1 -- start start 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44fe37cb80 1 Processor -- start 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6825031c0 0x55f6825a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44fe37cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f682604580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6825a5580 con 0x55f682597d80 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f682604ac0 con 0x55f6825031c0 2024-04-05T23:42:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44fe37cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f682604c80 con 0x55f68259d370 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44fe37cb80 10 client.0 _ll_register_callbacks cb 0x55f6822e7340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44e2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44e3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6825031c0 0x55f6825a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44e3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6825031c0 0x55f6825a48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42982/0 (socket says 172.21.15.16:42982) 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44e3400700 1 -- 172.21.15.16:0/709990407 learned_addr learned my addr 172.21.15.16:0/709990407 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44e2a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47308/0 (socket says 172.21.15.16:47308) 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.092+0000 7f44e3e00700 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f682604580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44e3400700 1 -- 172.21.15.16:0/709990407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 msgr2=0x55f682604580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44e3400700 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f682604580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44e3400700 1 -- 172.21.15.16:0/709990407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 msgr2=0x55f6825a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44e3400700 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44e3400700 1 -- 172.21.15.16:0/709990407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f68230db00 con 0x55f6825031c0 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44e2a00700 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:42:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44e3400700 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6825031c0 0x55f6825a48d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f44d4022040 tx=0x7f44d400f7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=751542e23960fd51 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44cd600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44cd600700 1 -- 172.21.15.16:0/709990407 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d401bec0 con 0x55f6825031c0 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44cd600700 1 -- 172.21.15.16:0/709990407 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f44d4022070 con 0x55f6825031c0 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.093+0000 7f44fe37cb80 1 -- 172.21.15.16:0/709990407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44d00a2600 con 0x55f6825031c0 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44cd600700 1 -- 172.21.15.16:0/709990407 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44d4004020 con 0x55f6825031c0 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44fe37cb80 10 client.26618 fetch_fsmap learned FSMap version 65 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44fe37cb80 1 -- 172.21.15.16:0/709990407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) v3 -- 0x55f6824a5e40 con 0x55f6825031c0 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44cd600700 1 -- 172.21.15.16:0/709990407 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44d4044020 con 0x55f6825031c0 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44cd600700 1 -- 172.21.15.16:0/709990407 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 65) v1 ==== 7514+0+0 (secure 0 0 0) 0x7f44d4005050 con 0x55f6825031c0 2024-04-05T23:42:59.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44fe37cb80 10 client.26618 fetch_fsmap finished waiting for FSMap version 65 2024-04-05T23:42:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44fe37cb80 10 client.26618 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:42:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44fe37cb80 20 client.26618 populate_metadata read hostname 'smithi016' 2024-04-05T23:42:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f68259dc90 0x55f682608080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:42:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44fe37cb80 4 client.26618 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:42:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44fe37cb80 1 -- 172.21.15.16:0/709990407 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f68232ec90 con 0x55f68259dc90 2024-04-05T23:42:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.094+0000 7f44e2a00700 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f68259dc90 0x55f682608080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:42:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.095+0000 7f44e2a00700 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f68259dc90 0x55f682608080 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=526a1d439fa75b25 server_cookie=b87905e041005ecf in_seq=0 out_seq=0 2024-04-05T23:42:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.095+0000 7f44cd600700 10 client.26618 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44cd600700 1 -- 172.21.15.16:0/709990407 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f68232ec90 con 0x55f68259dc90 2024-04-05T23:42:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44cd600700 10 client.26618 handle_command_reply: tid=0 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 client.26618 shutdown 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 -- 172.21.15.16:0/709990407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6825031c0 msgr2=0x55f6825a48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6825031c0 0x55f6825a48d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f44d4022040 tx=0x7f44d400f7a0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 -- 172.21.15.16:0/709990407 shutdown_connections 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55f68259dc90 0x55f682608080 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f68259d370 0x55f682604580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6825031c0 0x55f6825a48d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 --2- 172.21.15.16:0/709990407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f682597d80 0x55f6825a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44cd600700 0 client.26618 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 -- 172.21.15.16:0/709990407 >> 172.21.15.16:0/709990407 conn(0x55f6825984c0 msgr2=0x55f6825a18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 -- 172.21.15.16:0/709990407 shutdown_connections 2024-04-05T23:42:59.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.140+0000 7f44fe37cb80 1 -- 172.21.15.16:0/709990407 wait complete. 2024-04-05T23:42:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44fe37cb80 20 client.26618 trim_cache size 0 max 16384 2024-04-05T23:42:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:42:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:42:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:42:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:42:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 -- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f44c80a1290 msgr2=0x7f44c80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f44c80a1290 0x7f44c80a3750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009cb0 tx=0x7f44d4009260 comp rx=0 tx=0).stop 2024-04-05T23:42:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 -- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 msgr2=0x7f44e4073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:42:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 0x7f44e4073720 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f44d000e9f0 tx=0x7f44d000ed00 comp rx=0 tx=0).stop 2024-04-05T23:42:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 -- 172.21.15.16:0/2651326190 shutdown_connections 2024-04-05T23:42:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f44c80a1290 0x7f44c80a3750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44e4073c60 0x7f44e41a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44e4101030 0x7f44e40731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 --2- 172.21.15.16:0/2651326190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44e4106150 0x7f44e4073720 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:42:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 -- 172.21.15.16:0/2651326190 >> 172.21.15.16:0/2651326190 conn(0x7f44e40fc8b0 msgr2=0x7f44e40fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:42:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 -- 172.21.15.16:0/2651326190 shutdown_connections 2024-04-05T23:42:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:42:59.141+0000 7f44ea800700 1 -- 172.21.15.16:0/2651326190 wait complete. 2024-04-05T23:43:00.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462276 4.91 MB/sec execute 379 sec latency 1272.783 ms 2024-04-05T23:43:00.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:00 smithi016 ceph-mon[38248]: pgmap v294: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 837 KiB/s wr, 72 op/s 2024-04-05T23:43:00.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:00 smithi130 ceph-mon[79417]: pgmap v294: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 837 KiB/s wr, 72 op/s 2024-04-05T23:43:00.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:00 smithi027 ceph-mon[78597]: pgmap v294: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 837 KiB/s wr, 72 op/s 2024-04-05T23:43:01.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463512 4.91 MB/sec execute 380 sec latency 953.025 ms 2024-04-05T23:43:02.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463595 4.90 MB/sec execute 381 sec latency 753.834 ms 2024-04-05T23:43:02.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:02 smithi016 ceph-mon[38248]: pgmap v295: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-05T23:43:02.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:02 smithi130 ceph-mon[79417]: pgmap v295: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-05T23:43:02.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:02 smithi027 ceph-mon[78597]: pgmap v295: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-05T23:43:03.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463595 4.89 MB/sec execute 382 sec latency 1215.231 ms 2024-04-05T23:43:04.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463595 4.87 MB/sec execute 383 sec latency 2215.316 ms 2024-04-05T23:43:04.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:04 smithi016 ceph-mon[38248]: pgmap v296: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 884 KiB/s wr, 73 op/s 2024-04-05T23:43:04.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:04 smithi130 ceph-mon[79417]: pgmap v296: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 884 KiB/s wr, 73 op/s 2024-04-05T23:43:04.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:04 smithi027 ceph-mon[78597]: pgmap v296: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 884 KiB/s wr, 73 op/s 2024-04-05T23:43:05.021 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463595 4.86 MB/sec execute 384 sec latency 3215.414 ms 2024-04-05T23:43:06.022 INFO:tasks.workunit.client.0.smithi016.stdout: 1 463595 4.85 MB/sec execute 385 sec latency 4215.537 ms 2024-04-05T23:43:06.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:06 smithi016 ceph-mon[38248]: pgmap v297: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 884 KiB/s wr, 73 op/s 2024-04-05T23:43:06.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:06 smithi130 ceph-mon[79417]: pgmap v297: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 884 KiB/s wr, 73 op/s 2024-04-05T23:43:06.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:06 smithi027 ceph-mon[78597]: pgmap v297: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 884 KiB/s wr, 73 op/s 2024-04-05T23:43:07.022 INFO:tasks.workunit.client.0.smithi016.stdout: 1 464174 4.84 MB/sec execute 386 sec latency 4931.614 ms 2024-04-05T23:43:08.022 INFO:tasks.workunit.client.0.smithi016.stdout: 1 465563 4.86 MB/sec execute 387 sec latency 24.898 ms 2024-04-05T23:43:08.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:08 smithi016 ceph-mon[38248]: pgmap v298: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-04-05T23:43:08.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:08 smithi130 ceph-mon[79417]: pgmap v298: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-04-05T23:43:08.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:08 smithi027 ceph-mon[78597]: pgmap v298: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-04-05T23:43:09.022 INFO:tasks.workunit.client.0.smithi016.stdout: 1 466600 4.85 MB/sec execute 388 sec latency 38.711 ms 2024-04-05T23:43:09.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:09 smithi016 ceph-mon[38248]: osdmap e92: 12 total, 12 up, 12 in 2024-04-05T23:43:09.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:09 smithi130 ceph-mon[79417]: osdmap e92: 12 total, 12 up, 12 in 2024-04-05T23:43:09.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:09 smithi027 ceph-mon[78597]: osdmap e92: 12 total, 12 up, 12 in 2024-04-05T23:43:10.022 INFO:tasks.workunit.client.0.smithi016.stdout: 1 467138 4.85 MB/sec execute 389 sec latency 678.177 ms 2024-04-05T23:43:10.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:10 smithi016 ceph-mon[38248]: pgmap v300: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 944 KiB/s wr, 92 op/s 2024-04-05T23:43:10.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:10 smithi130 ceph-mon[79417]: pgmap v300: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 944 KiB/s wr, 92 op/s 2024-04-05T23:43:10.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:10 smithi027 ceph-mon[78597]: pgmap v300: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 944 KiB/s wr, 92 op/s 2024-04-05T23:43:11.022 INFO:tasks.workunit.client.0.smithi016.stdout: 1 467138 4.84 MB/sec execute 390 sec latency 1678.340 ms 2024-04-05T23:43:11.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:11 smithi130 ceph-mon[79417]: pgmap v301: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-05T23:43:11.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:11 smithi027 ceph-mon[78597]: pgmap v301: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-05T23:43:12.022 INFO:tasks.workunit.client.0.smithi016.stdout: 1 467836 4.83 MB/sec execute 391 sec latency 2393.800 ms 2024-04-05T23:43:12.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:11 smithi016 ceph-mon[38248]: pgmap v301: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-05T23:43:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.306+0000 7fca23000700 1 -- 172.21.15.16:0/491328204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca1c073bd0 msgr2=0x7fca1c10d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.306+0000 7fca23000700 1 --2- 172.21.15.16:0/491328204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca1c073bd0 0x7fca1c10d140 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fca08009990 tx=0x7fca08009ca0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.307+0000 7fca23000700 1 -- 172.21.15.16:0/491328204 shutdown_connections 2024-04-05T23:43:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.307+0000 7fca23000700 1 --2- 172.21.15.16:0/491328204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca1c10d680 0x7fca1c10fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.307+0000 7fca23000700 1 --2- 172.21.15.16:0/491328204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca1c0732b0 0x7fca1c073690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.307+0000 7fca23000700 1 --2- 172.21.15.16:0/491328204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca1c073bd0 0x7fca1c10d140 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.307+0000 7fca23000700 1 -- 172.21.15.16:0/491328204 >> 172.21.15.16:0/491328204 conn(0x7fca1c0fc8f0 msgr2=0x7fca1c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.307+0000 7fca23000700 1 -- 172.21.15.16:0/491328204 shutdown_connections 2024-04-05T23:43:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.307+0000 7fca23000700 1 -- 172.21.15.16:0/491328204 wait complete. 2024-04-05T23:43:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.308+0000 7fca23000700 1 Processor -- start 2024-04-05T23:43:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.308+0000 7fca23000700 1 -- start start 2024-04-05T23:43:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.308+0000 7fca23000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca1c0732b0 0x7fca1c199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.308+0000 7fca23000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca1c073bd0 0x7fca1c199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.308+0000 7fca23000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca1c10d680 0x7fca1c19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.308+0000 7fca23000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c19e430 con 0x7fca1c073bd0 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.308+0000 7fca23000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c19e5a0 con 0x7fca1c0732b0 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.308+0000 7fca23000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c19e760 con 0x7fca1c10d680 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca1c073bd0 0x7fca1c199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca21000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca1c10d680 0x7fca1c19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca1c0732b0 0x7fca1c199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca21000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca1c10d680 0x7fca1c19ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:33934/0 (socket says 172.21.15.16:33934) 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca1c073bd0 0x7fca1c199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47944/0 (socket says 172.21.15.16:47944) 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca21000700 1 -- 172.21.15.16:0/2858029715 learned_addr learned my addr 172.21.15.16:0/2858029715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1be00700 1 -- 172.21.15.16:0/2858029715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca1c10d680 msgr2=0x7fca1c19ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1be00700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca1c10d680 0x7fca1c19ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1be00700 1 -- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca1c073bd0 msgr2=0x7fca1c199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1be00700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca1c073bd0 0x7fca1c199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1be00700 1 -- 172.21.15.16:0/2858029715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca080095f0 con 0x7fca1c0732b0 2024-04-05T23:43:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1b400700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca1c073bd0 0x7fca1c199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:43:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca1be00700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca1c0732b0 0x7fca1c199510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00e9f0 tx=0x7fca0c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca18c00700 1 -- 172.21.15.16:0/2858029715 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c00c9c0 con 0x7fca1c0732b0 2024-04-05T23:43:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca18c00700 1 -- 172.21.15.16:0/2858029715 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca0c010430 con 0x7fca1c0732b0 2024-04-05T23:43:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.309+0000 7fca18c00700 1 -- 172.21.15.16:0/2858029715 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c018680 con 0x7fca1c0732b0 2024-04-05T23:43:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.310+0000 7fca23000700 1 -- 172.21.15.16:0/2858029715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c19ea40 con 0x7fca1c0732b0 2024-04-05T23:43:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.310+0000 7fca23000700 1 -- 172.21.15.16:0/2858029715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c1a6280 con 0x7fca1c0732b0 2024-04-05T23:43:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.313+0000 7fca18c00700 1 -- 172.21.15.16:0/2858029715 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fca0c010860 con 0x7fca1c0732b0 2024-04-05T23:43:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.314+0000 7fca36ab5b80 1 Processor -- start 2024-04-05T23:43:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.314+0000 7fca36ab5b80 1 -- start start 2024-04-05T23:43:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.315+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d49b80 0x562b81d49f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.315+0000 7fca18c00700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca000a1310 0x7fca000a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.315+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d44030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.315+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d445c0 0x562b81d44a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.315+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81caab80 con 0x562b81d49b80 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.315+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81c9b930 con 0x562b81d445c0 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.315+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81c992f0 con 0x562b81d4a4a0 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.315+0000 7fca18c00700 1 -- 172.21.15.16:0/2858029715 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fca0c014070 con 0x7fca1c0732b0 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca21000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d445c0 0x562b81d44a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca21000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d445c0 0x562b81d44a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:45070/0 (socket says 172.21.15.16:45070) 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca21000700 1 -- 172.21.15.16:0/3780489874 learned_addr learned my addr 172.21.15.16:0/3780489874 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca21000700 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d49b80 0x562b81d49f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca1be00700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca000a1310 0x7fca000a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca1b400700 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d44030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca21000700 1 -- 172.21.15.16:0/3780489874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 msgr2=0x562b81d44030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca21000700 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d44030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca21000700 1 -- 172.21.15.16:0/3780489874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d49b80 msgr2=0x562b81d49f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca21000700 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d49b80 0x562b81d49f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca21000700 1 -- 172.21.15.16:0/3780489874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b81b16350 con 0x562b81d445c0 2024-04-05T23:43:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca1b400700 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d44030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:43:12.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.316+0000 7fca1be00700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca000a1310 0x7fca000a37d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0a1e20 tx=0x7fca0c009790 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:12.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.317+0000 7fca21000700 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d445c0 0x562b81d44a20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca14008d60 tx=0x7fca14009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c403d9d0a4c09d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.317+0000 7fca11600700 1 -- 172.21.15.16:0/3780489874 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1400e070 con 0x562b81d445c0 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.317+0000 7fca11600700 1 -- 172.21.15.16:0/3780489874 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca1400f040 con 0x562b81d445c0 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.317+0000 7fca11600700 1 -- 172.21.15.16:0/3780489874 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14018c40 con 0x562b81d445c0 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 -- 172.21.15.16:0/3780489874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d445c0 msgr2=0x562b81d44a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d445c0 0x562b81d44a20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca14008d60 tx=0x7fca14009920 comp rx=0 tx=0).stop 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 -- 172.21.15.16:0/3780489874 shutdown_connections 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d49b80 0x562b81d49f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d445c0 0x562b81d44a20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/3780489874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d44030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 -- 172.21.15.16:0/3780489874 >> 172.21.15.16:0/3780489874 conn(0x562b81cb81c0 msgr2=0x562b81d3ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 -- 172.21.15.16:0/3780489874 shutdown_connections 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 -- 172.21.15.16:0/3780489874 wait complete. 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.318+0000 7fca36ab5b80 1 -- start start 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.319+0000 7fca36ab5b80 1 Processor -- start 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.319+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 0x562b81d50520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.319+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d4d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.319+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d4dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.319+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81c94be0 con 0x562b81d445c0 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.319+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81c30d20 con 0x562b81d49b80 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.319+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81c309c0 con 0x562b81d4a4a0 2024-04-05T23:43:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.319+0000 7fca36ab5b80 10 client.0 _ll_register_callbacks cb 0x562b81a9c340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca1b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d4d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca1b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d4d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:45074/0 (socket says 172.21.15.16:45074) 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca1b400700 1 -- 172.21.15.16:0/545435071 learned_addr learned my addr 172.21.15.16:0/545435071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca21000700 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 0x562b81d50520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca21000700 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d4dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca1b400700 1 -- 172.21.15.16:0/545435071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 msgr2=0x562b81d4dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca1b400700 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d4dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca1b400700 1 -- 172.21.15.16:0/545435071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 msgr2=0x562b81d50520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca1b400700 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 0x562b81d50520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca1b400700 1 -- 172.21.15.16:0/545435071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c1107c0 con 0x562b81d49b80 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca21000700 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 0x562b81d50520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca1b400700 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d4d8b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fca08009990 tx=0x7fca08005300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d76377c0a90bce37 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.320+0000 7fca06000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.321+0000 7fca06000700 1 -- 172.21.15.16:0/545435071 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0801d070 con 0x562b81d49b80 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.321+0000 7fca06000700 1 -- 172.21.15.16:0/545435071 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fca08004910 con 0x562b81d49b80 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.321+0000 7fca36ab5b80 1 -- 172.21.15.16:0/545435071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b81d371e0 con 0x562b81d49b80 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.321+0000 7fca06000700 1 -- 172.21.15.16:0/545435071 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca08004ad0 con 0x562b81d49b80 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.321+0000 7fca36ab5b80 10 client.26630 fetch_fsmap learned FSMap version 65 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.321+0000 7fca36ab5b80 1 -- 172.21.15.16:0/545435071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) v3 -- 0x7fca14008a40 con 0x562b81d49b80 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.321+0000 7fca06000700 1 -- 172.21.15.16:0/545435071 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562b81d371e0 con 0x562b81d49b80 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca06000700 1 -- 172.21.15.16:0/545435071 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 65) v1 ==== 7514+0+0 (secure 0 0 0) 0x7fca08024450 con 0x562b81d49b80 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca36ab5b80 10 client.26630 fetch_fsmap finished waiting for FSMap version 65 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca36ab5b80 10 client.26630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca36ab5b80 20 client.26630 populate_metadata read hostname 'smithi016' 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562b81d5def0 0x562b81d60310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca36ab5b80 4 client.26630 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca36ab5b80 1 -- 172.21.15.16:0/545435071 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b81c8d6a0 con 0x562b81d5def0 2024-04-05T23:43:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca21000700 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562b81d5def0 0x562b81d60310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca21000700 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562b81d5def0 0x562b81d60310 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78375464902f961b server_cookie=a0467f39e77de4b6 in_seq=0 out_seq=0 2024-04-05T23:43:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.322+0000 7fca06000700 10 client.26630 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca06000700 1 -- 172.21.15.16:0/545435071 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562b81c8d6a0 con 0x562b81d5def0 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca06000700 10 client.26630 handle_command_reply: tid=0 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 client.26630 shutdown 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 -- 172.21.15.16:0/545435071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 msgr2=0x562b81d4d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d4d8b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fca08009990 tx=0x7fca08005300 comp rx=0 tx=0).stop 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 -- 172.21.15.16:0/545435071 shutdown_connections 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562b81d5def0 0x562b81d60310 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 0x562b81d50520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d4dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca06000700 0 client.26630 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/545435071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d4d8b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 -- 172.21.15.16:0/545435071 >> 172.21.15.16:0/545435071 conn(0x562b81cb81c0 msgr2=0x562b81d3e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.323+0000 7fca36ab5b80 1 -- 172.21.15.16:0/545435071 shutdown_connections 2024-04-05T23:43:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.324+0000 7fca36ab5b80 1 -- 172.21.15.16:0/545435071 wait complete. 2024-04-05T23:43:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.324+0000 7fca36ab5b80 20 client.26630 trim_cache size 0 max 16384 2024-04-05T23:43:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.340+0000 7fca36ab5b80 1 Processor -- start 2024-04-05T23:43:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.340+0000 7fca36ab5b80 1 -- start start 2024-04-05T23:43:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.340+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d445c0 0x562b81d54f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d49090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d4a4a0 0x562b81d3ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81b18640 con 0x562b81d4a4a0 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81d48370 con 0x562b81d49b80 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81d40600 con 0x562b81d445c0 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca21000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d445c0 0x562b81d54f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca21000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d445c0 0x562b81d54f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:33940/0 (socket says 172.21.15.16:33940) 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca21000700 1 -- 172.21.15.16:0/1085851608 learned_addr learned my addr 172.21.15.16:0/1085851608 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca1b400700 1 --2- 172.21.15.16:0/1085851608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d49090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca21000700 1 --2- 172.21.15.16:0/1085851608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d4a4a0 0x562b81d3ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca1b400700 1 -- 172.21.15.16:0/1085851608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d445c0 msgr2=0x562b81d54f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca1b400700 1 --2- 172.21.15.16:0/1085851608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d445c0 0x562b81d54f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca1b400700 1 -- 172.21.15.16:0/1085851608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d4a4a0 msgr2=0x562b81d3ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca1b400700 1 --2- 172.21.15.16:0/1085851608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d4a4a0 0x562b81d3ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.341+0000 7fca1b400700 1 -- 172.21.15.16:0/1085851608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b81ac2b00 con 0x562b81d49b80 2024-04-05T23:43:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.342+0000 7fca1b400700 1 --2- 172.21.15.16:0/1085851608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d49090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fca080098e0 tx=0x7fca08005a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df8179ac5cb3e71d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.342+0000 7fca12000700 1 -- 172.21.15.16:0/1085851608 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0801d070 con 0x562b81d49b80 2024-04-05T23:43:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.342+0000 7fca12000700 1 -- 172.21.15.16:0/1085851608 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca0801e040 con 0x562b81d49b80 2024-04-05T23:43:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.342+0000 7fca12000700 1 -- 172.21.15.16:0/1085851608 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08016b70 con 0x562b81d49b80 2024-04-05T23:43:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1085851608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 msgr2=0x562b81d49090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1085851608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d49090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fca080098e0 tx=0x7fca08005a90 comp rx=0 tx=0).stop 2024-04-05T23:43:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1085851608 shutdown_connections 2024-04-05T23:43:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1085851608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d4a4a0 0x562b81d3ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1085851608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d445c0 0x562b81d54f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1085851608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d49090 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1085851608 >> 172.21.15.16:0/1085851608 conn(0x562b81cb81c0 msgr2=0x562b81d48980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1085851608 shutdown_connections 2024-04-05T23:43:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1085851608 wait complete. 2024-04-05T23:43:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.343+0000 7fca36ab5b80 1 -- start start 2024-04-05T23:43:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.344+0000 7fca36ab5b80 1 Processor -- start 2024-04-05T23:43:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.344+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 0x562b81d52f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.344+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d53480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.344+0000 7fca36ab5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.344+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81d55310 con 0x562b81d445c0 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.344+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81db0d20 con 0x562b81d49b80 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca36ab5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b81db0ee0 con 0x562b81d4a4a0 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca36ab5b80 10 client.0 _ll_register_callbacks cb 0x7fca0800be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca1b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d53480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca21000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 0x562b81d52f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca1b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d53480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:45088/0 (socket says 172.21.15.16:45088) 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca1b400700 1 -- 172.21.15.16:0/1160415152 learned_addr learned my addr 172.21.15.16:0/1160415152 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca1b400700 1 -- 172.21.15.16:0/1160415152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 msgr2=0x562b81d539c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca1b400700 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca1b400700 1 -- 172.21.15.16:0/1160415152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 msgr2=0x562b81d52f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca1b400700 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 0x562b81d52f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.345+0000 7fca1b400700 1 -- 172.21.15.16:0/1160415152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca14008a40 con 0x562b81d49b80 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca1b400700 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d53480 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fca08006010 tx=0x7fca0800b680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f724ee0f56c1a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca06000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca06000700 1 -- 172.21.15.16:0/1160415152 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0801d040 con 0x562b81d49b80 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca06000700 1 -- 172.21.15.16:0/1160415152 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fca08000e40 con 0x562b81d49b80 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1160415152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca1400f040 con 0x562b81d49b80 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca06000700 1 -- 172.21.15.16:0/1160415152 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca08004b20 con 0x562b81d49b80 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca36ab5b80 10 client.26636 fetch_fsmap learned FSMap version 65 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1160415152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) v3 -- 0x562b81c5ae40 con 0x562b81d49b80 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca06000700 1 -- 172.21.15.16:0/1160415152 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca1400f040 con 0x562b81d49b80 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.346+0000 7fca06000700 1 -- 172.21.15.16:0/1160415152 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 65) v1 ==== 7514+0+0 (secure 0 0 0) 0x7fca08037c90 con 0x562b81d49b80 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.347+0000 7fca36ab5b80 10 client.26636 fetch_fsmap finished waiting for FSMap version 65 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.347+0000 7fca36ab5b80 10 client.26636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.347+0000 7fca36ab5b80 20 client.26636 populate_metadata read hostname 'smithi016' 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.347+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562b81db2260 0x562b81db4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.347+0000 7fca36ab5b80 4 client.26636 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:43:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.347+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1160415152 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562b81ae3c90 con 0x562b81db2260 2024-04-05T23:43:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.347+0000 7fca21000700 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562b81db2260 0x562b81db4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.347+0000 7fca06000700 10 client.26636 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.347+0000 7fca21000700 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562b81db2260 0x562b81db4650 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=907c5ff76911c3cb server_cookie=b75089c333c9a63d in_seq=0 out_seq=0 2024-04-05T23:43:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.383+0000 7fca06000700 1 -- 172.21.15.16:0/1160415152 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562b81ae3c90 con 0x562b81db2260 2024-04-05T23:43:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.383+0000 7fca06000700 10 client.26636 handle_command_reply: tid=0 2024-04-05T23:43:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.383+0000 7fca36ab5b80 1 client.26636 shutdown 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.383+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1160415152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 msgr2=0x562b81d53480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.383+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d53480 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fca08006010 tx=0x7fca0800b680 comp rx=0 tx=0).stop 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1160415152 shutdown_connections 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562b81db2260 0x562b81db4650 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562b81d445c0 0x562b81d52f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b81d4a4a0 0x562b81d539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca36ab5b80 1 --2- 172.21.15.16:0/1160415152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562b81d49b80 0x562b81d53480 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1160415152 >> 172.21.15.16:0/1160415152 conn(0x562b81cb81c0 msgr2=0x562b81d59f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca06000700 0 client.26636 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1160415152 shutdown_connections 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca36ab5b80 1 -- 172.21.15.16:0/1160415152 wait complete. 2024-04-05T23:43:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.384+0000 7fca36ab5b80 20 client.26636 trim_cache size 0 max 16384 2024-04-05T23:43:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:43:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:43:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:43:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 -- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca000a1310 msgr2=0x7fca000a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca000a1310 0x7fca000a37d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0a1e20 tx=0x7fca0c009790 comp rx=0 tx=0).stop 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 -- 172.21.15.16:0/2858029715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca1c0732b0 msgr2=0x7fca1c199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca1c0732b0 0x7fca1c199510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00e9f0 tx=0x7fca0c00ed00 comp rx=0 tx=0).stop 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 -- 172.21.15.16:0/2858029715 shutdown_connections 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca000a1310 0x7fca000a37d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca1c10d680 0x7fca1c19ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca1c0732b0 0x7fca1c199510 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 --2- 172.21.15.16:0/2858029715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca1c073bd0 0x7fca1c199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.385+0000 7fca23000700 1 -- 172.21.15.16:0/2858029715 >> 172.21.15.16:0/2858029715 conn(0x7fca1c0fc8f0 msgr2=0x7fca1c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.386+0000 7fca23000700 1 -- 172.21.15.16:0/2858029715 shutdown_connections 2024-04-05T23:43:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:12.386+0000 7fca23000700 1 -- 172.21.15.16:0/2858029715 wait complete. 2024-04-05T23:43:13.022 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469357 4.85 MB/sec execute 392 sec latency 22.717 ms 2024-04-05T23:43:13.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:13 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:43:13.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:13 smithi027 ceph-mon[78597]: pgmap v302: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-05T23:43:14.022 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469378 4.83 MB/sec execute 393 sec latency 963.561 ms 2024-04-05T23:43:14.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:13 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:43:14.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:13 smithi016 ceph-mon[38248]: pgmap v302: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-05T23:43:14.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:13 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:43:14.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:13 smithi130 ceph-mon[79417]: pgmap v302: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-05T23:43:15.023 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469378 4.82 MB/sec execute 394 sec latency 1963.683 ms 2024-04-05T23:43:15.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:15 smithi016 ceph-mon[38248]: osdmap e93: 12 total, 12 up, 12 in 2024-04-05T23:43:15.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:15 smithi130 ceph-mon[79417]: osdmap e93: 12 total, 12 up, 12 in 2024-04-05T23:43:15.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:15 smithi027 ceph-mon[78597]: osdmap e93: 12 total, 12 up, 12 in 2024-04-05T23:43:16.023 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469378 4.81 MB/sec execute 395 sec latency 2963.805 ms 2024-04-05T23:43:16.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:16 smithi016 ceph-mon[38248]: pgmap v304: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-05T23:43:16.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:16 smithi130 ceph-mon[79417]: pgmap v304: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-05T23:43:16.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:16 smithi027 ceph-mon[78597]: pgmap v304: 97 pgs: 97 active+clean; 172 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-05T23:43:17.023 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469378 4.80 MB/sec execute 396 sec latency 3963.918 ms 2024-04-05T23:43:18.023 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469590 4.79 MB/sec execute 397 sec latency 4163.560 ms 2024-04-05T23:43:18.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:18 smithi016 ceph-mon[38248]: pgmap v305: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.0 MiB/s wr, 192 op/s 2024-04-05T23:43:18.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:18 smithi130 ceph-mon[79417]: pgmap v305: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.0 MiB/s wr, 192 op/s 2024-04-05T23:43:18.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:18 smithi027 ceph-mon[78597]: pgmap v305: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.0 MiB/s wr, 192 op/s 2024-04-05T23:43:19.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469590 4.77 MB/sec execute 398 sec latency 1528.016 ms 2024-04-05T23:43:19.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:19 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:43:19.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:19 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:43:19.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:19 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:43:20.023 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469590 4.76 MB/sec execute 399 sec latency 2528.181 ms 2024-04-05T23:43:20.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:20 smithi016 ceph-mon[38248]: pgmap v306: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 2.5 MiB/s wr, 157 op/s 2024-04-05T23:43:20.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:20 smithi130 ceph-mon[79417]: pgmap v306: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 2.5 MiB/s wr, 157 op/s 2024-04-05T23:43:20.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:20 smithi027 ceph-mon[78597]: pgmap v306: 97 pgs: 97 active+clean; 181 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 2.5 MiB/s wr, 157 op/s 2024-04-05T23:43:21.023 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469590 4.75 MB/sec execute 400 sec latency 3528.277 ms 2024-04-05T23:43:22.023 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469590 4.74 MB/sec execute 401 sec latency 4528.354 ms 2024-04-05T23:43:22.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:22 smithi016 ceph-mon[38248]: pgmap v307: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 2.6 MiB/s wr, 92 op/s 2024-04-05T23:43:22.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:22 smithi130 ceph-mon[79417]: pgmap v307: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 2.6 MiB/s wr, 92 op/s 2024-04-05T23:43:22.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:22 smithi027 ceph-mon[78597]: pgmap v307: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 2.6 MiB/s wr, 92 op/s 2024-04-05T23:43:23.023 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469590 4.73 MB/sec execute 402 sec latency 5528.434 ms 2024-04-05T23:43:24.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469590 4.71 MB/sec execute 403 sec latency 6528.575 ms 2024-04-05T23:43:24.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:24 smithi016 ceph-mon[38248]: pgmap v308: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 2.6 MiB/s wr, 92 op/s 2024-04-05T23:43:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:24 smithi130 ceph-mon[79417]: pgmap v308: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 2.6 MiB/s wr, 92 op/s 2024-04-05T23:43:24.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:24 smithi027 ceph-mon[78597]: pgmap v308: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 2.6 MiB/s wr, 92 op/s 2024-04-05T23:43:25.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 469590 4.70 MB/sec execute 404 sec latency 7528.739 ms 2024-04-05T23:43:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.559+0000 7fd363e00700 1 -- 172.21.15.16:0/2719813486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd35c073c30 msgr2=0x7fd35c10d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.559+0000 7fd363e00700 1 --2- 172.21.15.16:0/2719813486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd35c073c30 0x7fd35c10d0c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd348009990 tx=0x7fd348009ca0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.559+0000 7fd363e00700 1 -- 172.21.15.16:0/2719813486 shutdown_connections 2024-04-05T23:43:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.559+0000 7fd363e00700 1 --2- 172.21.15.16:0/2719813486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd35c073310 0x7fd35c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.559+0000 7fd363e00700 1 --2- 172.21.15.16:0/2719813486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd35c10d600 0x7fd35c10fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.559+0000 7fd363e00700 1 --2- 172.21.15.16:0/2719813486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd35c073c30 0x7fd35c10d0c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.559+0000 7fd363e00700 1 -- 172.21.15.16:0/2719813486 >> 172.21.15.16:0/2719813486 conn(0x7fd35c0fc8b0 msgr2=0x7fd35c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.559+0000 7fd363e00700 1 -- 172.21.15.16:0/2719813486 shutdown_connections 2024-04-05T23:43:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.559+0000 7fd363e00700 1 -- 172.21.15.16:0/2719813486 wait complete. 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.560+0000 7fd363e00700 1 Processor -- start 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.560+0000 7fd363e00700 1 -- start start 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd363e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd35c073310 0x7fd35c199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd363e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd35c073c30 0x7fd35c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd363e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd35c10d600 0x7fd35c19dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd363e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c19e3b0 con 0x7fd35c073310 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd363e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c19e520 con 0x7fd35c10d600 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd363e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c19e6e0 con 0x7fd35c073c30 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd361400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd35c073310 0x7fd35c199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd361400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd35c073310 0x7fd35c199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38014/0 (socket says 172.21.15.16:38014) 2024-04-05T23:43:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd361400700 1 -- 172.21.15.16:0/2635293365 learned_addr learned my addr 172.21.15.16:0/2635293365 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd360a00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd35c073c30 0x7fd35c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.561+0000 7fd361e00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd35c10d600 0x7fd35c19dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd361400700 1 -- 172.21.15.16:0/2635293365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd35c073c30 msgr2=0x7fd35c1999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd361400700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd35c073c30 0x7fd35c1999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd361400700 1 -- 172.21.15.16:0/2635293365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd35c10d600 msgr2=0x7fd35c19dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd361400700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd35c10d600 0x7fd35c19dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd361400700 1 -- 172.21.15.16:0/2635293365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480095f0 con 0x7fd35c073310 2024-04-05T23:43:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd361e00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd35c10d600 0x7fd35c19dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:43:25.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd361400700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd35c073310 0x7fd35c199490 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd35000e9a0 tx=0x7fd35000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:25.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd35a000700 1 -- 172.21.15.16:0/2635293365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350009790 con 0x7fd35c073310 2024-04-05T23:43:25.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd35a000700 1 -- 172.21.15.16:0/2635293365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd350010430 con 0x7fd35c073310 2024-04-05T23:43:25.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd363e00700 1 -- 172.21.15.16:0/2635293365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c19e9c0 con 0x7fd35c073310 2024-04-05T23:43:25.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd35a000700 1 -- 172.21.15.16:0/2635293365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350018680 con 0x7fd35c073310 2024-04-05T23:43:25.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.562+0000 7fd363e00700 1 -- 172.21.15.16:0/2635293365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c1a6200 con 0x7fd35c073310 2024-04-05T23:43:25.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.566+0000 7fd35a000700 1 -- 172.21.15.16:0/2635293365 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fd3500105a0 con 0x7fd35c073310 2024-04-05T23:43:25.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.568+0000 7fd35a000700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd3440a1290 0x7fd3440a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.568+0000 7fd3778b6b80 1 Processor -- start 2024-04-05T23:43:25.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.568+0000 7fd35a000700 1 -- 172.21.15.16:0/2635293365 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fd350014070 con 0x7fd35c073310 2024-04-05T23:43:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.569+0000 7fd3778b6b80 1 -- start start 2024-04-05T23:43:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.569+0000 7fd360a00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd3440a1290 0x7fd3440a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.569+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7083f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.569+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e70844a0 0x5558e707e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.569+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e707e5c0 0x5558e707ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.569+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6fe4b80 con 0x5558e70844a0 2024-04-05T23:43:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.569+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6fd5930 con 0x5558e707e5c0 2024-04-05T23:43:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.569+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6fd32f0 con 0x5558e7083b80 2024-04-05T23:43:25.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.569+0000 7fd361400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7083f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd361400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7083f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:49576/0 (socket says 172.21.15.16:49576) 2024-04-05T23:43:25.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd361400700 1 -- 172.21.15.16:0/1308522057 learned_addr learned my addr 172.21.15.16:0/1308522057 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:25.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd361e00700 1 --2- 172.21.15.16:0/1308522057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e707e5c0 0x5558e707ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd361400700 1 -- 172.21.15.16:0/1308522057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e707e5c0 msgr2=0x5558e707ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:25.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd361400700 1 --2- 172.21.15.16:0/1308522057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e707e5c0 0x5558e707ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd361400700 1 -- 172.21.15.16:0/1308522057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e70844a0 msgr2=0x5558e707e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:25.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd361400700 1 --2- 172.21.15.16:0/1308522057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e70844a0 0x5558e707e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd361400700 1 -- 172.21.15.16:0/1308522057 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558e6e50350 con 0x5558e7083b80 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd360a00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd3440a1290 0x7fd3440a3750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd35c19a9e0 tx=0x7fd348005fd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd361400700 1 --2- 172.21.15.16:0/1308522057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7083f60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd350096a30 tx=0x7fd350099c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcaa44d6aa75c075 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.570+0000 7fd34e600700 1 -- 172.21.15.16:0/1308522057 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35009ddf0 con 0x5558e7083b80 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.571+0000 7fd34e600700 1 -- 172.21.15.16:0/1308522057 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3500ade90 con 0x5558e7083b80 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.571+0000 7fd34e600700 1 -- 172.21.15.16:0/1308522057 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350099710 con 0x5558e7083b80 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.571+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1308522057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 msgr2=0x5558e7083f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.571+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1308522057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7083f60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd350096a30 tx=0x7fd350099c30 comp rx=0 tx=0).stop 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.571+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1308522057 shutdown_connections 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.571+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1308522057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e707e5c0 0x5558e707ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.571+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1308522057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7083f60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.571+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1308522057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e70844a0 0x5558e707e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.571+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1308522057 >> 172.21.15.16:0/1308522057 conn(0x5558e6ff21c0 msgr2=0x5558e7078be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.572+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1308522057 shutdown_connections 2024-04-05T23:43:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.572+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1308522057 wait complete. 2024-04-05T23:43:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.572+0000 7fd3778b6b80 1 -- start start 2024-04-05T23:43:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.572+0000 7fd3778b6b80 1 Processor -- start 2024-04-05T23:43:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e7087870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7087db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e70882f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6fcebe0 con 0x5558e707e5c0 2024-04-05T23:43:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6f6ad20 con 0x5558e70844a0 2024-04-05T23:43:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6f6a9c0 con 0x5558e7083b80 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd3778b6b80 10 client.0 _ll_register_callbacks cb 0x5558e6dd6340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd360a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7087db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd361400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e7087870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd361e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e70882f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd360a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7087db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:49582/0 (socket says 172.21.15.16:49582) 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.573+0000 7fd360a00700 1 -- 172.21.15.16:0/1441838492 learned_addr learned my addr 172.21.15.16:0/1441838492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd360a00700 1 -- 172.21.15.16:0/1441838492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 msgr2=0x5558e70882f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd360a00700 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e70882f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd360a00700 1 -- 172.21.15.16:0/1441838492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 msgr2=0x5558e7087870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd360a00700 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e7087870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd360a00700 1 -- 172.21.15.16:0/1441838492 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c1a6200 con 0x5558e7083b80 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd361400700 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e7087870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd360a00700 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7087db0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd3480098e0 tx=0x7fd348009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dc46b455dad1a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd342a00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd342a00700 1 -- 172.21.15.16:0/1441838492 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348004cf0 con 0x5558e7083b80 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd342a00700 1 -- 172.21.15.16:0/1441838492 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fd34800be70 con 0x5558e7083b80 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1441838492 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558e70711e0 con 0x5558e7083b80 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd342a00700 1 -- 172.21.15.16:0/1441838492 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd348004e50 con 0x5558e7083b80 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.574+0000 7fd3778b6b80 10 client.26671 fetch_fsmap learned FSMap version 67 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.575+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1441838492 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) v3 -- 0x5558e6dfcb00 con 0x5558e7083b80 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.575+0000 7fd342a00700 1 -- 172.21.15.16:0/1441838492 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd34803c020 con 0x5558e7083b80 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.575+0000 7fd342a00700 1 -- 172.21.15.16:0/1441838492 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 67) v1 ==== 7530+0+0 (secure 0 0 0) 0x7fd348024590 con 0x5558e7083b80 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.575+0000 7fd3778b6b80 10 client.26671 fetch_fsmap finished waiting for FSMap version 67 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.575+0000 7fd3778b6b80 10 client.26671 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.575+0000 7fd3778b6b80 20 client.26671 populate_metadata read hostname 'smithi016' 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.575+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5558e70980b0 0x5558e709a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.575+0000 7fd3778b6b80 4 client.26671 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:43:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.575+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1441838492 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558e6fc76a0 con 0x5558e70980b0 2024-04-05T23:43:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.576+0000 7fd361400700 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5558e70980b0 0x5558e709a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.576+0000 7fd361400700 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5558e70980b0 0x5558e709a4d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1627582c097e5918 server_cookie=57f6c81f35bf586f in_seq=0 out_seq=0 2024-04-05T23:43:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.576+0000 7fd342a00700 10 client.26671 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd342a00700 1 -- 172.21.15.16:0/1441838492 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5558e6fc76a0 con 0x5558e70980b0 2024-04-05T23:43:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd342a00700 10 client.26671 handle_command_reply: tid=0 2024-04-05T23:43:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 client.26671 shutdown 2024-04-05T23:43:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1441838492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 msgr2=0x5558e7087db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7087db0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd3480098e0 tx=0x7fd348009310 comp rx=0 tx=0).stop 2024-04-05T23:43:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1441838492 shutdown_connections 2024-04-05T23:43:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5558e70980b0 0x5558e709a4d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e7087870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e70882f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd342a00700 0 client.26671 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:25.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/1441838492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e7087db0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1441838492 >> 172.21.15.16:0/1441838492 conn(0x5558e6ff21c0 msgr2=0x5558e7078280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:25.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1441838492 shutdown_connections 2024-04-05T23:43:25.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 1 -- 172.21.15.16:0/1441838492 wait complete. 2024-04-05T23:43:25.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.577+0000 7fd3778b6b80 20 client.26671 trim_cache size 0 max 16384 2024-04-05T23:43:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.598+0000 7fd3778b6b80 1 Processor -- start 2024-04-05T23:43:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.598+0000 7fd3778b6b80 1 -- start start 2024-04-05T23:43:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e707e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e7083010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6e52640 con 0x5558e707e5c0 2024-04-05T23:43:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e7088ea0 con 0x5558e70844a0 2024-04-05T23:43:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e707a6a0 con 0x5558e7083b80 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd360a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd361e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e7083010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd361400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e707e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd360a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:49598/0 (socket says 172.21.15.16:49598) 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd360a00700 1 -- 172.21.15.16:0/4271779409 learned_addr learned my addr 172.21.15.16:0/4271779409 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.599+0000 7fd361e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e7083010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:44424/0 (socket says 172.21.15.16:44424) 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd361400700 1 -- 172.21.15.16:0/4271779409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 msgr2=0x5558e708ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd361400700 1 --2- 172.21.15.16:0/4271779409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd361400700 1 -- 172.21.15.16:0/4271779409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 msgr2=0x5558e7083010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd361400700 1 --2- 172.21.15.16:0/4271779409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e7083010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd361400700 1 -- 172.21.15.16:0/4271779409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558e6f94e40 con 0x5558e707e5c0 2024-04-05T23:43:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd361e00700 1 --2- 172.21.15.16:0/4271779409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e7083010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:43:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd360a00700 1 --2- 172.21.15.16:0/4271779409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:43:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd361400700 1 --2- 172.21.15.16:0/4271779409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e707e030 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd350096a30 tx=0x7fd350010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b792bf032a7add server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd34f000700 1 -- 172.21.15.16:0/4271779409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35009ddf0 con 0x5558e707e5c0 2024-04-05T23:43:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd34f000700 1 -- 172.21.15.16:0/4271779409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3500a2440 con 0x5558e707e5c0 2024-04-05T23:43:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.600+0000 7fd34f000700 1 -- 172.21.15.16:0/4271779409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3500a1fb0 con 0x5558e707e5c0 2024-04-05T23:43:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.601+0000 7fd3778b6b80 1 -- 172.21.15.16:0/4271779409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 msgr2=0x5558e707e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.601+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/4271779409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e707e030 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd350096a30 tx=0x7fd350010c80 comp rx=0 tx=0).stop 2024-04-05T23:43:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.601+0000 7fd3778b6b80 1 -- 172.21.15.16:0/4271779409 shutdown_connections 2024-04-05T23:43:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.601+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/4271779409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e707e030 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.601+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/4271779409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e7083010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.601+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/4271779409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.601+0000 7fd3778b6b80 1 -- 172.21.15.16:0/4271779409 >> 172.21.15.16:0/4271779409 conn(0x5558e6ff21c0 msgr2=0x5558e7082900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.602+0000 7fd3778b6b80 1 -- 172.21.15.16:0/4271779409 shutdown_connections 2024-04-05T23:43:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.602+0000 7fd3778b6b80 1 -- 172.21.15.16:0/4271779409 wait complete. 2024-04-05T23:43:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.602+0000 7fd3778b6b80 1 -- start start 2024-04-05T23:43:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.602+0000 7fd3778b6b80 1 Processor -- start 2024-04-05T23:43:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e708af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd3778b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e708b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e70eae90 con 0x5558e707e5c0 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e70eafd0 con 0x5558e70844a0 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd3778b6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e70eb110 con 0x5558e7083b80 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd3778b6b80 10 client.0 _ll_register_callbacks cb 0x7fd34800b750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd361400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e708af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd361400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e708af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38048/0 (socket says 172.21.15.16:38048) 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd361400700 1 -- 172.21.15.16:0/373473344 learned_addr learned my addr 172.21.15.16:0/373473344 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd360a00700 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.603+0000 7fd361e00700 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e708b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd361400700 1 -- 172.21.15.16:0/373473344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 msgr2=0x5558e708b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd361400700 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd361400700 1 -- 172.21.15.16:0/373473344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 msgr2=0x5558e708b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd361400700 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e708b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd361400700 1 -- 172.21.15.16:0/373473344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558e6dfcb00 con 0x5558e707e5c0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd360a00700 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd361400700 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e708af30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd350000c00 tx=0x7fd35009a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb3fac4655c7d2d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd342a00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd342a00700 1 -- 172.21.15.16:0/373473344 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35009ddf0 con 0x5558e707e5c0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd342a00700 1 -- 172.21.15.16:0/373473344 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fd350094bd0 con 0x5558e707e5c0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.604+0000 7fd3778b6b80 1 -- 172.21.15.16:0/373473344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3500ade90 con 0x5558e707e5c0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd342a00700 1 -- 172.21.15.16:0/373473344 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd350094d90 con 0x5558e707e5c0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd3778b6b80 10 client.16602 fetch_fsmap learned FSMap version 67 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd3778b6b80 1 -- 172.21.15.16:0/373473344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) v3 -- 0x7fd35400e610 con 0x5558e707e5c0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd342a00700 1 -- 172.21.15.16:0/373473344 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3500ade90 con 0x5558e707e5c0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd342a00700 1 -- 172.21.15.16:0/373473344 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 67) v1 ==== 7530+0+0 (secure 0 0 0) 0x7fd3500b2c90 con 0x5558e707e5c0 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd3778b6b80 10 client.16602 fetch_fsmap finished waiting for FSMap version 67 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd3778b6b80 10 client.16602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd3778b6b80 20 client.16602 populate_metadata read hostname 'smithi016' 2024-04-05T23:43:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5558e70ec3f0 0x5558e70ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd3778b6b80 4 client.16602 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:43:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd3778b6b80 1 -- 172.21.15.16:0/373473344 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5558e6e1dc90 con 0x5558e70ec3f0 2024-04-05T23:43:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.605+0000 7fd360a00700 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5558e70ec3f0 0x5558e70ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.606+0000 7fd360a00700 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5558e70ec3f0 0x5558e70ee7e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=764328b00d3d946d server_cookie=c10ff613f98c72a8 in_seq=0 out_seq=0 2024-04-05T23:43:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.606+0000 7fd342a00700 10 client.16602 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.637+0000 7fd342a00700 1 -- 172.21.15.16:0/373473344 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5558e6e1dc90 con 0x5558e70ec3f0 2024-04-05T23:43:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.637+0000 7fd342a00700 10 client.16602 handle_command_reply: tid=0 2024-04-05T23:43:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.637+0000 7fd3778b6b80 1 client.16602 shutdown 2024-04-05T23:43:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.637+0000 7fd3778b6b80 1 -- 172.21.15.16:0/373473344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 msgr2=0x5558e708af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.637+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e708af30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd350000c00 tx=0x7fd35009a6c0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd3778b6b80 1 -- 172.21.15.16:0/373473344 shutdown_connections 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5558e70ec3f0 0x5558e70ee7e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5558e707e5c0 0x5558e708af30 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5558e70844a0 0x5558e708b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd342a00700 0 client.16602 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd3778b6b80 1 --2- 172.21.15.16:0/373473344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5558e7083b80 0x5558e708b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd3778b6b80 1 -- 172.21.15.16:0/373473344 >> 172.21.15.16:0/373473344 conn(0x5558e6ff21c0 msgr2=0x5558e7082900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd3778b6b80 1 -- 172.21.15.16:0/373473344 shutdown_connections 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd3778b6b80 1 -- 172.21.15.16:0/373473344 wait complete. 2024-04-05T23:43:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.638+0000 7fd3778b6b80 20 client.16602 trim_cache size 0 max 16384 2024-04-05T23:43:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:43:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:43:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:43:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 -- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd3440a1290 msgr2=0x7fd3440a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd3440a1290 0x7fd3440a3750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd35c19a9e0 tx=0x7fd348005fd0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 -- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd35c073310 msgr2=0x7fd35c199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd35c073310 0x7fd35c199490 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd35000e9a0 tx=0x7fd35000ecb0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 -- 172.21.15.16:0/2635293365 shutdown_connections 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd3440a1290 0x7fd3440a3750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd35c073310 0x7fd35c199490 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd35c10d600 0x7fd35c19dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 --2- 172.21.15.16:0/2635293365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd35c073c30 0x7fd35c1999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 -- 172.21.15.16:0/2635293365 >> 172.21.15.16:0/2635293365 conn(0x7fd35c0fc8b0 msgr2=0x7fd35c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 -- 172.21.15.16:0/2635293365 shutdown_connections 2024-04-05T23:43:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:25.639+0000 7fd363e00700 1 -- 172.21.15.16:0/2635293365 wait complete. 2024-04-05T23:43:26.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 470890 4.70 MB/sec execute 405 sec latency 7626.597 ms 2024-04-05T23:43:26.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:26 smithi016 ceph-mon[38248]: pgmap v309: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 2.5 MiB/s wr, 90 op/s 2024-04-05T23:43:26.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:26 smithi130 ceph-mon[79417]: pgmap v309: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 2.5 MiB/s wr, 90 op/s 2024-04-05T23:43:26.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:26 smithi027 ceph-mon[78597]: pgmap v309: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 2.5 MiB/s wr, 90 op/s 2024-04-05T23:43:27.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 472780 4.73 MB/sec execute 406 sec latency 15.314 ms 2024-04-05T23:43:28.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 473992 4.73 MB/sec execute 407 sec latency 19.769 ms 2024-04-05T23:43:28.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:28 smithi016 ceph-mon[38248]: pgmap v310: 97 pgs: 97 active+clean; 187 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 2.5 MiB/s wr, 108 op/s 2024-04-05T23:43:28.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:28 smithi130 ceph-mon[79417]: pgmap v310: 97 pgs: 97 active+clean; 187 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 2.5 MiB/s wr, 108 op/s 2024-04-05T23:43:28.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:28 smithi027 ceph-mon[78597]: pgmap v310: 97 pgs: 97 active+clean; 187 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 2.5 MiB/s wr, 108 op/s 2024-04-05T23:43:29.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 475914 4.74 MB/sec execute 408 sec latency 15.853 ms 2024-04-05T23:43:30.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 477160 4.74 MB/sec execute 409 sec latency 13.970 ms 2024-04-05T23:43:30.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:30 smithi016 ceph-mon[38248]: pgmap v311: 97 pgs: 97 active+clean; 187 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-04-05T23:43:30.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:30 smithi130 ceph-mon[79417]: pgmap v311: 97 pgs: 97 active+clean; 187 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-04-05T23:43:30.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:30 smithi027 ceph-mon[78597]: pgmap v311: 97 pgs: 97 active+clean; 187 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-04-05T23:43:31.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 479099 4.76 MB/sec execute 410 sec latency 18.727 ms 2024-04-05T23:43:31.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:31 smithi027 ceph-mon[78597]: pgmap v312: 97 pgs: 97 active+clean; 192 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-04-05T23:43:32.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 480459 4.76 MB/sec execute 411 sec latency 13.689 ms 2024-04-05T23:43:32.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:31 smithi016 ceph-mon[38248]: pgmap v312: 97 pgs: 97 active+clean; 192 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-04-05T23:43:32.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:31 smithi130 ceph-mon[79417]: pgmap v312: 97 pgs: 97 active+clean; 192 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-04-05T23:43:33.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 482159 4.77 MB/sec execute 412 sec latency 18.956 ms 2024-04-05T23:43:34.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 483661 4.79 MB/sec execute 413 sec latency 15.497 ms 2024-04-05T23:43:34.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:33 smithi016 ceph-mon[38248]: pgmap v313: 97 pgs: 97 active+clean; 192 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-05T23:43:34.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:33 smithi130 ceph-mon[79417]: pgmap v313: 97 pgs: 97 active+clean; 192 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-05T23:43:34.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:33 smithi027 ceph-mon[78597]: pgmap v313: 97 pgs: 97 active+clean; 192 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-05T23:43:35.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 485359 4.79 MB/sec execute 414 sec latency 18.539 ms 2024-04-05T23:43:36.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 487062 4.81 MB/sec execute 415 sec latency 12.998 ms 2024-04-05T23:43:36.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:35 smithi016 ceph-mon[38248]: pgmap v314: 97 pgs: 97 active+clean; 192 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-05T23:43:36.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:35 smithi130 ceph-mon[79417]: pgmap v314: 97 pgs: 97 active+clean; 192 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-05T23:43:36.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:35 smithi027 ceph-mon[78597]: pgmap v314: 97 pgs: 97 active+clean; 192 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-04-05T23:43:37.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 488420 4.81 MB/sec execute 416 sec latency 19.039 ms 2024-04-05T23:43:38.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 490446 4.83 MB/sec execute 417 sec latency 16.246 ms 2024-04-05T23:43:38.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:37 smithi016 ceph-mon[38248]: pgmap v315: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-05T23:43:38.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:37 smithi130 ceph-mon[79417]: pgmap v315: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-05T23:43:38.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:37 smithi027 ceph-mon[78597]: pgmap v315: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-05T23:43:38.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.818+0000 7fe548c00700 1 -- 172.21.15.16:0/2916306307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe544101040 msgr2=0x7fe544101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:38.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.818+0000 7fe548c00700 1 --2- 172.21.15.16:0/2916306307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe544101040 0x7fe544101420 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe534009990 tx=0x7fe534009ca0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.818+0000 7fe548c00700 1 -- 172.21.15.16:0/2916306307 shutdown_connections 2024-04-05T23:43:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.818+0000 7fe548c00700 1 --2- 172.21.15.16:0/2916306307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe544106160 0x7fe54410dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.818+0000 7fe548c00700 1 --2- 172.21.15.16:0/2916306307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe544101040 0x7fe544101420 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.818+0000 7fe548c00700 1 --2- 172.21.15.16:0/2916306307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5441019f0 0x7fe544105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.818+0000 7fe548c00700 1 -- 172.21.15.16:0/2916306307 >> 172.21.15.16:0/2916306307 conn(0x7fe5440fc8e0 msgr2=0x7fe5440fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.818+0000 7fe548c00700 1 -- 172.21.15.16:0/2916306307 shutdown_connections 2024-04-05T23:43:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.818+0000 7fe548c00700 1 -- 172.21.15.16:0/2916306307 wait complete. 2024-04-05T23:43:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.819+0000 7fe548c00700 1 Processor -- start 2024-04-05T23:43:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe548c00700 1 -- start start 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe548c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe544101040 0x7fe5440731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe548c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe544106160 0x7fe544073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe548c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe544073c60 0x7fe5441a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe548c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5441a5b20 con 0x7fe544106160 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe548c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5441a5c90 con 0x7fe544073c60 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe548c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5441a5e50 con 0x7fe544101040 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe542800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe544073c60 0x7fe5441a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe542800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe544073c60 0x7fe5441a5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:40104/0 (socket says 172.21.15.16:40104) 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe542800700 1 -- 172.21.15.16:0/877212585 learned_addr learned my addr 172.21.15.16:0/877212585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe541400700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe544106160 0x7fe544073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.820+0000 7fe541e00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe544101040 0x7fe5440731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe542800700 1 -- 172.21.15.16:0/877212585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe544101040 msgr2=0x7fe5440731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe542800700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe544101040 0x7fe5440731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe542800700 1 -- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe544106160 msgr2=0x7fe544073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe542800700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe544106160 0x7fe544073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe542800700 1 -- 172.21.15.16:0/877212585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5340095f0 con 0x7fe544073c60 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe541e00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe544101040 0x7fe5440731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe542800700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe544073c60 0x7fe5441a5520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00ef30 tx=0x7fe53c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe53aa00700 1 -- 172.21.15.16:0/877212585 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c018070 con 0x7fe544073c60 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe53aa00700 1 -- 172.21.15.16:0/877212585 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe53c00f040 con 0x7fe544073c60 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe548c00700 1 -- 172.21.15.16:0/877212585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5441a60d0 con 0x7fe544073c60 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.821+0000 7fe53aa00700 1 -- 172.21.15.16:0/877212585 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c00cdb0 con 0x7fe544073c60 2024-04-05T23:43:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.822+0000 7fe548c00700 1 -- 172.21.15.16:0/877212585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5441a6630 con 0x7fe544073c60 2024-04-05T23:43:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.824+0000 7fe53aa00700 1 -- 172.21.15.16:0/877212585 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fe53c003a00 con 0x7fe544073c60 2024-04-05T23:43:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.826+0000 7fe53aa00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe5280a1310 0x7fe5280a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.826+0000 7fe53aa00700 1 -- 172.21.15.16:0/877212585 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe53c0cc860 con 0x7fe544073c60 2024-04-05T23:43:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.826+0000 7fe541e00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe5280a1310 0x7fe5280a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.827+0000 7fe55c76eb80 1 Processor -- start 2024-04-05T23:43:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.827+0000 7fe541e00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe5280a1310 0x7fe5280a37d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe534006010 tx=0x7fe53400b260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:38.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.827+0000 7fe55c76eb80 1 -- start start 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abf9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abf4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 0x55a15abf4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15ab5ab80 con 0x55a15abfa4a0 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15ab4b930 con 0x55a15abf9b80 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15ab492f0 con 0x55a15abf45c0 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe541e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abf9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe541400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abf4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe541e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abf9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:40112/0 (socket says 172.21.15.16:40112) 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe541e00700 1 -- 172.21.15.16:0/2728166711 learned_addr learned my addr 172.21.15.16:0/2728166711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe542800700 1 --2- 172.21.15.16:0/2728166711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 0x55a15abf4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe541e00700 1 -- 172.21.15.16:0/2728166711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 msgr2=0x55a15abf4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe541e00700 1 --2- 172.21.15.16:0/2728166711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 0x55a15abf4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe541e00700 1 -- 172.21.15.16:0/2728166711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 msgr2=0x55a15abf4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe541e00700 1 --2- 172.21.15.16:0/2728166711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abf4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.828+0000 7fe541e00700 1 -- 172.21.15.16:0/2728166711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a15a9c6350 con 0x55a15abf9b80 2024-04-05T23:43:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.829+0000 7fe541e00700 1 --2- 172.21.15.16:0/2728166711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abf9f60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe534005f30 tx=0x7fe53401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d074c98a04e74634 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.829+0000 7fe52f000700 1 -- 172.21.15.16:0/2728166711 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53402b070 con 0x55a15abf9b80 2024-04-05T23:43:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.829+0000 7fe52f000700 1 -- 172.21.15.16:0/2728166711 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe53401e040 con 0x55a15abf9b80 2024-04-05T23:43:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.829+0000 7fe52f000700 1 -- 172.21.15.16:0/2728166711 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534004cb0 con 0x55a15abf9b80 2024-04-05T23:43:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.830+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2728166711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 msgr2=0x55a15abf9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.830+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2728166711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abf9f60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe534005f30 tx=0x7fe53401f000 comp rx=0 tx=0).stop 2024-04-05T23:43:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.830+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2728166711 shutdown_connections 2024-04-05T23:43:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.830+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2728166711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 0x55a15abf4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.830+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2728166711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abf9f60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.830+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2728166711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abf4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.830+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2728166711 >> 172.21.15.16:0/2728166711 conn(0x55a15ab681c0 msgr2=0x55a15abeebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.830+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2728166711 shutdown_connections 2024-04-05T23:43:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.830+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2728166711 wait complete. 2024-04-05T23:43:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.831+0000 7fe55c76eb80 1 -- start start 2024-04-05T23:43:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.831+0000 7fe55c76eb80 1 Processor -- start 2024-04-05T23:43:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac00480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf9b80 0x55a15abfd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abfdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15ab44be0 con 0x55a15abfa4a0 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15aae0d20 con 0x55a15abf45c0 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15aae09c0 con 0x55a15abf9b80 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe55c76eb80 10 client.0 _ll_register_callbacks cb 0x55a15a94c340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf9b80 0x55a15abfd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac00480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac00480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:40122/0 (socket says 172.21.15.16:40122) 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf9b80 0x55a15abfd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:51170/0 (socket says 172.21.15.16:51170) 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541e00700 1 -- 172.21.15.16:0/2433715665 learned_addr learned my addr 172.21.15.16:0/2433715665 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe542800700 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abfdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541e00700 1 -- 172.21.15.16:0/2433715665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf9b80 msgr2=0x55a15abfd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541e00700 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf9b80 0x55a15abfd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541e00700 1 -- 172.21.15.16:0/2433715665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 msgr2=0x55a15abfdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541e00700 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abfdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe541e00700 1 -- 172.21.15.16:0/2433715665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54410e550 con 0x55a15abf45c0 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.832+0000 7fe542800700 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abfdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.833+0000 7fe541e00700 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac00480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe534005a40 tx=0x7fe534005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1ba8c03b595fd30 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.833+0000 7fe523400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.833+0000 7fe523400700 1 -- 172.21.15.16:0/2433715665 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53402b040 con 0x55a15abf45c0 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.833+0000 7fe523400700 1 -- 172.21.15.16:0/2433715665 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe534004ce0 con 0x55a15abf45c0 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.833+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2433715665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a15abe71e0 con 0x55a15abf45c0 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.833+0000 7fe523400700 1 -- 172.21.15.16:0/2433715665 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe53400f460 con 0x55a15abf45c0 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.833+0000 7fe55c76eb80 10 client.26666 fetch_fsmap learned FSMap version 67 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.833+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2433715665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) v3 -- 0x55a15a972b00 con 0x55a15abf45c0 2024-04-05T23:43:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.833+0000 7fe523400700 1 -- 172.21.15.16:0/2433715665 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe534044080 con 0x55a15abf45c0 2024-04-05T23:43:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.834+0000 7fe523400700 1 -- 172.21.15.16:0/2433715665 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 67) v1 ==== 7530+0+0 (secure 0 0 0) 0x7fe534024610 con 0x55a15abf45c0 2024-04-05T23:43:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.834+0000 7fe55c76eb80 10 client.26666 fetch_fsmap finished waiting for FSMap version 67 2024-04-05T23:43:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.834+0000 7fe55c76eb80 10 client.26666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:43:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.834+0000 7fe55c76eb80 20 client.26666 populate_metadata read hostname 'smithi016' 2024-04-05T23:43:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.834+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a15ac16260 0x55a15ac18680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.834+0000 7fe55c76eb80 4 client.26666 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:43:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.834+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2433715665 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a15ab3d6a0 con 0x55a15ac16260 2024-04-05T23:43:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.834+0000 7fe541400700 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a15ac16260 0x55a15ac18680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.834+0000 7fe541400700 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a15ac16260 0x55a15ac18680 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=806f258c0059cdc1 server_cookie=76927390377a1b24 in_seq=0 out_seq=0 2024-04-05T23:43:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.835+0000 7fe523400700 10 client.26666 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.835+0000 7fe523400700 1 -- 172.21.15.16:0/2433715665 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a15ab3d6a0 con 0x55a15ac16260 2024-04-05T23:43:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.835+0000 7fe523400700 10 client.26666 handle_command_reply: tid=0 2024-04-05T23:43:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.835+0000 7fe55c76eb80 1 client.26666 shutdown 2024-04-05T23:43:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.835+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2433715665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 msgr2=0x55a15ac00480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.835+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac00480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe534005a40 tx=0x7fe534005e80 comp rx=0 tx=0).stop 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2433715665 shutdown_connections 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a15ac16260 0x55a15ac18680 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abfdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac00480 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2433715665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf9b80 0x55a15abfd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe523400700 0 client.26666 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2433715665 >> 172.21.15.16:0/2433715665 conn(0x55a15ab681c0 msgr2=0x55a15abee280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2433715665 shutdown_connections 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2433715665 wait complete. 2024-04-05T23:43:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.836+0000 7fe55c76eb80 20 client.26666 trim_cache size 0 max 16384 2024-04-05T23:43:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.858+0000 7fe55c76eb80 1 Processor -- start 2024-04-05T23:43:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.858+0000 7fe55c76eb80 1 -- start start 2024-04-05T23:43:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.858+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 0x55a15abf4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.858+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abeea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.858+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abf7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.858+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15a9c8640 con 0x55a15abfa4a0 2024-04-05T23:43:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.858+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15abf84c0 con 0x55a15abf9b80 2024-04-05T23:43:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.858+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15ac18d30 con 0x55a15abf45c0 2024-04-05T23:43:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abeea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 0x55a15abf4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abeea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:40126/0 (socket says 172.21.15.16:40126) 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 0x55a15abf4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:51186/0 (socket says 172.21.15.16:51186) 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541400700 1 -- 172.21.15.16:0/4067736247 learned_addr learned my addr 172.21.15.16:0/4067736247 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe542800700 1 --2- 172.21.15.16:0/4067736247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abf7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541400700 1 -- 172.21.15.16:0/4067736247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 msgr2=0x55a15abf4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541400700 1 --2- 172.21.15.16:0/4067736247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 0x55a15abf4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541400700 1 -- 172.21.15.16:0/4067736247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 msgr2=0x55a15abf7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541400700 1 --2- 172.21.15.16:0/4067736247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abf7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.859+0000 7fe541400700 1 -- 172.21.15.16:0/4067736247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a15ab0ae40 con 0x55a15abf9b80 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.860+0000 7fe541400700 1 --2- 172.21.15.16:0/4067736247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abeea00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe530004790 tx=0x7fe530009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc274df880458185 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.860+0000 7fe52fa00700 1 -- 172.21.15.16:0/4067736247 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530017030 con 0x55a15abf9b80 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.860+0000 7fe52fa00700 1 -- 172.21.15.16:0/4067736247 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe530018040 con 0x55a15abf9b80 2024-04-05T23:43:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.860+0000 7fe52fa00700 1 -- 172.21.15.16:0/4067736247 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530003680 con 0x55a15abf9b80 2024-04-05T23:43:38.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 -- 172.21.15.16:0/4067736247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 msgr2=0x55a15abeea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:38.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/4067736247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abeea00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe530004790 tx=0x7fe530009260 comp rx=0 tx=0).stop 2024-04-05T23:43:38.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 -- 172.21.15.16:0/4067736247 shutdown_connections 2024-04-05T23:43:38.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/4067736247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abfa4a0 0x55a15abf7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/4067736247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abf45c0 0x55a15abf4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/4067736247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf9b80 0x55a15abeea00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 -- 172.21.15.16:0/4067736247 >> 172.21.15.16:0/4067736247 conn(0x55a15ab681c0 msgr2=0x55a15ac00120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:38.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 -- 172.21.15.16:0/4067736247 shutdown_connections 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 -- 172.21.15.16:0/4067736247 wait complete. 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 -- start start 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.861+0000 7fe55c76eb80 1 Processor -- start 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.862+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac0d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.862+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abf9b80 0x55a15abf5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.862+0000 7fe55c76eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abfa4a0 0x55a15abf63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.862+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15abf6ac0 con 0x55a15abf9b80 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.862+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15ac6a050 con 0x55a15abf45c0 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.862+0000 7fe55c76eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15ac6a190 con 0x55a15abfa4a0 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.862+0000 7fe55c76eb80 10 client.0 _ll_register_callbacks cb 0x7fe530009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.862+0000 7fe542800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abfa4a0 0x55a15abf63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abf9b80 0x55a15abf5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac0d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abf9b80 0x55a15abf5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57490/0 (socket says 172.21.15.16:57490) 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe542800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abfa4a0 0x55a15abf63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:51194/0 (socket says 172.21.15.16:51194) 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541400700 1 -- 172.21.15.16:0/2640573835 learned_addr learned my addr 172.21.15.16:0/2640573835 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541e00700 1 -- 172.21.15.16:0/2640573835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abfa4a0 msgr2=0x55a15abf63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541e00700 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abfa4a0 0x55a15abf63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541e00700 1 -- 172.21.15.16:0/2640573835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abf9b80 msgr2=0x55a15abf5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541e00700 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abf9b80 0x55a15abf5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541e00700 1 -- 172.21.15.16:0/2640573835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe530010430 con 0x55a15abf45c0 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe541e00700 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac0d920 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe534024940 tx=0x7fe534004f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df697ee99073e34c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe523400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe523400700 1 -- 172.21.15.16:0/2640573835 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53402b040 con 0x55a15abf45c0 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.863+0000 7fe523400700 1 -- 172.21.15.16:0/2640573835 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe534024c30 con 0x55a15abf45c0 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2640573835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe53401e040 con 0x55a15abf45c0 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe523400700 1 -- 172.21.15.16:0/2640573835 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe53400be10 con 0x55a15abf45c0 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe55c76eb80 10 client.26672 fetch_fsmap learned FSMap version 67 2024-04-05T23:43:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2640573835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) v3 -- 0x55a15a9cac70 con 0x55a15abf45c0 2024-04-05T23:43:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe523400700 1 -- 172.21.15.16:0/2640573835 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe53404f020 con 0x55a15abf45c0 2024-04-05T23:43:38.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe523400700 1 -- 172.21.15.16:0/2640573835 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 67) v1 ==== 7530+0+0 (secure 0 0 0) 0x7fe53404b430 con 0x55a15abf45c0 2024-04-05T23:43:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe55c76eb80 10 client.26672 fetch_fsmap finished waiting for FSMap version 67 2024-04-05T23:43:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe55c76eb80 10 client.26672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:43:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe55c76eb80 20 client.26672 populate_metadata read hostname 'smithi016' 2024-04-05T23:43:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a15ac6b570 0x55a15ac6d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe55c76eb80 4 client.26672 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:43:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.864+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2640573835 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a15a993c90 con 0x55a15ac6b570 2024-04-05T23:43:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.865+0000 7fe541400700 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a15ac6b570 0x55a15ac6d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.865+0000 7fe541400700 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a15ac6b570 0x55a15ac6d990 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=123b9ec3477a435e server_cookie=ff77b7c11ba6fa9e in_seq=0 out_seq=0 2024-04-05T23:43:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.865+0000 7fe523400700 10 client.26672 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:38.883 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.891+0000 7fe523400700 1 -- 172.21.15.16:0/2640573835 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a15a993c90 con 0x55a15ac6b570 2024-04-05T23:43:38.883 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.891+0000 7fe523400700 10 client.26672 handle_command_reply: tid=0 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.891+0000 7fe55c76eb80 1 client.26672 shutdown 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.891+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2640573835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 msgr2=0x55a15ac0d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.891+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac0d920 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe534024940 tx=0x7fe534004f20 comp rx=0 tx=0).stop 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2640573835 shutdown_connections 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55a15ac6b570 0x55a15ac6d990 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a15abfa4a0 0x55a15abf63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe523400700 0 client.26672 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a15abf45c0 0x55a15ac0d920 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe55c76eb80 1 --2- 172.21.15.16:0/2640573835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a15abf9b80 0x55a15abf5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2640573835 >> 172.21.15.16:0/2640573835 conn(0x55a15ab681c0 msgr2=0x55a15abf7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2640573835 shutdown_connections 2024-04-05T23:43:38.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe55c76eb80 1 -- 172.21.15.16:0/2640573835 wait complete. 2024-04-05T23:43:38.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.892+0000 7fe55c76eb80 20 client.26672 trim_cache size 0 max 16384 2024-04-05T23:43:38.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:43:38.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:43:38.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:43:38.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 -- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe5280a1310 msgr2=0x7fe5280a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe5280a1310 0x7fe5280a37d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe534006010 tx=0x7fe53400b260 comp rx=0 tx=0).stop 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 -- 172.21.15.16:0/877212585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe544073c60 msgr2=0x7fe5441a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe544073c60 0x7fe5441a5520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00ef30 tx=0x7fe53c00c3a0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 -- 172.21.15.16:0/877212585 shutdown_connections 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe5280a1310 0x7fe5280a37d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe544073c60 0x7fe5441a5520 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe544101040 0x7fe5440731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 --2- 172.21.15.16:0/877212585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe544106160 0x7fe544073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 -- 172.21.15.16:0/877212585 >> 172.21.15.16:0/877212585 conn(0x7fe5440fc8e0 msgr2=0x7fe5440fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:38.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 -- 172.21.15.16:0/877212585 shutdown_connections 2024-04-05T23:43:38.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:38.893+0000 7fe548c00700 1 -- 172.21.15.16:0/877212585 wait complete. 2024-04-05T23:43:39.024 INFO:tasks.workunit.client.0.smithi016.stdout: 1 491496 4.82 MB/sec execute 418 sec latency 23.660 ms 2024-04-05T23:43:40.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 493477 4.84 MB/sec execute 419 sec latency 19.880 ms 2024-04-05T23:43:40.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:39 smithi016 ceph-mon[38248]: pgmap v316: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-04-05T23:43:40.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:39 smithi130 ceph-mon[79417]: pgmap v316: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-04-05T23:43:40.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:39 smithi027 ceph-mon[78597]: pgmap v316: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-04-05T23:43:41.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 494760 4.84 MB/sec execute 420 sec latency 11.171 ms 2024-04-05T23:43:42.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 496610 4.86 MB/sec execute 421 sec latency 18.813 ms 2024-04-05T23:43:42.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:41 smithi016 ceph-mon[38248]: pgmap v317: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2024-04-05T23:43:42.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:41 smithi130 ceph-mon[79417]: pgmap v317: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2024-04-05T23:43:42.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:41 smithi027 ceph-mon[78597]: pgmap v317: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2024-04-05T23:43:43.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 497993 4.86 MB/sec execute 422 sec latency 11.919 ms 2024-04-05T23:43:44.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 499585 4.87 MB/sec execute 423 sec latency 36.321 ms 2024-04-05T23:43:44.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:43 smithi016 ceph-mon[38248]: pgmap v318: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-05T23:43:44.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:43 smithi130 ceph-mon[79417]: pgmap v318: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-05T23:43:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:43 smithi027 ceph-mon[78597]: pgmap v318: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-05T23:43:45.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 501238 4.89 MB/sec execute 424 sec latency 15.093 ms 2024-04-05T23:43:46.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 502563 4.89 MB/sec execute 425 sec latency 19.007 ms 2024-04-05T23:43:46.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:45 smithi016 ceph-mon[38248]: pgmap v319: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-05T23:43:46.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:45 smithi130 ceph-mon[79417]: pgmap v319: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-05T23:43:46.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:45 smithi027 ceph-mon[78597]: pgmap v319: 97 pgs: 97 active+clean; 200 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-05T23:43:47.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 504577 4.90 MB/sec execute 426 sec latency 20.156 ms 2024-04-05T23:43:48.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 505670 4.90 MB/sec execute 427 sec latency 12.417 ms 2024-04-05T23:43:48.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:47 smithi016 ceph-mon[38248]: pgmap v320: 97 pgs: 97 active+clean; 207 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 292 op/s 2024-04-05T23:43:48.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:47 smithi130 ceph-mon[79417]: pgmap v320: 97 pgs: 97 active+clean; 207 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 292 op/s 2024-04-05T23:43:48.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:47 smithi027 ceph-mon[78597]: pgmap v320: 97 pgs: 97 active+clean; 207 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 292 op/s 2024-04-05T23:43:49.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 506183 4.90 MB/sec execute 428 sec latency 678.703 ms 2024-04-05T23:43:50.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 506183 4.89 MB/sec execute 429 sec latency 1678.805 ms 2024-04-05T23:43:50.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:49 smithi016 ceph-mon[38248]: pgmap v321: 97 pgs: 97 active+clean; 207 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 198 op/s 2024-04-05T23:43:50.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:49 smithi130 ceph-mon[79417]: pgmap v321: 97 pgs: 97 active+clean; 207 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 198 op/s 2024-04-05T23:43:50.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:49 smithi027 ceph-mon[78597]: pgmap v321: 97 pgs: 97 active+clean; 207 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 198 op/s 2024-04-05T23:43:51.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 506183 4.87 MB/sec execute 430 sec latency 2678.890 ms 2024-04-05T23:43:51.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:50 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:43:51.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:50 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:43:51.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:50 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:43:52.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 506868 4.87 MB/sec execute 431 sec latency 3390.351 ms 2024-04-05T23:43:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.077+0000 7fe3c9000700 1 -- 172.21.15.16:0/3691268277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3c4111de0 msgr2=0x7fe3c41141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.077+0000 7fe3c9000700 1 --2- 172.21.15.16:0/3691268277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3c4111de0 0x7fe3c41141d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009f90 tx=0x7fe3bc00b330 comp rx=0 tx=0).stop 2024-04-05T23:43:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.077+0000 7fe3c9000700 1 -- 172.21.15.16:0/3691268277 shutdown_connections 2024-04-05T23:43:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.077+0000 7fe3c9000700 1 --2- 172.21.15.16:0/3691268277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3c4073280 0x7fe3c4073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.077+0000 7fe3c9000700 1 --2- 172.21.15.16:0/3691268277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3c4111de0 0x7fe3c41141d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.077+0000 7fe3c9000700 1 --2- 172.21.15.16:0/3691268277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c4073ba0 0x7fe3c41118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.077+0000 7fe3c9000700 1 -- 172.21.15.16:0/3691268277 >> 172.21.15.16:0/3691268277 conn(0x7fe3c40fc8e0 msgr2=0x7fe3c40fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.077+0000 7fe3c9000700 1 -- 172.21.15.16:0/3691268277 shutdown_connections 2024-04-05T23:43:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.077+0000 7fe3c9000700 1 -- 172.21.15.16:0/3691268277 wait complete. 2024-04-05T23:43:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.079+0000 7fe3c9000700 1 Processor -- start 2024-04-05T23:43:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.079+0000 7fe3c9000700 1 -- start start 2024-04-05T23:43:52.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c9000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3c4073280 0x7fe3c41113c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c9000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3c4073ba0 0x7fe3c410ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c9000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c4111de0 0x7fe3c410f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c9000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c410f990 con 0x7fe3c4073280 2024-04-05T23:43:52.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c9000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c410fb00 con 0x7fe3c4111de0 2024-04-05T23:43:52.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c9000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c410fcc0 con 0x7fe3c4073ba0 2024-04-05T23:43:52.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3c4073280 0x7fe3c41113c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3c4073ba0 0x7fe3c410ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c4111de0 0x7fe3c410f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3c4073ba0 0x7fe3c410ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:50462/0 (socket says 172.21.15.16:50462) 2024-04-05T23:43:52.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c1400700 1 -- 172.21.15.16:0/991392066 learned_addr learned my addr 172.21.15.16:0/991392066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:52.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3c4073280 0x7fe3c41113c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44692/0 (socket says 172.21.15.16:44692) 2024-04-05T23:43:52.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c2800700 1 -- 172.21.15.16:0/991392066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3c4073ba0 msgr2=0x7fe3c410ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:52.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c2800700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3c4073ba0 0x7fe3c410ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c2800700 1 -- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3c4073280 msgr2=0x7fe3c41113c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:52.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.080+0000 7fe3c2800700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3c4073280 0x7fe3c41113c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.081+0000 7fe3c2800700 1 -- 172.21.15.16:0/991392066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b40095f0 con 0x7fe3c4111de0 2024-04-05T23:43:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.081+0000 7fe3c2800700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c4111de0 0x7fe3c410f450 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00b6c0 tx=0x7fe3bc0094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.081+0000 7fe3baa00700 1 -- 172.21.15.16:0/991392066 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00e040 con 0x7fe3c4111de0 2024-04-05T23:43:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.081+0000 7fe3c9000700 1 -- 172.21.15.16:0/991392066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc009c40 con 0x7fe3c4111de0 2024-04-05T23:43:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.081+0000 7fe3baa00700 1 -- 172.21.15.16:0/991392066 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3bc01b450 con 0x7fe3c4111de0 2024-04-05T23:43:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.081+0000 7fe3baa00700 1 -- 172.21.15.16:0/991392066 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc0052c0 con 0x7fe3c4111de0 2024-04-05T23:43:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.081+0000 7fe3c9000700 1 -- 172.21.15.16:0/991392066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c410a8f0 con 0x7fe3c4111de0 2024-04-05T23:43:52.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.086+0000 7fe3baa00700 1 -- 172.21.15.16:0/991392066 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fe3bc01b5c0 con 0x7fe3c4111de0 2024-04-05T23:43:52.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.087+0000 7fe3dcb51b80 1 Processor -- start 2024-04-05T23:43:52.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.087+0000 7fe3dcb51b80 1 -- start start 2024-04-05T23:43:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.087+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 0x55e74389cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.087+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 0x55e743897030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 0x55e743897a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7437fdb80 con 0x55e74389d4a0 2024-04-05T23:43:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7437ee930 con 0x55e74389cb80 2024-04-05T23:43:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7437ec2f0 con 0x55e7438975c0 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3c1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 0x55e743897030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3c1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 0x55e743897030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44706/0 (socket says 172.21.15.16:44706) 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3c1e00700 1 -- 172.21.15.16:0/1204701745 learned_addr learned my addr 172.21.15.16:0/1204701745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3baa00700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3a80a1310 0x7fe3a80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3baa00700 1 -- 172.21.15.16:0/991392066 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe3bc0ce0a0 con 0x7fe3c4111de0 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3c1e00700 1 -- 172.21.15.16:0/1204701745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 msgr2=0x55e743897a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/1204701745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 0x55e743897a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3c1e00700 1 -- 172.21.15.16:0/1204701745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 msgr2=0x55e74389cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/1204701745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 0x55e74389cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.088+0000 7fe3c1e00700 1 -- 172.21.15.16:0/1204701745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e743669350 con 0x55e74389d4a0 2024-04-05T23:43:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.089+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/1204701745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 0x55e743897030 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009f90 tx=0x7fe3b4011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e1cb72f46402af server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.089+0000 7fe3af400700 1 -- 172.21.15.16:0/1204701745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b400e670 con 0x55e74389d4a0 2024-04-05T23:43:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.089+0000 7fe3af400700 1 -- 172.21.15.16:0/1204701745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3b400ec90 con 0x55e74389d4a0 2024-04-05T23:43:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.089+0000 7fe3af400700 1 -- 172.21.15.16:0/1204701745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4019d60 con 0x55e74389d4a0 2024-04-05T23:43:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.089+0000 7fe3c2800700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3a80a1310 0x7fe3a80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.090+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1204701745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 msgr2=0x55e743897030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.090+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1204701745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 0x55e743897030 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009f90 tx=0x7fe3b4011720 comp rx=0 tx=0).stop 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.091+0000 7fe3c2800700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3a80a1310 0x7fe3a80a37d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc09b150 tx=0x7fe3bc009040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.091+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1204701745 shutdown_connections 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.091+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1204701745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 0x55e743897a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.091+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1204701745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 0x55e74389cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.091+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1204701745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 0x55e743897030 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.091+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1204701745 >> 172.21.15.16:0/1204701745 conn(0x55e74380b1c0 msgr2=0x55e743891be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.091+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1204701745 shutdown_connections 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.091+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1204701745 wait complete. 2024-04-05T23:43:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.091+0000 7fe3dcb51b80 1 -- start start 2024-04-05T23:43:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.092+0000 7fe3dcb51b80 1 Processor -- start 2024-04-05T23:43:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7438975c0 0x55e7438a34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389cb80 0x55e7438a0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 0x55e7438a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7437e7be0 con 0x55e74389d4a0 2024-04-05T23:43:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e743783d20 con 0x55e7438975c0 2024-04-05T23:43:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7437839c0 con 0x55e74389cb80 2024-04-05T23:43:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3dcb51b80 10 client.0 _ll_register_callbacks cb 0x55e7435ef340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:43:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3c1e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389cb80 0x55e7438a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3c1e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389cb80 0x55e7438a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:50474/0 (socket says 172.21.15.16:50474) 2024-04-05T23:43:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3c1e00700 1 -- 172.21.15.16:0/1291001132 learned_addr learned my addr 172.21.15.16:0/1291001132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:52.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.093+0000 7fe3c1e00700 1 -- 172.21.15.16:0/1291001132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7438975c0 msgr2=0x55e7438a34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:52.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.094+0000 7fe3c1400700 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7438975c0 0x55e7438a34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.094+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7438975c0 0x55e7438a34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.094+0000 7fe3c1e00700 1 -- 172.21.15.16:0/1291001132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 msgr2=0x55e7438a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:52.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.094+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 0x55e7438a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.094+0000 7fe3c1e00700 1 -- 172.21.15.16:0/1291001132 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b4009bc0 con 0x55e74389cb80 2024-04-05T23:43:52.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.094+0000 7fe3c1400700 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7438975c0 0x55e7438a34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:43:52.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.095+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389cb80 0x55e7438a0830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4005950 tx=0x7fe3b4019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7971557419a693e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:52.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.095+0000 7fe3a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:43:52.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.095+0000 7fe3a3e00700 1 -- 172.21.15.16:0/1291001132 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4004260 con 0x55e74389cb80 2024-04-05T23:43:52.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.095+0000 7fe3a3e00700 1 -- 172.21.15.16:0/1291001132 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe3b40043c0 con 0x55e74389cb80 2024-04-05T23:43:52.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.095+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1291001132 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e74388a1e0 con 0x55e74389cb80 2024-04-05T23:43:52.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.096+0000 7fe3a3e00700 1 -- 172.21.15.16:0/1291001132 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3b4004580 con 0x55e74389cb80 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.096+0000 7fe3dcb51b80 10 client.26704 fetch_fsmap learned FSMap version 68 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.096+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1291001132 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x55e743615b00 con 0x55e74389cb80 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.096+0000 7fe3a3e00700 1 -- 172.21.15.16:0/1291001132 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e74388a1e0 con 0x55e74389cb80 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3a3e00700 1 -- 172.21.15.16:0/1291001132 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 68) v1 ==== 7542+0+0 (secure 0 0 0) 0x7fe3b4004750 con 0x55e74389cb80 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3dcb51b80 10 client.26704 fetch_fsmap finished waiting for FSMap version 68 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3dcb51b80 10 client.26704 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3dcb51b80 20 client.26704 populate_metadata read hostname 'smithi016' 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e7438c1f10 0x55e7438c4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3dcb51b80 4 client.26704 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1291001132 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7437e06a0 con 0x55e7438c1f10 2024-04-05T23:43:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3c1400700 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e7438c1f10 0x55e7438c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3c1400700 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e7438c1f10 0x55e7438c4330 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=137aa192e2214320 server_cookie=7c3ddba49148b462 in_seq=0 out_seq=0 2024-04-05T23:43:52.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.097+0000 7fe3a3e00700 10 client.26704 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.098+0000 7fe3a3e00700 1 -- 172.21.15.16:0/1291001132 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e7437e06a0 con 0x55e7438c1f10 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.098+0000 7fe3a3e00700 10 client.26704 handle_command_reply: tid=0 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 client.26704 shutdown 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1291001132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389cb80 msgr2=0x55e7438a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389cb80 0x55e7438a0830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4005950 tx=0x7fe3b4019660 comp rx=0 tx=0).stop 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1291001132 shutdown_connections 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e7438c1f10 0x55e7438c4330 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3a3e00700 0 client.26704 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389d4a0 0x55e7438a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7438975c0 0x55e7438a34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/1291001132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389cb80 0x55e7438a0830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1291001132 >> 172.21.15.16:0/1291001132 conn(0x55e74380b1c0 msgr2=0x55e743891280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1291001132 shutdown_connections 2024-04-05T23:43:52.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.099+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/1291001132 wait complete. 2024-04-05T23:43:52.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.100+0000 7fe3dcb51b80 20 client.26704 trim_cache size 0 max 16384 2024-04-05T23:43:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.123+0000 7fe3dcb51b80 1 Processor -- start 2024-04-05T23:43:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.123+0000 7fe3dcb51b80 1 -- start start 2024-04-05T23:43:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.123+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7438975c0 0x55e743897030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.123+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 0x55e7438a7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.123+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389d4a0 0x55e743891a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.123+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e74366b640 con 0x55e7438975c0 2024-04-05T23:43:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.123+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e74389e890 con 0x55e74389cb80 2024-04-05T23:43:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.123+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7438926d0 con 0x55e74389d4a0 2024-04-05T23:43:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 0x55e7438a7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7438975c0 0x55e743897030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 0x55e7438a7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42142/0 (socket says 172.21.15.16:42142) 2024-04-05T23:43:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1e00700 1 -- 172.21.15.16:0/3839275519 learned_addr learned my addr 172.21.15.16:0/3839275519 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1400700 1 --2- 172.21.15.16:0/3839275519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389d4a0 0x55e743891a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1e00700 1 -- 172.21.15.16:0/3839275519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389d4a0 msgr2=0x55e743891a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/3839275519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389d4a0 0x55e743891a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1e00700 1 -- 172.21.15.16:0/3839275519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7438975c0 msgr2=0x55e743897030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/3839275519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7438975c0 0x55e743897030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1e00700 1 -- 172.21.15.16:0/3839275519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7437ade40 con 0x55e74389cb80 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1400700 1 --2- 172.21.15.16:0/3839275519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389d4a0 0x55e743891a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.124+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/3839275519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 0x55e7438a7f80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009fe0 tx=0x7fe3b4004e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35d3c703ee0f4dab server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.125+0000 7fe3afe00700 1 -- 172.21.15.16:0/3839275519 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4039070 con 0x55e74389cb80 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.125+0000 7fe3afe00700 1 -- 172.21.15.16:0/3839275519 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3b4019870 con 0x55e74389cb80 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.125+0000 7fe3afe00700 1 -- 172.21.15.16:0/3839275519 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b40045e0 con 0x55e74389cb80 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/3839275519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 msgr2=0x55e7438a7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/3839275519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 0x55e7438a7f80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009fe0 tx=0x7fe3b4004e80 comp rx=0 tx=0).stop 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/3839275519 shutdown_connections 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/3839275519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e7438975c0 0x55e743897030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/3839275519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e74389d4a0 0x55e743891a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/3839275519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389cb80 0x55e7438a7f80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/3839275519 >> 172.21.15.16:0/3839275519 conn(0x55e74380b1c0 msgr2=0x55e74389a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:52.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/3839275519 shutdown_connections 2024-04-05T23:43:52.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/3839275519 wait complete. 2024-04-05T23:43:52.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.126+0000 7fe3dcb51b80 1 -- start start 2024-04-05T23:43:52.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.127+0000 7fe3dcb51b80 1 Processor -- start 2024-04-05T23:43:52.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.127+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 0x55e7438a8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.127+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389cb80 0x55e7438a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.127+0000 7fe3dcb51b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389d4a0 0x55e7438a65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.127+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7438a6c80 con 0x55e74389cb80 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.127+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7438a6df0 con 0x55e74389d4a0 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.127+0000 7fe3dcb51b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e743914d70 con 0x55e7438975c0 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.127+0000 7fe3dcb51b80 10 client.0 _ll_register_callbacks cb 0x7fe3b4009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 0x55e7438a8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 0x55e7438a8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:50500/0 (socket says 172.21.15.16:50500) 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 -- 172.21.15.16:0/2090457638 learned_addr learned my addr 172.21.15.16:0/2090457638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389d4a0 0x55e7438a65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389cb80 0x55e7438a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 -- 172.21.15.16:0/2090457638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389d4a0 msgr2=0x55e7438a65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389d4a0 0x55e7438a65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 -- 172.21.15.16:0/2090457638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389cb80 msgr2=0x55e7438a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:43:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389cb80 0x55e7438a5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 -- 172.21.15.16:0/2090457638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e743615b00 con 0x55e7438975c0 2024-04-05T23:43:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3c1400700 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 0x55e7438a8180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55e7438c4c40 tx=0x7fe3b0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=936238ad5d678d60 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:43:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:43:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.128+0000 7fe3a3e00700 1 -- 172.21.15.16:0/2090457638 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0004030 con 0x55e7438975c0 2024-04-05T23:43:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3a3e00700 1 -- 172.21.15.16:0/2090457638 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe3b0017070 con 0x55e7438975c0 2024-04-05T23:43:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/2090457638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3b400ec90 con 0x55e7438975c0 2024-04-05T23:43:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3a3e00700 1 -- 172.21.15.16:0/2090457638 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3b0004190 con 0x55e7438975c0 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3dcb51b80 10 client.26710 fetch_fsmap learned FSMap version 68 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/2090457638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x55e74366dc70 con 0x55e7438975c0 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3a3e00700 1 -- 172.21.15.16:0/2090457638 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3b400ec90 con 0x55e7438975c0 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3a3e00700 1 -- 172.21.15.16:0/2090457638 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 68) v1 ==== 7542+0+0 (secure 0 0 0) 0x7fe3b0012510 con 0x55e7438975c0 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3dcb51b80 10 client.26710 fetch_fsmap finished waiting for FSMap version 68 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3dcb51b80 10 client.26710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3dcb51b80 20 client.26710 populate_metadata read hostname 'smithi016' 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e743916640 0x55e743918a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3dcb51b80 4 client.26710 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:43:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.129+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/2090457638 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e743636c90 con 0x55e743916640 2024-04-05T23:43:52.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.130+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e743916640 0x55e743918a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:43:52.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.130+0000 7fe3c1e00700 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e743916640 0x55e743918a60 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b77a21f208af6f58 server_cookie=92a0e7eab932ace1 in_seq=0 out_seq=0 2024-04-05T23:43:52.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.130+0000 7fe3a3e00700 10 client.26710 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3a3e00700 1 -- 172.21.15.16:0/2090457638 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e743636c90 con 0x55e743916640 2024-04-05T23:43:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3a3e00700 10 client.26710 handle_command_reply: tid=0 2024-04-05T23:43:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 client.26710 shutdown 2024-04-05T23:43:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/2090457638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 msgr2=0x55e7438a8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 0x55e7438a8180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55e7438c4c40 tx=0x7fe3b0009260 comp rx=0 tx=0).stop 2024-04-05T23:43:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/2090457638 shutdown_connections 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55e743916640 0x55e743918a60 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e74389d4a0 0x55e7438a65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7438975c0 0x55e7438a8180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 --2- 172.21.15.16:0/2090457638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e74389cb80 0x55e7438a5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/2090457638 >> 172.21.15.16:0/2090457638 conn(0x55e74380b1c0 msgr2=0x55e74389c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3a3e00700 0 client.26710 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/2090457638 shutdown_connections 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.163+0000 7fe3dcb51b80 1 -- 172.21.15.16:0/2090457638 wait complete. 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.164+0000 7fe3dcb51b80 20 client.26710 trim_cache size 0 max 16384 2024-04-05T23:43:52.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:43:52.157 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:43:52.157 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:43:52.157 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:43:52.157 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.164+0000 7fe3c9000700 1 -- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3a80a1310 msgr2=0x7fe3a80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:52.157 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.164+0000 7fe3c9000700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3a80a1310 0x7fe3a80a37d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc09b150 tx=0x7fe3bc009040 comp rx=0 tx=0).stop 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.164+0000 7fe3c9000700 1 -- 172.21.15.16:0/991392066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c4111de0 msgr2=0x7fe3c410f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.164+0000 7fe3c9000700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c4111de0 0x7fe3c410f450 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00b6c0 tx=0x7fe3bc0094b0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.165+0000 7fe3c9000700 1 -- 172.21.15.16:0/991392066 shutdown_connections 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.165+0000 7fe3c9000700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe3a80a1310 0x7fe3a80a37d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.165+0000 7fe3c9000700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3c4073280 0x7fe3c41113c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.165+0000 7fe3c9000700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c4111de0 0x7fe3c410f450 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.165+0000 7fe3c9000700 1 --2- 172.21.15.16:0/991392066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3c4073ba0 0x7fe3c410ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.165+0000 7fe3c9000700 1 -- 172.21.15.16:0/991392066 >> 172.21.15.16:0/991392066 conn(0x7fe3c40fc8e0 msgr2=0x7fe3c40fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.165+0000 7fe3c9000700 1 -- 172.21.15.16:0/991392066 shutdown_connections 2024-04-05T23:43:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:43:52.165+0000 7fe3c9000700 1 -- 172.21.15.16:0/991392066 wait complete. 2024-04-05T23:43:52.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:51 smithi027 ceph-mon[78597]: pgmap v322: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 250 op/s 2024-04-05T23:43:52.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:51 smithi016 ceph-mon[38248]: pgmap v322: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 250 op/s 2024-04-05T23:43:52.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:51 smithi130 ceph-mon[79417]: pgmap v322: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 250 op/s 2024-04-05T23:43:53.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 508318 4.88 MB/sec execute 432 sec latency 22.035 ms 2024-04-05T23:43:53.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:43:53.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi027 to 182.4M 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: Unable to set osd_memory_target on smithi027 to 191349760: error parsing value: Value '191349760' is below minimum 939524096 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi130 to 182.4M 2024-04-05T23:43:53.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: Unable to set osd_memory_target on smithi130 to 191357644: error parsing value: Value '191357644' is below minimum 939524096 2024-04-05T23:43:53.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:52 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:43:53.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi027 to 182.4M 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: Unable to set osd_memory_target on smithi027 to 191349760: error parsing value: Value '191349760' is below minimum 939524096 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi130 to 182.4M 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: Unable to set osd_memory_target on smithi130 to 191357644: error parsing value: Value '191357644' is below minimum 939524096 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:52 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi027 to 182.4M 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: Unable to set osd_memory_target on smithi027 to 191349760: error parsing value: Value '191349760' is below minimum 939524096 2024-04-05T23:43:53.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:53.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi130 to 182.4M 2024-04-05T23:43:53.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: Unable to set osd_memory_target on smithi130 to 191357644: error parsing value: Value '191357644' is below minimum 939524096 2024-04-05T23:43:53.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:53.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:52 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:54.025 INFO:tasks.workunit.client.0.smithi016.stdout: 1 509557 4.88 MB/sec execute 433 sec latency 21.617 ms 2024-04-05T23:43:54.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: pgmap v323: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-05T23:43:54.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: Adjusting osd_memory_target on smithi016 to 222.6M 2024-04-05T23:43:54.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: Unable to set osd_memory_target on smithi016 to 233463910: error parsing value: Value '233463910' is below minimum 939524096 2024-04-05T23:43:54.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:43:54.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:43:54.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:53 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: pgmap v323: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: Adjusting osd_memory_target on smithi016 to 222.6M 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: Unable to set osd_memory_target on smithi016 to 233463910: error parsing value: Value '233463910' is below minimum 939524096 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:43:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:53 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: pgmap v323: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: Adjusting osd_memory_target on smithi016 to 222.6M 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: Unable to set osd_memory_target on smithi016 to 233463910: error parsing value: Value '233463910' is below minimum 939524096 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:43:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:53 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:43:55.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 509726 4.87 MB/sec execute 434 sec latency 903.946 ms 2024-04-05T23:43:56.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 509726 4.86 MB/sec execute 435 sec latency 1904.042 ms 2024-04-05T23:43:56.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:55 smithi027 ceph-mon[78597]: pgmap v324: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-05T23:43:56.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:55 smithi016 ceph-mon[38248]: pgmap v324: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-05T23:43:56.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:55 smithi130 ceph-mon[79417]: pgmap v324: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-05T23:43:57.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 510424 4.86 MB/sec execute 436 sec latency 2614.069 ms 2024-04-05T23:43:58.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 511970 4.87 MB/sec execute 437 sec latency 13.250 ms 2024-04-05T23:43:58.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:57 smithi027 ceph-mon[78597]: pgmap v325: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-05T23:43:58.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:57 smithi016 ceph-mon[38248]: pgmap v325: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-05T23:43:58.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:57 smithi130 ceph-mon[79417]: pgmap v325: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-05T23:43:59.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 513272 4.87 MB/sec execute 438 sec latency 64.764 ms 2024-04-05T23:44:00.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 513272 4.86 MB/sec execute 439 sec latency 1064.914 ms 2024-04-05T23:44:00.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:43:59 smithi027 ceph-mon[78597]: pgmap v326: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 820 KiB/s wr, 102 op/s 2024-04-05T23:44:00.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:43:59 smithi016 ceph-mon[38248]: pgmap v326: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 820 KiB/s wr, 102 op/s 2024-04-05T23:44:00.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:43:59 smithi130 ceph-mon[79417]: pgmap v326: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 820 KiB/s wr, 102 op/s 2024-04-05T23:44:01.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 513272 4.85 MB/sec execute 440 sec latency 2064.979 ms 2024-04-05T23:44:02.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 513272 4.84 MB/sec execute 441 sec latency 3065.041 ms 2024-04-05T23:44:02.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:01 smithi027 ceph-mon[78597]: pgmap v327: 97 pgs: 97 active+clean; 219 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-04-05T23:44:02.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:01 smithi016 ceph-mon[38248]: pgmap v327: 97 pgs: 97 active+clean; 219 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-04-05T23:44:02.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:01 smithi130 ceph-mon[79417]: pgmap v327: 97 pgs: 97 active+clean; 219 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-04-05T23:44:03.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 513441 4.83 MB/sec execute 442 sec latency 3259.442 ms 2024-04-05T23:44:04.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 513441 4.82 MB/sec execute 443 sec latency 1712.531 ms 2024-04-05T23:44:04.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:03 smithi027 ceph-mon[78597]: pgmap v328: 97 pgs: 97 active+clean; 219 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:44:04.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:03 smithi016 ceph-mon[38248]: pgmap v328: 97 pgs: 97 active+clean; 219 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:44:04.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:03 smithi130 ceph-mon[79417]: pgmap v328: 97 pgs: 97 active+clean; 219 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:44:05.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 514011 4.81 MB/sec execute 444 sec latency 2089.610 ms 2024-04-05T23:44:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.331+0000 7f2303600700 1 -- 172.21.15.16:0/860554416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1006a0 msgr2=0x7f22fc10cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.331+0000 7f2303600700 1 --2- 172.21.15.16:0/860554416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1006a0 0x7f22fc10cb40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009f90 tx=0x7f22f400b330 comp rx=0 tx=0).stop 2024-04-05T23:44:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.331+0000 7f2303600700 1 -- 172.21.15.16:0/860554416 shutdown_connections 2024-04-05T23:44:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.331+0000 7f2303600700 1 --2- 172.21.15.16:0/860554416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1006a0 0x7f22fc10cb40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.331+0000 7f2303600700 1 --2- 172.21.15.16:0/860554416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22fc105da0 0x7f22fc106180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.331+0000 7f2303600700 1 --2- 172.21.15.16:0/860554416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22fc0ffce0 0x7f22fc100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.331+0000 7f2303600700 1 -- 172.21.15.16:0/860554416 >> 172.21.15.16:0/860554416 conn(0x7f22fc076e50 msgr2=0x7f22fc077260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.331+0000 7f2303600700 1 -- 172.21.15.16:0/860554416 shutdown_connections 2024-04-05T23:44:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.331+0000 7f2303600700 1 -- 172.21.15.16:0/860554416 wait complete. 2024-04-05T23:44:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.333+0000 7f2303600700 1 Processor -- start 2024-04-05T23:44:05.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.333+0000 7f2303600700 1 -- start start 2024-04-05T23:44:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.333+0000 7f2303600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22fc0ffce0 0x7f22fc1973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.333+0000 7f2303600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22fc1006a0 0x7f22fc1978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.333+0000 7f2303600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc105da0 0x7f22fc19bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.333+0000 7f2303600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc19c280 con 0x7f22fc1006a0 2024-04-05T23:44:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.333+0000 7f2303600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc19c3f0 con 0x7f22fc105da0 2024-04-05T23:44:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.333+0000 7f2303600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc19c5b0 con 0x7f22fc0ffce0 2024-04-05T23:44:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2300c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22fc0ffce0 0x7f22fc1973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22fc1006a0 0x7f22fc1978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2300c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22fc0ffce0 0x7f22fc1973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44078/0 (socket says 172.21.15.16:44078) 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2300c00700 1 -- 172.21.15.16:0/1132776715 learned_addr learned my addr 172.21.15.16:0/1132776715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2301600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc105da0 0x7f22fc19bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2301600700 1 -- 172.21.15.16:0/1132776715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22fc0ffce0 msgr2=0x7f22fc1973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2301600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22fc0ffce0 0x7f22fc1973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2301600700 1 -- 172.21.15.16:0/1132776715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22fc1006a0 msgr2=0x7f22fc1978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2301600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22fc1006a0 0x7f22fc1978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2301600700 1 -- 172.21.15.16:0/1132776715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f4009c40 con 0x7f22fc105da0 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.334+0000 7f2301600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc105da0 0x7f22fc19bc40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f22f400b6c0 tx=0x7f22f40094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.335+0000 7f22f9600700 1 -- 172.21.15.16:0/1132776715 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f400e040 con 0x7f22fc105da0 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.335+0000 7f22f9600700 1 -- 172.21.15.16:0/1132776715 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22f401b450 con 0x7f22fc105da0 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.335+0000 7f2303600700 1 -- 172.21.15.16:0/1132776715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc19c830 con 0x7f22fc105da0 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.335+0000 7f22f9600700 1 -- 172.21.15.16:0/1132776715 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f40052a0 con 0x7f22fc105da0 2024-04-05T23:44:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.335+0000 7f2303600700 1 -- 172.21.15.16:0/1132776715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc101aa0 con 0x7f22fc105da0 2024-04-05T23:44:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.338+0000 7f22f9600700 1 -- 172.21.15.16:0/1132776715 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f22f401b5c0 con 0x7f22fc105da0 2024-04-05T23:44:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.339+0000 7f22f9600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f22e40a1310 0x7f22e40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.339+0000 7f22f9600700 1 -- 172.21.15.16:0/1132776715 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f22f40cdff0 con 0x7f22fc105da0 2024-04-05T23:44:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.339+0000 7f2300c00700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f22e40a1310 0x7f22e40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.340+0000 7f23171b4b80 1 Processor -- start 2024-04-05T23:44:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.340+0000 7f2300c00700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f22e40a1310 0x7f22e40a37d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f22f0005950 tx=0x7f22f0009310 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.340+0000 7f23171b4b80 1 -- start start 2024-04-05T23:44:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.340+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb76cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.340+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb767030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.340+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb767a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.340+0000 7f23171b4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb6cdb80 con 0x55efdb76d4a0 2024-04-05T23:44:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.340+0000 7f23171b4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb6be930 con 0x55efdb76cb80 2024-04-05T23:44:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.340+0000 7f23171b4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb6bc2f0 con 0x55efdb7675c0 2024-04-05T23:44:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb767030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f2300c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb76cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f2301600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb767a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f2301600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb767a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44086/0 (socket says 172.21.15.16:44086) 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f2300c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb76cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38788/0 (socket says 172.21.15.16:38788) 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f2301600700 1 -- 172.21.15.16:0/2804945389 learned_addr learned my addr 172.21.15.16:0/2804945389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f22fbe00700 1 -- 172.21.15.16:0/2804945389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 msgr2=0x55efdb767a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f22fbe00700 1 --2- 172.21.15.16:0/2804945389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb767a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f22fbe00700 1 -- 172.21.15.16:0/2804945389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 msgr2=0x55efdb76cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f22fbe00700 1 --2- 172.21.15.16:0/2804945389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb76cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f22fbe00700 1 -- 172.21.15.16:0/2804945389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efdb539350 con 0x55efdb76d4a0 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f2301600700 1 --2- 172.21.15.16:0/2804945389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb767a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.341+0000 7f22fbe00700 1 --2- 172.21.15.16:0/2804945389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb767030 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f22e8009c80 tx=0x7f22e8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be9351cc7994acb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.342+0000 7f22edc00700 1 -- 172.21.15.16:0/2804945389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8011cb0 con 0x55efdb76d4a0 2024-04-05T23:44:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.342+0000 7f22edc00700 1 -- 172.21.15.16:0/2804945389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22e8011e10 con 0x55efdb76d4a0 2024-04-05T23:44:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.342+0000 7f22edc00700 1 -- 172.21.15.16:0/2804945389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8005280 con 0x55efdb76d4a0 2024-04-05T23:44:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.342+0000 7f23171b4b80 1 -- 172.21.15.16:0/2804945389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 msgr2=0x55efdb767030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.342+0000 7f23171b4b80 1 --2- 172.21.15.16:0/2804945389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb767030 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f22e8009c80 tx=0x7f22e8009f90 comp rx=0 tx=0).stop 2024-04-05T23:44:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.342+0000 7f23171b4b80 1 -- 172.21.15.16:0/2804945389 shutdown_connections 2024-04-05T23:44:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.343+0000 7f23171b4b80 1 --2- 172.21.15.16:0/2804945389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb767a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.343+0000 7f23171b4b80 1 --2- 172.21.15.16:0/2804945389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb76cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.343+0000 7f23171b4b80 1 --2- 172.21.15.16:0/2804945389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb767030 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.343+0000 7f23171b4b80 1 -- 172.21.15.16:0/2804945389 >> 172.21.15.16:0/2804945389 conn(0x55efdb6db1c0 msgr2=0x55efdb761be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.343+0000 7f23171b4b80 1 -- 172.21.15.16:0/2804945389 shutdown_connections 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.343+0000 7f23171b4b80 1 -- 172.21.15.16:0/2804945389 wait complete. 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.343+0000 7f23171b4b80 1 -- start start 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.344+0000 7f23171b4b80 1 Processor -- start 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.344+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb7675c0 0x55efdb773540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.344+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb770830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.344+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb770f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.344+0000 7f23171b4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb6b7be0 con 0x55efdb76d4a0 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.344+0000 7f23171b4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb653d20 con 0x55efdb7675c0 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.344+0000 7f23171b4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb6539c0 con 0x55efdb76cb80 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.344+0000 7f23171b4b80 10 client.0 _ll_register_callbacks cb 0x55efdb4bf340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2300c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb7675c0 0x55efdb773540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2300c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb7675c0 0x55efdb773540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38794/0 (socket says 172.21.15.16:38794) 2024-04-05T23:44:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2300c00700 1 -- 172.21.15.16:0/277224550 learned_addr learned my addr 172.21.15.16:0/277224550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f22fbe00700 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb770830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2301600700 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb770f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2300c00700 1 -- 172.21.15.16:0/277224550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 msgr2=0x55efdb770830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2300c00700 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb770830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2300c00700 1 -- 172.21.15.16:0/277224550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 msgr2=0x55efdb770f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2300c00700 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb770f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2300c00700 1 -- 172.21.15.16:0/277224550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc101aa0 con 0x55efdb7675c0 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f2300c00700 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb7675c0 0x55efdb773540 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f22f0003a00 tx=0x7f22f00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5109d499f4a39c3d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.345+0000 7f22e2000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f22e2000700 1 -- 172.21.15.16:0/277224550 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f001fac0 con 0x55efdb7675c0 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f23171b4b80 1 -- 172.21.15.16:0/277224550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55efdb75a1e0 con 0x55efdb7675c0 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f22e2000700 1 -- 172.21.15.16:0/277224550 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f22f001fc20 con 0x55efdb7675c0 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f22e2000700 1 -- 172.21.15.16:0/277224550 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22f001fe20 con 0x55efdb7675c0 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f23171b4b80 10 client.26696 fetch_fsmap learned FSMap version 68 2024-04-05T23:44:05.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f23171b4b80 1 -- 172.21.15.16:0/277224550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x7f22e80098e0 con 0x55efdb7675c0 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f22e2000700 1 -- 172.21.15.16:0/277224550 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22f0034080 con 0x55efdb7675c0 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f22e2000700 1 -- 172.21.15.16:0/277224550 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 68) v1 ==== 7542+0+0 (secure 0 0 0) 0x7f22f0030620 con 0x55efdb7675c0 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f23171b4b80 10 client.26696 fetch_fsmap finished waiting for FSMap version 68 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.346+0000 7f23171b4b80 10 client.26696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.347+0000 7f23171b4b80 20 client.26696 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.347+0000 7f23171b4b80 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55efdb781080 0x55efdb7834a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.347+0000 7f23171b4b80 4 client.26696 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.347+0000 7f23171b4b80 1 -- 172.21.15.16:0/277224550 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55efdb6b06a0 con 0x55efdb781080 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.347+0000 7f22fbe00700 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55efdb781080 0x55efdb7834a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.347+0000 7f22fbe00700 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55efdb781080 0x55efdb7834a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e10e2edbd01796ef server_cookie=90816d2600465ec1 in_seq=0 out_seq=0 2024-04-05T23:44:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.347+0000 7f22e2000700 10 client.26696 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:05.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f22e2000700 1 -- 172.21.15.16:0/277224550 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55efdb6b06a0 con 0x55efdb781080 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f22e2000700 10 client.26696 handle_command_reply: tid=0 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 client.26696 shutdown 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 -- 172.21.15.16:0/277224550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb7675c0 msgr2=0x55efdb773540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb7675c0 0x55efdb773540 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f22f0003a00 tx=0x7f22f00047c0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 -- 172.21.15.16:0/277224550 shutdown_connections 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55efdb781080 0x55efdb7834a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb770f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb7675c0 0x55efdb773540 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 --2- 172.21.15.16:0/277224550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb770830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f22e2000700 0 client.26696 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 -- 172.21.15.16:0/277224550 >> 172.21.15.16:0/277224550 conn(0x55efdb6db1c0 msgr2=0x55efdb761280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 -- 172.21.15.16:0/277224550 shutdown_connections 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 1 -- 172.21.15.16:0/277224550 wait complete. 2024-04-05T23:44:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.349+0000 7f23171b4b80 20 client.26696 trim_cache size 0 max 16384 2024-04-05T23:44:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f23171b4b80 1 Processor -- start 2024-04-05T23:44:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f23171b4b80 1 -- start start 2024-04-05T23:44:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb777f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb761a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb76bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f23171b4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb53b640 con 0x55efdb76d4a0 2024-04-05T23:44:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f23171b4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb76eda0 con 0x55efdb76cb80 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f23171b4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb77d300 con 0x55efdb7675c0 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb761a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f2300c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb777f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb761a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:38796/0 (socket says 172.21.15.16:38796) 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f2300c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb777f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44094/0 (socket says 172.21.15.16:44094) 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f22fbe00700 1 -- 172.21.15.16:0/4009502032 learned_addr learned my addr 172.21.15.16:0/4009502032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.366+0000 7f2301600700 1 --2- 172.21.15.16:0/4009502032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb76bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.367+0000 7f22fbe00700 1 -- 172.21.15.16:0/4009502032 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 msgr2=0x55efdb777f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.367+0000 7f22fbe00700 1 --2- 172.21.15.16:0/4009502032 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb777f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.367+0000 7f22fbe00700 1 -- 172.21.15.16:0/4009502032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 msgr2=0x55efdb76bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.367+0000 7f22fbe00700 1 --2- 172.21.15.16:0/4009502032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb76bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.367+0000 7f22fbe00700 1 -- 172.21.15.16:0/4009502032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efdb4e5b00 con 0x55efdb76cb80 2024-04-05T23:44:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.367+0000 7f22fbe00700 1 --2- 172.21.15.16:0/4009502032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb761a00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f22e8014040 tx=0x7f22e8009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d8d615b43b2120b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.367+0000 7f22ee600700 1 -- 172.21.15.16:0/4009502032 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8004b00 con 0x55efdb76cb80 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.367+0000 7f22ee600700 1 -- 172.21.15.16:0/4009502032 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22e8004510 con 0x55efdb76cb80 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.367+0000 7f22ee600700 1 -- 172.21.15.16:0/4009502032 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8003d20 con 0x55efdb76cb80 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.368+0000 7f23171b4b80 1 -- 172.21.15.16:0/4009502032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 msgr2=0x55efdb761a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.368+0000 7f23171b4b80 1 --2- 172.21.15.16:0/4009502032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb761a00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f22e8014040 tx=0x7f22e8009bb0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.368+0000 7f23171b4b80 1 -- 172.21.15.16:0/4009502032 shutdown_connections 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.368+0000 7f23171b4b80 1 --2- 172.21.15.16:0/4009502032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb76d4a0 0x55efdb76bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.368+0000 7f23171b4b80 1 --2- 172.21.15.16:0/4009502032 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb7675c0 0x55efdb777f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.368+0000 7f23171b4b80 1 --2- 172.21.15.16:0/4009502032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76cb80 0x55efdb761a00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.368+0000 7f23171b4b80 1 -- 172.21.15.16:0/4009502032 >> 172.21.15.16:0/4009502032 conn(0x55efdb6db1c0 msgr2=0x55efdb76b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.368+0000 7f23171b4b80 1 -- 172.21.15.16:0/4009502032 shutdown_connections 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.368+0000 7f23171b4b80 1 -- 172.21.15.16:0/4009502032 wait complete. 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.369+0000 7f23171b4b80 1 -- start start 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.369+0000 7f23171b4b80 1 Processor -- start 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.369+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb7675c0 0x55efdb769ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.369+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb7688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f23171b4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76d4a0 0x55efdb768de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f23171b4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb769320 con 0x55efdb7675c0 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f23171b4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb769490 con 0x55efdb76d4a0 2024-04-05T23:44:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f23171b4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efdb769650 con 0x55efdb76cb80 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f23171b4b80 10 client.0 _ll_register_callbacks cb 0x55efdb4bf340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb7688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb7688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:44104/0 (socket says 172.21.15.16:44104) 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f22fbe00700 1 -- 172.21.15.16:0/1521662109 learned_addr learned my addr 172.21.15.16:0/1521662109 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f2300c00700 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb7675c0 0x55efdb769ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f2301600700 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76d4a0 0x55efdb768de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f22fbe00700 1 -- 172.21.15.16:0/1521662109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76d4a0 msgr2=0x55efdb768de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f22fbe00700 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76d4a0 0x55efdb768de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f22fbe00700 1 -- 172.21.15.16:0/1521662109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb7675c0 msgr2=0x55efdb769ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f22fbe00700 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb7675c0 0x55efdb769ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f22fbe00700 1 -- 172.21.15.16:0/1521662109 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e801d040 con 0x55efdb76cb80 2024-04-05T23:44:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.370+0000 7f2301600700 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76d4a0 0x55efdb768de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.371+0000 7f22fbe00700 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb7688a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f22e800a010 tx=0x7f22e80117d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b6495d8e3c1b97a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.371+0000 7f22e2000700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.371+0000 7f22e2000700 1 -- 172.21.15.16:0/1521662109 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8003680 con 0x55efdb76cb80 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.371+0000 7f22e2000700 1 -- 172.21.15.16:0/1521662109 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f22e8004050 con 0x55efdb76cb80 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.371+0000 7f23171b4b80 1 -- 172.21.15.16:0/1521662109 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22e8011e10 con 0x55efdb76cb80 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.371+0000 7f22e2000700 1 -- 172.21.15.16:0/1521662109 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22e8004210 con 0x55efdb76cb80 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.371+0000 7f23171b4b80 10 client.26728 fetch_fsmap learned FSMap version 68 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.371+0000 7f23171b4b80 1 -- 172.21.15.16:0/1521662109 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x55efdb67de40 con 0x55efdb76cb80 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.371+0000 7f22e2000700 1 -- 172.21.15.16:0/1521662109 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22e803f020 con 0x55efdb76cb80 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f22e2000700 1 -- 172.21.15.16:0/1521662109 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 68) v1 ==== 7542+0+0 (secure 0 0 0) 0x7f22e8031070 con 0x55efdb76cb80 2024-04-05T23:44:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f23171b4b80 10 client.26728 fetch_fsmap finished waiting for FSMap version 68 2024-04-05T23:44:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f23171b4b80 10 client.26728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f23171b4b80 20 client.26728 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f23171b4b80 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55efdb7d60e0 0x55efdb7d8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f23171b4b80 4 client.26728 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:44:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f23171b4b80 1 -- 172.21.15.16:0/1521662109 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55efdb506c90 con 0x55efdb7d60e0 2024-04-05T23:44:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f2300c00700 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55efdb7d60e0 0x55efdb7d8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f2300c00700 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55efdb7d60e0 0x55efdb7d8500 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=348eee4ed544090c server_cookie=38e85838d84f6926 in_seq=0 out_seq=0 2024-04-05T23:44:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.372+0000 7f22e2000700 10 client.26728 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f22e2000700 1 -- 172.21.15.16:0/1521662109 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55efdb506c90 con 0x55efdb7d60e0 2024-04-05T23:44:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f22e2000700 10 client.26728 handle_command_reply: tid=0 2024-04-05T23:44:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f23171b4b80 1 client.26728 shutdown 2024-04-05T23:44:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f23171b4b80 1 -- 172.21.15.16:0/1521662109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 msgr2=0x55efdb7688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f23171b4b80 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb7688a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f22e800a010 tx=0x7f22e80117d0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f23171b4b80 1 -- 172.21.15.16:0/1521662109 shutdown_connections 2024-04-05T23:44:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f23171b4b80 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55efdb7d60e0 0x55efdb7d8500 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f23171b4b80 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55efdb7675c0 0x55efdb769ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f23171b4b80 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55efdb76d4a0 0x55efdb768de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f23171b4b80 1 --2- 172.21.15.16:0/1521662109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efdb76cb80 0x55efdb7688a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f22e2000700 0 client.26728 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.404+0000 7f23171b4b80 1 -- 172.21.15.16:0/1521662109 >> 172.21.15.16:0/1521662109 conn(0x55efdb6db1c0 msgr2=0x55efdb76c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.405+0000 7f23171b4b80 1 -- 172.21.15.16:0/1521662109 shutdown_connections 2024-04-05T23:44:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.405+0000 7f23171b4b80 1 -- 172.21.15.16:0/1521662109 wait complete. 2024-04-05T23:44:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.405+0000 7f23171b4b80 20 client.26728 trim_cache size 0 max 16384 2024-04-05T23:44:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:44:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:44:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:44:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:44:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.405+0000 7f2303600700 1 -- 172.21.15.16:0/1132776715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f22e40a1310 msgr2=0x7f22e40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.405+0000 7f2303600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f22e40a1310 0x7f22e40a37d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f22f0005950 tx=0x7f22f0009310 comp rx=0 tx=0).stop 2024-04-05T23:44:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.405+0000 7f2303600700 1 -- 172.21.15.16:0/1132776715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc105da0 msgr2=0x7f22fc19bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.405+0000 7f2303600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc105da0 0x7f22fc19bc40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f22f400b6c0 tx=0x7f22f40094b0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.406+0000 7f2303600700 1 -- 172.21.15.16:0/1132776715 shutdown_connections 2024-04-05T23:44:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.406+0000 7f2303600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f22e40a1310 0x7f22e40a37d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.406+0000 7f2303600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc105da0 0x7f22fc19bc40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.406+0000 7f2303600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22fc0ffce0 0x7f22fc1973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.406+0000 7f2303600700 1 --2- 172.21.15.16:0/1132776715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22fc1006a0 0x7f22fc1978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.406+0000 7f2303600700 1 -- 172.21.15.16:0/1132776715 >> 172.21.15.16:0/1132776715 conn(0x7f22fc076e50 msgr2=0x7f22fc0683a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.406+0000 7f2303600700 1 -- 172.21.15.16:0/1132776715 shutdown_connections 2024-04-05T23:44:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:05.406+0000 7f2303600700 1 -- 172.21.15.16:0/1132776715 wait complete. 2024-04-05T23:44:06.026 INFO:tasks.workunit.client.0.smithi016.stdout: 1 515488 4.83 MB/sec execute 445 sec latency 21.301 ms 2024-04-05T23:44:06.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:06 smithi016 ceph-mon[38248]: pgmap v329: 97 pgs: 97 active+clean; 219 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:44:06.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:06 smithi130 ceph-mon[79417]: pgmap v329: 97 pgs: 97 active+clean; 219 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:44:06.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:06 smithi027 ceph-mon[78597]: pgmap v329: 97 pgs: 97 active+clean; 219 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-04-05T23:44:07.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 516810 4.83 MB/sec execute 446 sec latency 16.159 ms 2024-04-05T23:44:08.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 518485 4.84 MB/sec execute 447 sec latency 186.043 ms 2024-04-05T23:44:08.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:08 smithi016 ceph-mon[38248]: pgmap v330: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-05T23:44:08.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:08 smithi130 ceph-mon[79417]: pgmap v330: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-05T23:44:08.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:08 smithi027 ceph-mon[78597]: pgmap v330: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-05T23:44:09.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 519563 4.84 MB/sec execute 448 sec latency 31.820 ms 2024-04-05T23:44:09.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:09 smithi016 ceph-mon[38248]: osdmap e94: 12 total, 12 up, 12 in 2024-04-05T23:44:09.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:09 smithi130 ceph-mon[79417]: osdmap e94: 12 total, 12 up, 12 in 2024-04-05T23:44:09.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:09 smithi027 ceph-mon[78597]: osdmap e94: 12 total, 12 up, 12 in 2024-04-05T23:44:10.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 520365 4.84 MB/sec execute 449 sec latency 428.545 ms 2024-04-05T23:44:10.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:10 smithi016 ceph-mon[38248]: pgmap v332: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-04-05T23:44:10.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:10 smithi130 ceph-mon[79417]: pgmap v332: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-04-05T23:44:10.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:10 smithi027 ceph-mon[78597]: pgmap v332: 97 pgs: 97 active+clean; 211 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-04-05T23:44:11.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 520365 4.83 MB/sec execute 450 sec latency 1428.631 ms 2024-04-05T23:44:12.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 520365 4.82 MB/sec execute 451 sec latency 2428.732 ms 2024-04-05T23:44:12.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:12 smithi016 ceph-mon[38248]: pgmap v333: 97 pgs: 97 active+clean; 217 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.2 MiB/s wr, 208 op/s 2024-04-05T23:44:12.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:12 smithi130 ceph-mon[79417]: pgmap v333: 97 pgs: 97 active+clean; 217 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.2 MiB/s wr, 208 op/s 2024-04-05T23:44:12.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:12 smithi027 ceph-mon[78597]: pgmap v333: 97 pgs: 97 active+clean; 217 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.2 MiB/s wr, 208 op/s 2024-04-05T23:44:13.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 522010 4.83 MB/sec execute 452 sec latency 2619.165 ms 2024-04-05T23:44:14.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 523140 4.83 MB/sec execute 453 sec latency 12.703 ms 2024-04-05T23:44:14.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:14 smithi016 ceph-mon[38248]: pgmap v334: 97 pgs: 97 active+clean; 217 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.2 MiB/s wr, 208 op/s 2024-04-05T23:44:14.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:14 smithi016 ceph-mon[38248]: osdmap e95: 12 total, 12 up, 12 in 2024-04-05T23:44:14.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:14 smithi130 ceph-mon[79417]: pgmap v334: 97 pgs: 97 active+clean; 217 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.2 MiB/s wr, 208 op/s 2024-04-05T23:44:14.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:14 smithi130 ceph-mon[79417]: osdmap e95: 12 total, 12 up, 12 in 2024-04-05T23:44:14.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:14 smithi027 ceph-mon[78597]: pgmap v334: 97 pgs: 97 active+clean; 217 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.2 MiB/s wr, 208 op/s 2024-04-05T23:44:14.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:14 smithi027 ceph-mon[78597]: osdmap e95: 12 total, 12 up, 12 in 2024-04-05T23:44:15.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 523906 4.83 MB/sec execute 454 sec latency 540.387 ms 2024-04-05T23:44:16.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 523906 4.82 MB/sec execute 455 sec latency 1540.521 ms 2024-04-05T23:44:16.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:16 smithi016 ceph-mon[38248]: pgmap v336: 97 pgs: 97 active+clean; 217 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-04-05T23:44:16.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:16 smithi130 ceph-mon[79417]: pgmap v336: 97 pgs: 97 active+clean; 217 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-04-05T23:44:16.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:16 smithi027 ceph-mon[78597]: pgmap v336: 97 pgs: 97 active+clean; 217 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-04-05T23:44:17.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 523906 4.81 MB/sec execute 456 sec latency 2540.647 ms 2024-04-05T23:44:18.027 INFO:tasks.workunit.client.0.smithi016.stdout: 1 525571 4.82 MB/sec execute 457 sec latency 2736.525 ms 2024-04-05T23:44:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.560+0000 7f8b59000700 1 -- 172.21.15.16:0/2755642349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b54106f40 msgr2=0x7f8b54107320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.560+0000 7f8b59000700 1 --2- 172.21.15.16:0/2755642349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b54106f40 0x7f8b54107320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009990 tx=0x7f8b44009ca0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.561+0000 7f8b59000700 1 -- 172.21.15.16:0/2755642349 shutdown_connections 2024-04-05T23:44:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.561+0000 7f8b59000700 1 --2- 172.21.15.16:0/2755642349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b54068be0 0x7f8b5410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.561+0000 7f8b59000700 1 --2- 172.21.15.16:0/2755642349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b54106f40 0x7f8b54107320 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.561+0000 7f8b59000700 1 --2- 172.21.15.16:0/2755642349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b54068220 0x7f8b540686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.561+0000 7f8b59000700 1 -- 172.21.15.16:0/2755642349 >> 172.21.15.16:0/2755642349 conn(0x7f8b54076e50 msgr2=0x7f8b54077260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.561+0000 7f8b59000700 1 -- 172.21.15.16:0/2755642349 shutdown_connections 2024-04-05T23:44:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.561+0000 7f8b59000700 1 -- 172.21.15.16:0/2755642349 wait complete. 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.562+0000 7f8b59000700 1 Processor -- start 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.562+0000 7f8b59000700 1 -- start start 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.562+0000 7f8b59000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b54068220 0x7f8b54197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.562+0000 7f8b59000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b54068be0 0x7f8b541977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.562+0000 7f8b59000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b54106f40 0x7f8b5419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.562+0000 7f8b59000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5419c140 con 0x7f8b54068be0 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.562+0000 7f8b59000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5419c2b0 con 0x7f8b54068220 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.562+0000 7f8b59000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5419c470 con 0x7f8b54106f40 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b54068220 0x7f8b54197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b54068be0 0x7f8b541977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b54068220 0x7f8b54197270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48880/0 (socket says 172.21.15.16:48880) 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51e00700 1 -- 172.21.15.16:0/3133635105 learned_addr learned my addr 172.21.15.16:0/3133635105 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b52800700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b54106f40 0x7f8b5419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51e00700 1 -- 172.21.15.16:0/3133635105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b54106f40 msgr2=0x7f8b5419bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51e00700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b54106f40 0x7f8b5419bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51e00700 1 -- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b54068be0 msgr2=0x7f8b541977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51e00700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b54068be0 0x7f8b541977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51e00700 1 -- 172.21.15.16:0/3133635105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b440095f0 con 0x7f8b54068220 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.563+0000 7f8b51400700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b54068be0 0x7f8b541977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.564+0000 7f8b51e00700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b54068220 0x7f8b54197270 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8b44000c00 tx=0x7f8b44005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.564+0000 7f8b4aa00700 1 -- 172.21.15.16:0/3133635105 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4401d070 con 0x7f8b54068220 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.564+0000 7f8b4aa00700 1 -- 172.21.15.16:0/3133635105 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b44022470 con 0x7f8b54068220 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.564+0000 7f8b59000700 1 -- 172.21.15.16:0/3133635105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5419c6f0 con 0x7f8b54068220 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.564+0000 7f8b4aa00700 1 -- 172.21.15.16:0/3133635105 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44017650 con 0x7f8b54068220 2024-04-05T23:44:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.564+0000 7f8b59000700 1 -- 172.21.15.16:0/3133635105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b541a3f90 con 0x7f8b54068220 2024-04-05T23:44:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.567+0000 7f8b4aa00700 1 -- 172.21.15.16:0/3133635105 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f8b440225e0 con 0x7f8b54068220 2024-04-05T23:44:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.568+0000 7f8b4aa00700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8b380a1310 0x7f8b380a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b4aa00700 1 -- 172.21.15.16:0/3133635105 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f8b440cdf00 con 0x7f8b54068220 2024-04-05T23:44:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b51400700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8b380a1310 0x7f8b380a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b6ca91b80 1 Processor -- start 2024-04-05T23:44:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b6ca91b80 1 -- start start 2024-04-05T23:44:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b51400700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8b380a1310 0x7f8b380a37d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8b54198770 tx=0x7f8b40009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce8996df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 0x55ce89968030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 0x55ce89968a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce898ceb80 con 0x55ce899685c0 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.569+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce898bf930 con 0x55ce8996e4a0 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce898bd2f0 con 0x55ce8996db80 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce8996df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b52800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 0x55ce89968a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 0x55ce89968030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce8996df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55032/0 (socket says 172.21.15.16:55032) 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51e00700 1 -- 172.21.15.16:0/1676621615 learned_addr learned my addr 172.21.15.16:0/1676621615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51e00700 1 -- 172.21.15.16:0/1676621615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 msgr2=0x55ce89968030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51e00700 1 --2- 172.21.15.16:0/1676621615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 0x55ce89968030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51e00700 1 -- 172.21.15.16:0/1676621615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 msgr2=0x55ce89968a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51e00700 1 --2- 172.21.15.16:0/1676621615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 0x55ce89968a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51e00700 1 -- 172.21.15.16:0/1676621615 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce8973a350 con 0x55ce8996db80 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.570+0000 7f8b51e00700 1 --2- 172.21.15.16:0/1676621615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce8996df60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8b44097640 tx=0x7f8b4409af10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc0887c9b7f7730d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.571+0000 7f8b3f400700 1 -- 172.21.15.16:0/1676621615 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4409b840 con 0x55ce8996db80 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.571+0000 7f8b3f400700 1 -- 172.21.15.16:0/1676621615 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b4409fe90 con 0x55ce8996db80 2024-04-05T23:44:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.571+0000 7f8b3f400700 1 -- 172.21.15.16:0/1676621615 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44022990 con 0x55ce8996db80 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.571+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/1676621615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 msgr2=0x55ce8996df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.571+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/1676621615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce8996df60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8b44097640 tx=0x7f8b4409af10 comp rx=0 tx=0).stop 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.572+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/1676621615 shutdown_connections 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.572+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/1676621615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 0x55ce89968a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.572+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/1676621615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce8996df60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.572+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/1676621615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 0x55ce89968030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.572+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/1676621615 >> 172.21.15.16:0/1676621615 conn(0x55ce898dc1c0 msgr2=0x55ce89962be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.572+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/1676621615 shutdown_connections 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.572+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/1676621615 wait complete. 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.572+0000 7f8b6ca91b80 1 -- start start 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.572+0000 7f8b6ca91b80 1 Processor -- start 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce899685c0 0x55ce89971990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996db80 0x55ce89971ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 0x55ce89972410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce898b8be0 con 0x55ce8996db80 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce89854d20 con 0x55ce8996e4a0 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce898549c0 con 0x55ce899685c0 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b6ca91b80 10 client.0 _ll_register_callbacks cb 0x55ce896c0340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b51e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce899685c0 0x55ce89971990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b51e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce899685c0 0x55ce89971990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55034/0 (socket says 172.21.15.16:55034) 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b51e00700 1 -- 172.21.15.16:0/670003520 learned_addr learned my addr 172.21.15.16:0/670003520 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b52800700 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 0x55ce89972410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.573+0000 7f8b51400700 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996db80 0x55ce89971ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b51e00700 1 -- 172.21.15.16:0/670003520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 msgr2=0x55ce89972410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b51e00700 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 0x55ce89972410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b51e00700 1 -- 172.21.15.16:0/670003520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996db80 msgr2=0x55ce89971ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b51e00700 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996db80 0x55ce89971ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b51e00700 1 -- 172.21.15.16:0/670003520 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b541a3f90 con 0x55ce899685c0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b51e00700 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce899685c0 0x55ce89971990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8b4409b080 tx=0x7f8b44022af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9f34189bade626a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b33e00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b33e00700 1 -- 172.21.15.16:0/670003520 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4409b840 con 0x55ce899685c0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b33e00700 1 -- 172.21.15.16:0/670003520 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f8b4409b140 con 0x55ce899685c0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/670003520 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ce8995b1e0 con 0x55ce899685c0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.574+0000 7f8b33e00700 1 -- 172.21.15.16:0/670003520 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b440b9410 con 0x55ce899685c0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b6ca91b80 10 client.26740 fetch_fsmap learned FSMap version 68 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/670003520 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x55ce896e6b00 con 0x55ce899685c0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b33e00700 1 -- 172.21.15.16:0/670003520 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ce8995b1e0 con 0x55ce899685c0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b33e00700 1 -- 172.21.15.16:0/670003520 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 68) v1 ==== 7542+0+0 (secure 0 0 0) 0x7f8b440cbd20 con 0x55ce899685c0 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b6ca91b80 10 client.26740 fetch_fsmap finished waiting for FSMap version 68 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b6ca91b80 10 client.26740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b6ca91b80 20 client.26740 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ce899817d0 0x55ce89983bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b6ca91b80 4 client.26740 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:44:18.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/670003520 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce898b16a0 con 0x55ce899817d0 2024-04-05T23:44:18.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.575+0000 7f8b51400700 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ce899817d0 0x55ce89983bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.576+0000 7f8b33e00700 10 client.26740 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.576+0000 7f8b51400700 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ce899817d0 0x55ce89983bf0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43cb0212fabf9ea1 server_cookie=94afac57dffb3d7f in_seq=0 out_seq=0 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b33e00700 1 -- 172.21.15.16:0/670003520 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ce898b16a0 con 0x55ce899817d0 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b33e00700 10 client.26740 handle_command_reply: tid=0 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 client.26740 shutdown 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/670003520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce899685c0 msgr2=0x55ce89971990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce899685c0 0x55ce89971990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8b4409b080 tx=0x7f8b44022af0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/670003520 shutdown_connections 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ce899817d0 0x55ce89983bf0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996e4a0 0x55ce89972410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce899685c0 0x55ce89971990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b33e00700 0 client.26740 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/670003520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996db80 0x55ce89971ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/670003520 >> 172.21.15.16:0/670003520 conn(0x55ce898dc1c0 msgr2=0x55ce89962280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/670003520 shutdown_connections 2024-04-05T23:44:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/670003520 wait complete. 2024-04-05T23:44:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.577+0000 7f8b6ca91b80 20 client.26740 trim_cache size 0 max 16384 2024-04-05T23:44:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.593+0000 7f8b6ca91b80 1 Processor -- start 2024-04-05T23:44:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.593+0000 7f8b6ca91b80 1 -- start start 2024-04-05T23:44:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce899685c0 0x55ce89968030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce89978cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996e4a0 0x55ce8996d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8973c640 con 0x55ce8996e4a0 2024-04-05T23:44:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce89972d10 con 0x55ce899685c0 2024-04-05T23:44:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce899636d0 con 0x55ce8996db80 2024-04-05T23:44:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b51e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce899685c0 0x55ce89968030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b51e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce899685c0 0x55ce89968030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48914/0 (socket says 172.21.15.16:48914) 2024-04-05T23:44:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b51e00700 1 -- 172.21.15.16:0/2293028184 learned_addr learned my addr 172.21.15.16:0/2293028184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b52800700 1 --2- 172.21.15.16:0/2293028184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996e4a0 0x55ce8996d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b51400700 1 --2- 172.21.15.16:0/2293028184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce89978cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b51e00700 1 -- 172.21.15.16:0/2293028184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 msgr2=0x55ce89978cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b51e00700 1 --2- 172.21.15.16:0/2293028184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce89978cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b51e00700 1 -- 172.21.15.16:0/2293028184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996e4a0 msgr2=0x55ce8996d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b51e00700 1 --2- 172.21.15.16:0/2293028184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996e4a0 0x55ce8996d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.594+0000 7f8b51e00700 1 -- 172.21.15.16:0/2293028184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce8987ee40 con 0x55ce899685c0 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.595+0000 7f8b51e00700 1 --2- 172.21.15.16:0/2293028184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce899685c0 0x55ce89968030 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8b44097640 tx=0x7f8b440a1380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1df70412907d46e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.595+0000 7f8b3fe00700 1 -- 172.21.15.16:0/2293028184 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4409b840 con 0x55ce899685c0 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.595+0000 7f8b3fe00700 1 -- 172.21.15.16:0/2293028184 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b440a3310 con 0x55ce899685c0 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.595+0000 7f8b3fe00700 1 -- 172.21.15.16:0/2293028184 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4409a890 con 0x55ce899685c0 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/2293028184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce899685c0 msgr2=0x55ce89968030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/2293028184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce899685c0 0x55ce89968030 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8b44097640 tx=0x7f8b440a1380 comp rx=0 tx=0).stop 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/2293028184 shutdown_connections 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/2293028184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce8996e4a0 0x55ce8996d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/2293028184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce899685c0 0x55ce89968030 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/2293028184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996db80 0x55ce89978cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/2293028184 >> 172.21.15.16:0/2293028184 conn(0x55ce898dc1c0 msgr2=0x55ce8996ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/2293028184 shutdown_connections 2024-04-05T23:44:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/2293028184 wait complete. 2024-04-05T23:44:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 -- start start 2024-04-05T23:44:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.596+0000 7f8b6ca91b80 1 Processor -- start 2024-04-05T23:44:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 0x55ce899c9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996db80 0x55ce899c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b6ca91b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996e4a0 0x55ce899d4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce899d4b50 con 0x55ce899685c0 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce899d4c90 con 0x55ce8996db80 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b6ca91b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce899d4dd0 con 0x55ce8996e4a0 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b6ca91b80 10 client.0 _ll_register_callbacks cb 0x7f8b4c009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b51400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996db80 0x55ce899c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b51e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 0x55ce899c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b51400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996db80 0x55ce899c9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:48928/0 (socket says 172.21.15.16:48928) 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b51400700 1 -- 172.21.15.16:0/563502494 learned_addr learned my addr 172.21.15.16:0/563502494 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.597+0000 7f8b52800700 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996e4a0 0x55ce899d4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b51400700 1 -- 172.21.15.16:0/563502494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996e4a0 msgr2=0x55ce899d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b51400700 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996e4a0 0x55ce899d4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b51400700 1 -- 172.21.15.16:0/563502494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 msgr2=0x55ce899c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b51400700 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 0x55ce899c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b51400700 1 -- 172.21.15.16:0/563502494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce896e6b00 con 0x55ce8996db80 2024-04-05T23:44:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b51e00700 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 0x55ce899c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b51400700 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996db80 0x55ce899c9cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55ce899844d0 tx=0x7f8b40013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d69362f37240a17a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b33e00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b33e00700 1 -- 172.21.15.16:0/563502494 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40024040 con 0x55ce8996db80 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/563502494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b4409fe90 con 0x55ce8996db80 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b33e00700 1 -- 172.21.15.16:0/563502494 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f8b40028e40 con 0x55ce8996db80 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b33e00700 1 -- 172.21.15.16:0/563502494 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b40028730 con 0x55ce8996db80 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.598+0000 7f8b6ca91b80 10 client.26720 fetch_fsmap learned FSMap version 68 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/563502494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x55ce8973ec70 con 0x55ce8996db80 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b33e00700 1 -- 172.21.15.16:0/563502494 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b4409fe90 con 0x55ce8996db80 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b33e00700 1 -- 172.21.15.16:0/563502494 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 68) v1 ==== 7542+0+0 (secure 0 0 0) 0x7f8b40031780 con 0x55ce8996db80 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b6ca91b80 10 client.26720 fetch_fsmap finished waiting for FSMap version 68 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b6ca91b80 10 client.26720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b6ca91b80 20 client.26720 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ce899d66b0 0x55ce899d8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b6ca91b80 4 client.26720 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:44:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/563502494 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ce89707c90 con 0x55ce899d66b0 2024-04-05T23:44:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.599+0000 7f8b51e00700 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ce899d66b0 0x55ce899d8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.600+0000 7f8b33e00700 10 client.26720 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.600+0000 7f8b51e00700 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ce899d66b0 0x55ce899d8ad0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43786de9a10f8dcd server_cookie=c67830bf5fb33047 in_seq=0 out_seq=0 2024-04-05T23:44:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.631+0000 7f8b33e00700 1 -- 172.21.15.16:0/563502494 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ce89707c90 con 0x55ce899d66b0 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.631+0000 7f8b33e00700 10 client.26720 handle_command_reply: tid=0 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.631+0000 7f8b6ca91b80 1 client.26720 shutdown 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/563502494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996db80 msgr2=0x55ce899c9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996db80 0x55ce899c9cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55ce899844d0 tx=0x7f8b40013040 comp rx=0 tx=0).stop 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/563502494 shutdown_connections 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ce899d66b0 0x55ce899d8ad0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce899685c0 0x55ce899c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce8996e4a0 0x55ce899d4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b33e00700 0 client.26720 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 --2- 172.21.15.16:0/563502494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce8996db80 0x55ce899c9cc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/563502494 >> 172.21.15.16:0/563502494 conn(0x55ce898dc1c0 msgr2=0x55ce899730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/563502494 shutdown_connections 2024-04-05T23:44:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 1 -- 172.21.15.16:0/563502494 wait complete. 2024-04-05T23:44:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.632+0000 7f8b6ca91b80 20 client.26720 trim_cache size 0 max 16384 2024-04-05T23:44:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:44:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:44:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:44:18.625 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:44:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 -- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8b380a1310 msgr2=0x7f8b380a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8b380a1310 0x7f8b380a37d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8b54198770 tx=0x7f8b40009260 comp rx=0 tx=0).stop 2024-04-05T23:44:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 -- 172.21.15.16:0/3133635105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b54068220 msgr2=0x7f8b54197270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b54068220 0x7f8b54197270 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8b44000c00 tx=0x7f8b44005740 comp rx=0 tx=0).stop 2024-04-05T23:44:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 -- 172.21.15.16:0/3133635105 shutdown_connections 2024-04-05T23:44:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8b380a1310 0x7f8b380a37d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b54106f40 0x7f8b5419bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b54068220 0x7f8b54197270 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 --2- 172.21.15.16:0/3133635105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b54068be0 0x7f8b541977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 -- 172.21.15.16:0/3133635105 >> 172.21.15.16:0/3133635105 conn(0x7f8b54076e50 msgr2=0x7f8b540feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 -- 172.21.15.16:0/3133635105 shutdown_connections 2024-04-05T23:44:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:18.633+0000 7f8b59000700 1 -- 172.21.15.16:0/3133635105 wait complete. 2024-04-05T23:44:18.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:18 smithi130 ceph-mon[79417]: pgmap v337: 97 pgs: 97 active+clean; 222 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-04-05T23:44:18.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:18 smithi027 ceph-mon[78597]: pgmap v337: 97 pgs: 97 active+clean; 222 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-04-05T23:44:19.028 INFO:tasks.workunit.client.0.smithi016.stdout: 1 526638 4.82 MB/sec execute 458 sec latency 25.009 ms 2024-04-05T23:44:19.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:18 smithi016 ceph-mon[38248]: pgmap v337: 97 pgs: 97 active+clean; 222 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-04-05T23:44:20.028 INFO:tasks.workunit.client.0.smithi016.stdout: 1 527456 4.82 MB/sec execute 459 sec latency 503.837 ms 2024-04-05T23:44:20.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:20 smithi027 ceph-mon[78597]: pgmap v338: 97 pgs: 97 active+clean; 222 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-04-05T23:44:21.028 INFO:tasks.workunit.client.0.smithi016.stdout: 1 529395 4.83 MB/sec execute 460 sec latency 580.119 ms 2024-04-05T23:44:21.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:20 smithi016 ceph-mon[38248]: pgmap v338: 97 pgs: 97 active+clean; 222 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-04-05T23:44:21.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:20 smithi130 ceph-mon[79417]: pgmap v338: 97 pgs: 97 active+clean; 222 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-04-05T23:44:21.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:21 smithi027 ceph-mon[78597]: pgmap v339: 97 pgs: 97 active+clean; 213 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 2.2 MiB/s wr, 176 op/s 2024-04-05T23:44:22.028 INFO:tasks.workunit.client.0.smithi016.stdout: 1 530538 4.83 MB/sec execute 461 sec latency 14.453 ms 2024-04-05T23:44:22.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:21 smithi016 ceph-mon[38248]: pgmap v339: 97 pgs: 97 active+clean; 213 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 2.2 MiB/s wr, 176 op/s 2024-04-05T23:44:22.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:21 smithi130 ceph-mon[79417]: pgmap v339: 97 pgs: 97 active+clean; 213 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 2.2 MiB/s wr, 176 op/s 2024-04-05T23:44:23.028 INFO:tasks.workunit.client.0.smithi016.stdout: 1 530999 4.83 MB/sec execute 462 sec latency 720.190 ms 2024-04-05T23:44:24.028 INFO:tasks.workunit.client.0.smithi016.stdout: 1 530999 4.82 MB/sec execute 463 sec latency 1720.327 ms 2024-04-05T23:44:24.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:23 smithi016 ceph-mon[38248]: pgmap v340: 97 pgs: 97 active+clean; 213 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 2.2 MiB/s wr, 176 op/s 2024-04-05T23:44:24.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:23 smithi130 ceph-mon[79417]: pgmap v340: 97 pgs: 97 active+clean; 213 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 2.2 MiB/s wr, 176 op/s 2024-04-05T23:44:24.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:23 smithi027 ceph-mon[78597]: pgmap v340: 97 pgs: 97 active+clean; 213 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 2.2 MiB/s wr, 176 op/s 2024-04-05T23:44:25.028 INFO:tasks.workunit.client.0.smithi016.stdout: 1 530999 4.81 MB/sec execute 464 sec latency 2720.477 ms 2024-04-05T23:44:26.028 INFO:tasks.workunit.client.0.smithi016.stdout: 1 530999 4.80 MB/sec execute 465 sec latency 3720.590 ms 2024-04-05T23:44:26.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:25 smithi016 ceph-mon[38248]: pgmap v341: 97 pgs: 97 active+clean; 213 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 2.1 MiB/s wr, 173 op/s 2024-04-05T23:44:26.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:25 smithi130 ceph-mon[79417]: pgmap v341: 97 pgs: 97 active+clean; 213 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 2.1 MiB/s wr, 173 op/s 2024-04-05T23:44:26.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:25 smithi027 ceph-mon[78597]: pgmap v341: 97 pgs: 97 active+clean; 213 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 2.1 MiB/s wr, 173 op/s 2024-04-05T23:44:27.028 INFO:tasks.workunit.client.0.smithi016.stdout: 1 530999 4.79 MB/sec execute 466 sec latency 4720.709 ms 2024-04-05T23:44:28.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 532678 4.80 MB/sec execute 467 sec latency 4916.322 ms 2024-04-05T23:44:28.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:27 smithi016 ceph-mon[38248]: pgmap v342: 97 pgs: 97 active+clean; 216 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.4 MiB/s wr, 195 op/s 2024-04-05T23:44:28.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:27 smithi130 ceph-mon[79417]: pgmap v342: 97 pgs: 97 active+clean; 216 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.4 MiB/s wr, 195 op/s 2024-04-05T23:44:28.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:27 smithi027 ceph-mon[78597]: pgmap v342: 97 pgs: 97 active+clean; 216 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.4 MiB/s wr, 195 op/s 2024-04-05T23:44:29.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 533866 4.80 MB/sec execute 468 sec latency 12.336 ms 2024-04-05T23:44:30.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 534579 4.80 MB/sec execute 469 sec latency 535.045 ms 2024-04-05T23:44:30.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:29 smithi016 ceph-mon[38248]: pgmap v343: 97 pgs: 97 active+clean; 216 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-04-05T23:44:30.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:29 smithi130 ceph-mon[79417]: pgmap v343: 97 pgs: 97 active+clean; 216 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-04-05T23:44:30.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:29 smithi027 ceph-mon[78597]: pgmap v343: 97 pgs: 97 active+clean; 216 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-04-05T23:44:31.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 534579 4.79 MB/sec execute 470 sec latency 1535.215 ms 2024-04-05T23:44:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.796+0000 7fb5e7c00700 1 -- 172.21.15.16:0/2011704719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e01019e0 msgr2=0x7fb5e0105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.796+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/2011704719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e01019e0 0x7fb5e0105a20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009990 tx=0x7fb5cc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.797+0000 7fb5e7c00700 1 -- 172.21.15.16:0/2011704719 shutdown_connections 2024-04-05T23:44:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.797+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/2011704719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5e0106150 0x7fb5e010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.797+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/2011704719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e0101030 0x7fb5e0101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.797+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/2011704719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e01019e0 0x7fb5e0105a20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.797+0000 7fb5e7c00700 1 -- 172.21.15.16:0/2011704719 >> 172.21.15.16:0/2011704719 conn(0x7fb5e00fc8b0 msgr2=0x7fb5e00fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.797+0000 7fb5e7c00700 1 -- 172.21.15.16:0/2011704719 shutdown_connections 2024-04-05T23:44:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.797+0000 7fb5e7c00700 1 -- 172.21.15.16:0/2011704719 wait complete. 2024-04-05T23:44:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.798+0000 7fb5e7c00700 1 Processor -- start 2024-04-05T23:44:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.798+0000 7fb5e7c00700 1 -- start start 2024-04-05T23:44:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.798+0000 7fb5e7c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e0101030 0x7fb5e0199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.798+0000 7fb5e7c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e0106150 0x7fb5e0199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.798+0000 7fb5e7c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5e019de10 0x7fb5e019e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.798+0000 7fb5e7c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e019e890 con 0x7fb5e0101030 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.798+0000 7fb5e7c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e019ea00 con 0x7fb5e0106150 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.798+0000 7fb5e7c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e019ebc0 con 0x7fb5e019de10 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e0101030 0x7fb5e0199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e0101030 0x7fb5e0199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45542/0 (socket says 172.21.15.16:45542) 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5200700 1 -- 172.21.15.16:0/69476373 learned_addr learned my addr 172.21.15.16:0/69476373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5c00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5e019de10 0x7fb5e019e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e0106150 0x7fb5e0199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5200700 1 -- 172.21.15.16:0/69476373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5e019de10 msgr2=0x7fb5e019e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5200700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5e019de10 0x7fb5e019e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5200700 1 -- 172.21.15.16:0/69476373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e0106150 msgr2=0x7fb5e0199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5200700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e0106150 0x7fb5e0199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5200700 1 -- 172.21.15.16:0/69476373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc0095f0 con 0x7fb5e0101030 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5e5200700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e0101030 0x7fb5e0199530 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb5d400e9a0 tx=0x7fb5d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.799+0000 7fb5dd600700 1 -- 172.21.15.16:0/69476373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4009790 con 0x7fb5e0101030 2024-04-05T23:44:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.800+0000 7fb5dd600700 1 -- 172.21.15.16:0/69476373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5d4010430 con 0x7fb5e0101030 2024-04-05T23:44:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.800+0000 7fb5e7c00700 1 -- 172.21.15.16:0/69476373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e01a6120 con 0x7fb5e0101030 2024-04-05T23:44:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.800+0000 7fb5dd600700 1 -- 172.21.15.16:0/69476373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4018680 con 0x7fb5e0101030 2024-04-05T23:44:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.800+0000 7fb5e7c00700 1 -- 172.21.15.16:0/69476373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e01a6650 con 0x7fb5e0101030 2024-04-05T23:44:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.803+0000 7fb5dd600700 1 -- 172.21.15.16:0/69476373 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fb5d40105a0 con 0x7fb5e0101030 2024-04-05T23:44:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.805+0000 7fb5fb63db80 1 Processor -- start 2024-04-05T23:44:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.805+0000 7fb5dd600700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c80a1310 0x7fb5c80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.805+0000 7fb5fb63db80 1 -- start start 2024-04-05T23:44:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.805+0000 7fb5dd600700 1 -- 172.21.15.16:0/69476373 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fb5d4014070 con 0x7fb5e0101030 2024-04-05T23:44:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.805+0000 7fb5e5c00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c80a1310 0x7fb5c80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8beb80 0x562ade8bef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8bf4a0 0x562ade8b9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 0x562ade8b9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade81fb80 con 0x562ade8beb80 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade810930 con 0x562ade8bf4a0 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade80e2f0 con 0x562ade8b95c0 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8beb80 0x562ade8bef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8beb80 0x562ade8bef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45548/0 (socket says 172.21.15.16:45548) 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5200700 1 -- 172.21.15.16:0/2577476830 learned_addr learned my addr 172.21.15.16:0/2577476830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8bf4a0 0x562ade8b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5c00700 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 0x562ade8b9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5200700 1 -- 172.21.15.16:0/2577476830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 msgr2=0x562ade8b9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5200700 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 0x562ade8b9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5200700 1 -- 172.21.15.16:0/2577476830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8bf4a0 msgr2=0x562ade8b9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5200700 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8bf4a0 0x562ade8b9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5200700 1 -- 172.21.15.16:0/2577476830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ade68b350 con 0x562ade8beb80 2024-04-05T23:44:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8bf4a0 0x562ade8b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5c00700 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 0x562ade8b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.806+0000 7fb5e5c00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c80a1310 0x7fb5c80a37d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009550 tx=0x7fb5d8009130 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.807+0000 7fb5e5200700 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8beb80 0x562ade8bef60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40969b0 tx=0x7fb5d4099bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f4c226b81a7cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.807+0000 7fb5d2000700 1 -- 172.21.15.16:0/2577476830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40a23c0 con 0x562ade8beb80 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.807+0000 7fb5d2000700 1 -- 172.21.15.16:0/2577476830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5d4010880 con 0x562ade8beb80 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.807+0000 7fb5d2000700 1 -- 172.21.15.16:0/2577476830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4099700 con 0x562ade8beb80 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 -- 172.21.15.16:0/2577476830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8beb80 msgr2=0x562ade8bef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8beb80 0x562ade8bef60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40969b0 tx=0x7fb5d4099bb0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 -- 172.21.15.16:0/2577476830 shutdown_connections 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8beb80 0x562ade8bef60 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 0x562ade8b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/2577476830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8bf4a0 0x562ade8b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 -- 172.21.15.16:0/2577476830 >> 172.21.15.16:0/2577476830 conn(0x562ade82d1c0 msgr2=0x562ade8b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 -- 172.21.15.16:0/2577476830 shutdown_connections 2024-04-05T23:44:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 -- 172.21.15.16:0/2577476830 wait complete. 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 -- start start 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.808+0000 7fb5fb63db80 1 Processor -- start 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 0x562ade8c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8beb80 0x562ade8c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8c3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade809be0 con 0x562ade8bf4a0 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade7a5d20 con 0x562ade8beb80 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade7a59c0 con 0x562ade8b95c0 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5fb63db80 10 client.0 _ll_register_callbacks cb 0x562ade611340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8beb80 0x562ade8c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5e5c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5e5200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 0x562ade8c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8beb80 0x562ade8c2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37826/0 (socket says 172.21.15.16:37826) 2024-04-05T23:44:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.809+0000 7fb5dfe00700 1 -- 172.21.15.16:0/3301842673 learned_addr learned my addr 172.21.15.16:0/3301842673 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5dfe00700 1 -- 172.21.15.16:0/3301842673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 msgr2=0x562ade8c2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 0x562ade8c2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5dfe00700 1 -- 172.21.15.16:0/3301842673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 msgr2=0x562ade8c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8c3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5dfe00700 1 -- 172.21.15.16:0/3301842673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e01a6650 con 0x562ade8beb80 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8beb80 0x562ade8c2e40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009990 tx=0x7fb5cc005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99d935281eff8090 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5c6a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5c6a00700 1 -- 172.21.15.16:0/3301842673 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc01d070 con 0x562ade8beb80 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3301842673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ade8ac1e0 con 0x562ade8beb80 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.810+0000 7fb5c6a00700 1 -- 172.21.15.16:0/3301842673 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fb5cc004500 con 0x562ade8beb80 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5c6a00700 1 -- 172.21.15.16:0/3301842673 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5cc0058e0 con 0x562ade8beb80 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5fb63db80 10 client.26732 fetch_fsmap learned FSMap version 68 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3301842673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x562ade637b00 con 0x562ade8beb80 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5c6a00700 1 -- 172.21.15.16:0/3301842673 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562ade8ac1e0 con 0x562ade8beb80 2024-04-05T23:44:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5c6a00700 1 -- 172.21.15.16:0/3301842673 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 68) v1 ==== 7542+0+0 (secure 0 0 0) 0x7fb5cc0245a0 con 0x562ade8beb80 2024-04-05T23:44:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5fb63db80 10 client.26732 fetch_fsmap finished waiting for FSMap version 68 2024-04-05T23:44:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5fb63db80 10 client.26732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5fb63db80 20 client.26732 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562ade8d3150 0x562ade8d5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5fb63db80 4 client.26732 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:44:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.811+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3301842673 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ade8026a0 con 0x562ade8d3150 2024-04-05T23:44:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.812+0000 7fb5e5200700 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562ade8d3150 0x562ade8d5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.812+0000 7fb5e5200700 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562ade8d3150 0x562ade8d5570 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c59ec433de4e6c95 server_cookie=eeb7f688a458e3f5 in_seq=0 out_seq=0 2024-04-05T23:44:31.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.812+0000 7fb5c6a00700 10 client.26732 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5c6a00700 1 -- 172.21.15.16:0/3301842673 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562ade8026a0 con 0x562ade8d3150 2024-04-05T23:44:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5c6a00700 10 client.26732 handle_command_reply: tid=0 2024-04-05T23:44:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 client.26732 shutdown 2024-04-05T23:44:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3301842673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8beb80 msgr2=0x562ade8c2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8beb80 0x562ade8c2e40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009990 tx=0x7fb5cc005800 comp rx=0 tx=0).stop 2024-04-05T23:44:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3301842673 shutdown_connections 2024-04-05T23:44:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562ade8d3150 0x562ade8d5570 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8b95c0 0x562ade8c2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3301842673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8beb80 0x562ade8c2e40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5c6a00700 0 client.26732 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3301842673 >> 172.21.15.16:0/3301842673 conn(0x562ade82d1c0 msgr2=0x562ade8b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3301842673 shutdown_connections 2024-04-05T23:44:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.813+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3301842673 wait complete. 2024-04-05T23:44:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.814+0000 7fb5fb63db80 20 client.26732 trim_cache size 0 max 16384 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.829+0000 7fb5fb63db80 1 Processor -- start 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.829+0000 7fb5fb63db80 1 -- start start 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.829+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8b9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.829+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 0x562ade8c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.829+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8be050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.829+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade68d640 con 0x562ade8bf4a0 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.829+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade8026a0 con 0x562ade8b95c0 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.829+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade8bf1b0 con 0x562ade8beb80 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8b9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37832/0 (socket says 172.21.15.16:37832) 2024-04-05T23:44:31.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5200700 1 -- 172.21.15.16:0/3703619377 learned_addr learned my addr 172.21.15.16:0/3703619377 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5c00700 1 --2- 172.21.15.16:0/3703619377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8be050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/3703619377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 0x562ade8c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5200700 1 -- 172.21.15.16:0/3703619377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 msgr2=0x562ade8c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5200700 1 --2- 172.21.15.16:0/3703619377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 0x562ade8c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5200700 1 -- 172.21.15.16:0/3703619377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 msgr2=0x562ade8be050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5200700 1 --2- 172.21.15.16:0/3703619377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8be050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5200700 1 -- 172.21.15.16:0/3703619377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ade7cfe40 con 0x562ade8b95c0 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5e5200700 1 --2- 172.21.15.16:0/3703619377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8b9030 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4098290 tx=0x7fb5d409a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b9b946597ff41a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.830+0000 7fb5d2a00700 1 -- 172.21.15.16:0/3703619377 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40a23c0 con 0x562ade8b95c0 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5d2a00700 1 -- 172.21.15.16:0/3703619377 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5d4098360 con 0x562ade8b95c0 2024-04-05T23:44:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5d2a00700 1 -- 172.21.15.16:0/3703619377 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40a1f30 con 0x562ade8b95c0 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3703619377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 msgr2=0x562ade8b9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3703619377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8b9030 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4098290 tx=0x7fb5d409a080 comp rx=0 tx=0).stop 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3703619377 shutdown_connections 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3703619377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8be050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3703619377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8b9030 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/3703619377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 0x562ade8c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3703619377 >> 172.21.15.16:0/3703619377 conn(0x562ade82d1c0 msgr2=0x562ade8bd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3703619377 shutdown_connections 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.831+0000 7fb5fb63db80 1 -- 172.21.15.16:0/3703619377 wait complete. 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5fb63db80 1 -- start start 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5fb63db80 1 Processor -- start 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8c7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 0x562ade8c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5fb63db80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade926130 con 0x562ade8bf4a0 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade9262a0 con 0x562ade8b95c0 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5fb63db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ade926460 con 0x562ade8beb80 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5fb63db80 10 client.0 _ll_register_callbacks cb 0x7fb5cc005350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5e5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.832+0000 7fb5e5200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8c7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:37844/0 (socket says 172.21.15.16:37844) 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5e5200700 1 -- 172.21.15.16:0/1973453089 learned_addr learned my addr 172.21.15.16:0/1973453089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5e5c00700 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 0x562ade8c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5e5200700 1 -- 172.21.15.16:0/1973453089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 msgr2=0x562ade8c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5e5200700 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 0x562ade8c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5e5200700 1 -- 172.21.15.16:0/1973453089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 msgr2=0x562ade8c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5e5200700 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5e5200700 1 -- 172.21.15.16:0/1973453089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ade68b350 con 0x562ade8b95c0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5e5200700 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8c7930 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4099ed0 tx=0x7fb5d409dec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56b810e141d9a53c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5c6a00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5c6a00700 1 -- 172.21.15.16:0/1973453089 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d409a3a0 con 0x562ade8b95c0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.833+0000 7fb5fb63db80 1 -- 172.21.15.16:0/1973453089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ade658c90 con 0x562ade8b95c0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5c6a00700 1 -- 172.21.15.16:0/1973453089 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fb5d40a1f30 con 0x562ade8b95c0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5c6a00700 1 -- 172.21.15.16:0/1973453089 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5d40a2130 con 0x562ade8b95c0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5fb63db80 10 client.26738 fetch_fsmap learned FSMap version 68 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5fb63db80 1 -- 172.21.15.16:0/1973453089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x562ade68fc70 con 0x562ade8b95c0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5c6a00700 1 -- 172.21.15.16:0/1973453089 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562ade658c90 con 0x562ade8b95c0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5c6a00700 1 -- 172.21.15.16:0/1973453089 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 68) v1 ==== 7542+0+0 (secure 0 0 0) 0x7fb5d4098ea0 con 0x562ade8b95c0 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5fb63db80 10 client.26738 fetch_fsmap finished waiting for FSMap version 68 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5fb63db80 10 client.26738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5fb63db80 20 client.26738 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562ade927760 0x562ade929b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:31.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5fb63db80 4 client.26738 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:44:31.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5fb63db80 1 -- 172.21.15.16:0/1973453089 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562ade68dc80 con 0x562ade927760 2024-04-05T23:44:31.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.834+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562ade927760 0x562ade929b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:31.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.835+0000 7fb5dfe00700 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562ade927760 0x562ade929b50 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59b0c323e0f53701 server_cookie=2779eca74d588910 in_seq=0 out_seq=0 2024-04-05T23:44:31.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.835+0000 7fb5c6a00700 10 client.26738 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5c6a00700 1 -- 172.21.15.16:0/1973453089 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562ade68dc80 con 0x562ade927760 2024-04-05T23:44:31.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5c6a00700 10 client.26738 handle_command_reply: tid=0 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 client.26738 shutdown 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 -- 172.21.15.16:0/1973453089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 msgr2=0x562ade8c7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8c7930 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4099ed0 tx=0x7fb5d409dec0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 -- 172.21.15.16:0/1973453089 shutdown_connections 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x562ade927760 0x562ade929b50 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ade8bf4a0 0x562ade8c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5c6a00700 0 client.26738 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ade8b95c0 0x562ade8c7930 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 --2- 172.21.15.16:0/1973453089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ade8beb80 0x562ade8c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 -- 172.21.15.16:0/1973453089 >> 172.21.15.16:0/1973453089 conn(0x562ade82d1c0 msgr2=0x562ade8c31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 -- 172.21.15.16:0/1973453089 shutdown_connections 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.874+0000 7fb5fb63db80 1 -- 172.21.15.16:0/1973453089 wait complete. 2024-04-05T23:44:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5fb63db80 20 client.26738 trim_cache size 0 max 16384 2024-04-05T23:44:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:44:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:44:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:44:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 -- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c80a1310 msgr2=0x7fb5c80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c80a1310 0x7fb5c80a37d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009550 tx=0x7fb5d8009130 comp rx=0 tx=0).stop 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 -- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e0101030 msgr2=0x7fb5e0199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e0101030 0x7fb5e0199530 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb5d400e9a0 tx=0x7fb5d400ecb0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 -- 172.21.15.16:0/69476373 shutdown_connections 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb5c80a1310 0x7fb5c80a37d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e0101030 0x7fb5e0199530 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5e019de10 0x7fb5e019e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 --2- 172.21.15.16:0/69476373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e0106150 0x7fb5e0199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.875+0000 7fb5e7c00700 1 -- 172.21.15.16:0/69476373 >> 172.21.15.16:0/69476373 conn(0x7fb5e00fc8b0 msgr2=0x7fb5e00fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.876+0000 7fb5e7c00700 1 -- 172.21.15.16:0/69476373 shutdown_connections 2024-04-05T23:44:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:31.876+0000 7fb5e7c00700 1 -- 172.21.15.16:0/69476373 wait complete. 2024-04-05T23:44:32.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 534579 4.78 MB/sec execute 471 sec latency 2535.355 ms 2024-04-05T23:44:32.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:31 smithi130 ceph-mon[79417]: pgmap v344: 97 pgs: 97 active+clean; 219 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-05T23:44:32.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:31 smithi027 ceph-mon[78597]: pgmap v344: 97 pgs: 97 active+clean; 219 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-05T23:44:32.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:31 smithi016 ceph-mon[38248]: pgmap v344: 97 pgs: 97 active+clean; 219 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-05T23:44:33.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 536250 4.78 MB/sec execute 472 sec latency 2727.847 ms 2024-04-05T23:44:34.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 537374 4.79 MB/sec execute 473 sec latency 14.539 ms 2024-04-05T23:44:34.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:33 smithi027 ceph-mon[78597]: pgmap v345: 97 pgs: 97 active+clean; 219 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-05T23:44:34.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:33 smithi016 ceph-mon[38248]: pgmap v345: 97 pgs: 97 active+clean; 219 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-05T23:44:34.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:33 smithi130 ceph-mon[79417]: pgmap v345: 97 pgs: 97 active+clean; 219 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-05T23:44:35.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 538122 4.79 MB/sec execute 474 sec latency 507.543 ms 2024-04-05T23:44:36.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 540037 4.80 MB/sec execute 475 sec latency 589.942 ms 2024-04-05T23:44:36.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:35 smithi027 ceph-mon[78597]: pgmap v346: 97 pgs: 97 active+clean; 219 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-05T23:44:36.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:35 smithi016 ceph-mon[38248]: pgmap v346: 97 pgs: 97 active+clean; 219 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-05T23:44:36.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:35 smithi130 ceph-mon[79417]: pgmap v346: 97 pgs: 97 active+clean; 219 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-05T23:44:37.029 INFO:tasks.workunit.client.0.smithi016.stdout: 1 541121 4.80 MB/sec execute 476 sec latency 13.273 ms 2024-04-05T23:44:37.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:36 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:44:37.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:36 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:44:37.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:36 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:44:38.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 541671 4.80 MB/sec execute 477 sec latency 648.468 ms 2024-04-05T23:44:38.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:37 smithi027 ceph-mon[78597]: pgmap v347: 97 pgs: 97 active+clean; 217 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-05T23:44:38.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:37 smithi016 ceph-mon[38248]: pgmap v347: 97 pgs: 97 active+clean; 217 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-05T23:44:38.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:37 smithi130 ceph-mon[79417]: pgmap v347: 97 pgs: 97 active+clean; 217 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-05T23:44:39.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 541671 4.79 MB/sec execute 478 sec latency 1648.625 ms 2024-04-05T23:44:40.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 541671 4.78 MB/sec execute 479 sec latency 2648.746 ms 2024-04-05T23:44:40.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:39 smithi027 ceph-mon[78597]: pgmap v348: 97 pgs: 97 active+clean; 217 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-05T23:44:40.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:39 smithi016 ceph-mon[38248]: pgmap v348: 97 pgs: 97 active+clean; 217 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-05T23:44:40.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:39 smithi130 ceph-mon[79417]: pgmap v348: 97 pgs: 97 active+clean; 217 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-05T23:44:41.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 541671 4.77 MB/sec execute 480 sec latency 3648.800 ms 2024-04-05T23:44:42.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 541671 4.76 MB/sec execute 481 sec latency 4648.909 ms 2024-04-05T23:44:42.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:41 smithi027 ceph-mon[78597]: pgmap v349: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-04-05T23:44:42.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:41 smithi016 ceph-mon[38248]: pgmap v349: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-04-05T23:44:42.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:41 smithi130 ceph-mon[79417]: pgmap v349: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-04-05T23:44:43.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 543328 4.76 MB/sec execute 482 sec latency 4843.459 ms 2024-04-05T23:44:44.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 544510 4.77 MB/sec execute 483 sec latency 12.438 ms 2024-04-05T23:44:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:43 smithi027 ceph-mon[78597]: pgmap v350: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:44:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:43 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:44:44.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:43 smithi016 ceph-mon[38248]: pgmap v350: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:44:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:43 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:44:44.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:43 smithi130 ceph-mon[79417]: pgmap v350: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:44:44.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:43 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:44:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.026+0000 7fdf77200700 1 -- 172.21.15.16:0/1279367891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf701019e0 msgr2=0x7fdf70105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.026+0000 7fdf77200700 1 --2- 172.21.15.16:0/1279367891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf701019e0 0x7fdf70105a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0098f0 tx=0x7fdf5c009c00 comp rx=0 tx=0).stop 2024-04-05T23:44:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.026+0000 7fdf77200700 1 -- 172.21.15.16:0/1279367891 shutdown_connections 2024-04-05T23:44:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.026+0000 7fdf77200700 1 --2- 172.21.15.16:0/1279367891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70106150 0x7fdf7010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.026+0000 7fdf77200700 1 --2- 172.21.15.16:0/1279367891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf70101030 0x7fdf70101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.026+0000 7fdf77200700 1 --2- 172.21.15.16:0/1279367891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf701019e0 0x7fdf70105a20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.026+0000 7fdf77200700 1 -- 172.21.15.16:0/1279367891 >> 172.21.15.16:0/1279367891 conn(0x7fdf700fc8b0 msgr2=0x7fdf700fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.026+0000 7fdf77200700 1 -- 172.21.15.16:0/1279367891 shutdown_connections 2024-04-05T23:44:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.026+0000 7fdf77200700 1 -- 172.21.15.16:0/1279367891 wait complete. 2024-04-05T23:44:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.027+0000 7fdf77200700 1 Processor -- start 2024-04-05T23:44:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.027+0000 7fdf77200700 1 -- start start 2024-04-05T23:44:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf77200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf70101030 0x7fdf700731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf77200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70106150 0x7fdf70073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf77200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf70073c60 0x7fdf701a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf77200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf701a5ab0 con 0x7fdf70106150 2024-04-05T23:44:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf77200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf701a5c20 con 0x7fdf70101030 2024-04-05T23:44:45.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf77200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf701a5de0 con 0x7fdf70073c60 2024-04-05T23:44:45.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70106150 0x7fdf70073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70106150 0x7fdf70073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41160/0 (socket says 172.21.15.16:41160) 2024-04-05T23:44:45.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6f400700 1 -- 172.21.15.16:0/994632630 learned_addr learned my addr 172.21.15.16:0/994632630 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:45.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf70101030 0x7fdf700731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf75200700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf70073c60 0x7fdf701a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6f400700 1 -- 172.21.15.16:0/994632630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf70073c60 msgr2=0x7fdf701a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:45.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6f400700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf70073c60 0x7fdf701a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6f400700 1 -- 172.21.15.16:0/994632630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf70101030 msgr2=0x7fdf700731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6f400700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf70101030 0x7fdf700731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6f400700 1 -- 172.21.15.16:0/994632630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf600095f0 con 0x7fdf70106150 2024-04-05T23:44:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.028+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf70101030 0x7fdf700731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.029+0000 7fdf6f400700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70106150 0x7fdf70073720 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009230 tx=0x7fdf5c00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.029+0000 7fdf6cc00700 1 -- 172.21.15.16:0/994632630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c01c070 con 0x7fdf70106150 2024-04-05T23:44:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.029+0000 7fdf6cc00700 1 -- 172.21.15.16:0/994632630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf5c00ba20 con 0x7fdf70106150 2024-04-05T23:44:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.029+0000 7fdf77200700 1 -- 172.21.15.16:0/994632630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c009520 con 0x7fdf70106150 2024-04-05T23:44:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.029+0000 7fdf6cc00700 1 -- 172.21.15.16:0/994632630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0038e0 con 0x7fdf70106150 2024-04-05T23:44:45.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.029+0000 7fdf77200700 1 -- 172.21.15.16:0/994632630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf701a62f0 con 0x7fdf70106150 2024-04-05T23:44:45.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.032+0000 7fdf6cc00700 1 -- 172.21.15.16:0/994632630 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fdf5c020070 con 0x7fdf70106150 2024-04-05T23:44:45.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.033+0000 7fdf6cc00700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdf540a1290 0x7fdf540a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf6cc00700 1 -- 172.21.15.16:0/994632630 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdf5c0cd6f0 con 0x7fdf70106150 2024-04-05T23:44:45.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf8adfcb80 1 Processor -- start 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf8adfcb80 1 -- start start 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdf540a1290 0x7fdf540a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c8082b80 0x5560c8082f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c80834a0 0x5560c807d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c7fe3b80 con 0x5560c80834a0 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c7fd4930 con 0x5560c807d5c0 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.034+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c7fd22f0 con 0x5560c8082b80 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf6fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c8082b80 0x5560c8082f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf6f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c80834a0 0x5560c807d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf6fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c8082b80 0x5560c8082f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:52822/0 (socket says 172.21.15.16:52822) 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf6f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c80834a0 0x5560c807d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41174/0 (socket says 172.21.15.16:41174) 2024-04-05T23:44:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf6fe00700 1 -- 172.21.15.16:0/770332782 learned_addr learned my addr 172.21.15.16:0/770332782 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf75200700 1 --2- 172.21.15.16:0/770332782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdf540a1290 0x7fdf540a3750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdf60005fd0 tx=0x7fdf60009310 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf75200700 1 -- 172.21.15.16:0/770332782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c8082b80 msgr2=0x5560c8082f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf75200700 1 --2- 172.21.15.16:0/770332782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c8082b80 0x5560c8082f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf75200700 1 -- 172.21.15.16:0/770332782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c80834a0 msgr2=0x5560c807d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf75200700 1 --2- 172.21.15.16:0/770332782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c80834a0 0x5560c807d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf75200700 1 -- 172.21.15.16:0/770332782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560c7e4f350 con 0x5560c807d5c0 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/770332782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c8082b80 0x5560c8082f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.035+0000 7fdf75200700 1 --2- 172.21.15.16:0/770332782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807da20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdf68011580 tx=0x7fdf68009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c6dae3757db215b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.036+0000 7fdf65600700 1 -- 172.21.15.16:0/770332782 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6800e040 con 0x5560c807d5c0 2024-04-05T23:44:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.036+0000 7fdf65600700 1 -- 172.21.15.16:0/770332782 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf6800f040 con 0x5560c807d5c0 2024-04-05T23:44:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.036+0000 7fdf65600700 1 -- 172.21.15.16:0/770332782 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68004370 con 0x5560c807d5c0 2024-04-05T23:44:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.036+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/770332782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 msgr2=0x5560c807da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.036+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/770332782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807da20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdf68011580 tx=0x7fdf68009920 comp rx=0 tx=0).stop 2024-04-05T23:44:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.037+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/770332782 shutdown_connections 2024-04-05T23:44:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.037+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/770332782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807da20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.037+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/770332782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c8082b80 0x5560c8082f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.037+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/770332782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c80834a0 0x5560c807d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.037+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/770332782 >> 172.21.15.16:0/770332782 conn(0x5560c7ff11c0 msgr2=0x5560c8077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.037+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/770332782 shutdown_connections 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.037+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/770332782 wait complete. 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.037+0000 7fdf8adfcb80 1 -- start start 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.037+0000 7fdf8adfcb80 1 Processor -- start 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c807d5c0 0x5560c8089460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c8086750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c80834a0 0x5560c8086df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c7fcdbe0 con 0x5560c8082b80 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c7f69d20 con 0x5560c80834a0 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c7f699c0 con 0x5560c807d5c0 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf8adfcb80 10 client.0 _ll_register_callbacks cb 0x5560c7dd5340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf6fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c807d5c0 0x5560c8089460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf6f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c8086750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf6fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c807d5c0 0x5560c8089460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:52836/0 (socket says 172.21.15.16:52836) 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf6fe00700 1 -- 172.21.15.16:0/1398589609 learned_addr learned my addr 172.21.15.16:0/1398589609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf75200700 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c80834a0 0x5560c8086df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf6f400700 1 -- 172.21.15.16:0/1398589609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c807d5c0 msgr2=0x5560c8089460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf6f400700 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c807d5c0 0x5560c8089460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf6f400700 1 -- 172.21.15.16:0/1398589609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c80834a0 msgr2=0x5560c8086df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf6f400700 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c80834a0 0x5560c8086df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf6f400700 1 -- 172.21.15.16:0/1398589609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf60017050 con 0x5560c8082b80 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.038+0000 7fdf75200700 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c80834a0 0x5560c8086df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.039+0000 7fdf6f400700 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c8086750 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0cadc0 tx=0x7fdf5c09a260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7356b3ca35125697 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.039+0000 7fdf5a000700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:44:45.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 545212 4.77 MB/sec execute 484 sec latency 555.273 ms 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.039+0000 7fdf5a000700 1 -- 172.21.15.16:0/1398589609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0a2c00 con 0x5560c8082b80 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.039+0000 7fdf5a000700 1 -- 172.21.15.16:0/1398589609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdf5c005040 con 0x5560c8082b80 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.039+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1398589609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560c80701e0 con 0x5560c8082b80 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.039+0000 7fdf5a000700 1 -- 172.21.15.16:0/1398589609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf5c005200 con 0x5560c8082b80 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.039+0000 7fdf8adfcb80 10 client.16701 fetch_fsmap learned FSMap version 70 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.039+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1398589609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) v3 -- 0x7fdf68011230 con 0x5560c8082b80 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.039+0000 7fdf5a000700 1 -- 172.21.15.16:0/1398589609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5560c80701e0 con 0x5560c8082b80 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.040+0000 7fdf5a000700 1 -- 172.21.15.16:0/1398589609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 70) v1 ==== 7522+0+0 (secure 0 0 0) 0x7fdf5c099d00 con 0x5560c8082b80 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.040+0000 7fdf8adfcb80 10 client.16701 fetch_fsmap finished waiting for FSMap version 70 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.040+0000 7fdf8adfcb80 10 client.16701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.040+0000 7fdf8adfcb80 20 client.16701 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.040+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5560c8096c50 0x5560c8099070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.040+0000 7fdf8adfcb80 4 client.16701 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.040+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1398589609 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560c7fc66a0 con 0x5560c8096c50 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.040+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5560c8096c50 0x5560c8099070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5560c8096c50 0x5560c8099070 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fde33cb9abc69254 server_cookie=ab4691801edd4be0 in_seq=0 out_seq=0 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf5a000700 10 client.16701 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf5a000700 1 -- 172.21.15.16:0/1398589609 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5560c7fc66a0 con 0x5560c8096c50 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf5a000700 10 client.16701 handle_command_reply: tid=0 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf8adfcb80 1 client.16701 shutdown 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1398589609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 msgr2=0x5560c8086750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c8086750 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0cadc0 tx=0x7fdf5c09a260 comp rx=0 tx=0).stop 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1398589609 shutdown_connections 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5560c8096c50 0x5560c8099070 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c80834a0 0x5560c8086df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c807d5c0 0x5560c8089460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf5a000700 0 client.16701 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1398589609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c8086750 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.041+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1398589609 >> 172.21.15.16:0/1398589609 conn(0x5560c7ff11c0 msgr2=0x5560c8077280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.042+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1398589609 shutdown_connections 2024-04-05T23:44:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.042+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1398589609 wait complete. 2024-04-05T23:44:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.042+0000 7fdf8adfcb80 20 client.16701 trim_cache size 0 max 16384 2024-04-05T23:44:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf8adfcb80 1 Processor -- start 2024-04-05T23:44:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf8adfcb80 1 -- start start 2024-04-05T23:44:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c807d5c0 0x5560c807d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c8082b80 0x5560c8077a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 0x5560c80926b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c7e51640 con 0x5560c807d5c0 2024-04-05T23:44:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c8079600 con 0x5560c8082b80 2024-04-05T23:44:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c8082f60 con 0x5560c80834a0 2024-04-05T23:44:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf6fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c807d5c0 0x5560c807d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf6fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c807d5c0 0x5560c807d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41190/0 (socket says 172.21.15.16:41190) 2024-04-05T23:44:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.058+0000 7fdf6fe00700 1 -- 172.21.15.16:0/1322549076 learned_addr learned my addr 172.21.15.16:0/1322549076 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf6f400700 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c8082b80 0x5560c8077a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf75200700 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 0x5560c80926b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf6fe00700 1 -- 172.21.15.16:0/1322549076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 msgr2=0x5560c80926b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 0x5560c80926b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf6fe00700 1 -- 172.21.15.16:0/1322549076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c8082b80 msgr2=0x5560c8077a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c8082b80 0x5560c8077a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf6fe00700 1 -- 172.21.15.16:0/1322549076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560c7dfbb00 con 0x5560c807d5c0 2024-04-05T23:44:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf6f400700 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c8082b80 0x5560c8077a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c807d5c0 0x5560c807d030 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x5560c8099920 tx=0x7fdf60019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b156d67bb2ddf54b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf66000700 1 -- 172.21.15.16:0/1322549076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60021030 con 0x5560c807d5c0 2024-04-05T23:44:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf66000700 1 -- 172.21.15.16:0/1322549076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf60022040 con 0x5560c807d5c0 2024-04-05T23:44:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.059+0000 7fdf66000700 1 -- 172.21.15.16:0/1322549076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60003d20 con 0x5560c807d5c0 2024-04-05T23:44:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.060+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1322549076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c807d5c0 msgr2=0x5560c807d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.060+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c807d5c0 0x5560c807d030 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x5560c8099920 tx=0x7fdf60019000 comp rx=0 tx=0).stop 2024-04-05T23:44:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.060+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1322549076 shutdown_connections 2024-04-05T23:44:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.060+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c807d5c0 0x5560c807d030 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.060+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 0x5560c80926b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.060+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/1322549076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c8082b80 0x5560c8077a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.060+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1322549076 >> 172.21.15.16:0/1322549076 conn(0x5560c7ff11c0 msgr2=0x5560c8085040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.060+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1322549076 shutdown_connections 2024-04-05T23:44:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.060+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/1322549076 wait complete. 2024-04-05T23:44:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.061+0000 7fdf8adfcb80 1 -- start start 2024-04-05T23:44:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.061+0000 7fdf8adfcb80 1 Processor -- start 2024-04-05T23:44:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.061+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.061+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c807e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.061+0000 7fdf8adfcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 0x5560c807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.061+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c80802d0 con 0x5560c8082b80 2024-04-05T23:44:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.061+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c807f080 con 0x5560c807d5c0 2024-04-05T23:44:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.061+0000 7fdf8adfcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560c807f240 con 0x5560c80834a0 2024-04-05T23:44:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.061+0000 7fdf8adfcb80 10 client.0 _ll_register_callbacks cb 0x7fdf5c09a6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:45.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c807e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:56144/0 (socket says 172.21.15.16:56144) 2024-04-05T23:44:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6fe00700 1 -- 172.21.15.16:0/2911222587 learned_addr learned my addr 172.21.15.16:0/2911222587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf75200700 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 0x5560c807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6fe00700 1 -- 172.21.15.16:0/2911222587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 msgr2=0x5560c807eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:45.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 0x5560c807eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6fe00700 1 -- 172.21.15.16:0/2911222587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 msgr2=0x5560c807e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c807e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6fe00700 1 -- 172.21.15.16:0/2911222587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf68011230 con 0x5560c807d5c0 2024-04-05T23:44:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6f400700 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c807e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf6fe00700 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807fbc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000fe20 tx=0x7fdf60005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f247951caa0bb5c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf5a000700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:44:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf5a000700 1 -- 172.21.15.16:0/2911222587 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60021030 con 0x5560c807d5c0 2024-04-05T23:44:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/2911222587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdf6800f040 con 0x5560c807d5c0 2024-04-05T23:44:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.062+0000 7fdf5a000700 1 -- 172.21.15.16:0/2911222587 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdf60003700 con 0x5560c807d5c0 2024-04-05T23:44:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf5a000700 1 -- 172.21.15.16:0/2911222587 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf600039a0 con 0x5560c807d5c0 2024-04-05T23:44:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf8adfcb80 10 client.26756 fetch_fsmap learned FSMap version 70 2024-04-05T23:44:45.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/2911222587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) v3 -- 0x7fdf60012220 con 0x5560c807d5c0 2024-04-05T23:44:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf5a000700 1 -- 172.21.15.16:0/2911222587 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf6800f040 con 0x5560c807d5c0 2024-04-05T23:44:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf5a000700 1 -- 172.21.15.16:0/2911222587 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 70) v1 ==== 7522+0+0 (secure 0 0 0) 0x7fdf600125d0 con 0x5560c807d5c0 2024-04-05T23:44:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf8adfcb80 10 client.26756 fetch_fsmap finished waiting for FSMap version 70 2024-04-05T23:44:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf8adfcb80 10 client.26756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf8adfcb80 20 client.26756 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:45.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5560c80ebd30 0x5560c80ee150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf8adfcb80 4 client.26756 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:44:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.063+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/2911222587 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5560c7e1cc90 con 0x5560c80ebd30 2024-04-05T23:44:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.064+0000 7fdf6f400700 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5560c80ebd30 0x5560c80ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.064+0000 7fdf6f400700 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5560c80ebd30 0x5560c80ee150 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d31606ecc9ecb085 server_cookie=315f44e605cc526d in_seq=0 out_seq=0 2024-04-05T23:44:45.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.064+0000 7fdf5a000700 10 client.26756 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.092+0000 7fdf5a000700 1 -- 172.21.15.16:0/2911222587 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5560c7e1cc90 con 0x5560c80ebd30 2024-04-05T23:44:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.092+0000 7fdf5a000700 10 client.26756 handle_command_reply: tid=0 2024-04-05T23:44:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.092+0000 7fdf8adfcb80 1 client.26756 shutdown 2024-04-05T23:44:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.092+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/2911222587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 msgr2=0x5560c807fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.092+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807fbc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000fe20 tx=0x7fdf60005dc0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/2911222587 shutdown_connections 2024-04-05T23:44:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5560c80ebd30 0x5560c80ee150 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560c80834a0 0x5560c807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf5a000700 0 client.26756 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5560c807d5c0 0x5560c807fbc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf8adfcb80 1 --2- 172.21.15.16:0/2911222587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560c8082b80 0x5560c807e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/2911222587 >> 172.21.15.16:0/2911222587 conn(0x5560c7ff11c0 msgr2=0x5560c80817e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/2911222587 shutdown_connections 2024-04-05T23:44:45.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf8adfcb80 1 -- 172.21.15.16:0/2911222587 wait complete. 2024-04-05T23:44:45.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.093+0000 7fdf8adfcb80 20 client.26756 trim_cache size 0 max 16384 2024-04-05T23:44:45.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:44:45.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:44:45.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:44:45.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:44:45.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 -- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdf540a1290 msgr2=0x7fdf540a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:45.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdf540a1290 0x7fdf540a3750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdf60005fd0 tx=0x7fdf60009310 comp rx=0 tx=0).stop 2024-04-05T23:44:45.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 -- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70106150 msgr2=0x7fdf70073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:45.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70106150 0x7fdf70073720 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009230 tx=0x7fdf5c00fa00 comp rx=0 tx=0).stop 2024-04-05T23:44:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 -- 172.21.15.16:0/994632630 shutdown_connections 2024-04-05T23:44:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdf540a1290 0x7fdf540a3750 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf70073c60 0x7fdf701a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf70101030 0x7fdf700731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 --2- 172.21.15.16:0/994632630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf70106150 0x7fdf70073720 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 -- 172.21.15.16:0/994632630 >> 172.21.15.16:0/994632630 conn(0x7fdf700fc8b0 msgr2=0x7fdf700fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 -- 172.21.15.16:0/994632630 shutdown_connections 2024-04-05T23:44:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:45.094+0000 7fdf77200700 1 -- 172.21.15.16:0/994632630 wait complete. 2024-04-05T23:44:46.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 545212 4.76 MB/sec execute 485 sec latency 1555.380 ms 2024-04-05T23:44:46.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:45 smithi027 ceph-mon[78597]: pgmap v351: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:44:46.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:45 smithi016 ceph-mon[38248]: pgmap v351: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:44:46.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:45 smithi130 ceph-mon[79417]: pgmap v351: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-05T23:44:47.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 545212 4.75 MB/sec execute 486 sec latency 2555.518 ms 2024-04-05T23:44:48.030 INFO:tasks.workunit.client.0.smithi016.stdout: 1 546891 4.76 MB/sec execute 487 sec latency 2746.405 ms 2024-04-05T23:44:48.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:47 smithi027 ceph-mon[78597]: pgmap v352: 97 pgs: 97 active+clean; 221 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-05T23:44:48.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:47 smithi016 ceph-mon[38248]: pgmap v352: 97 pgs: 97 active+clean; 221 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-05T23:44:48.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:47 smithi130 ceph-mon[79417]: pgmap v352: 97 pgs: 97 active+clean; 221 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-05T23:44:49.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 548049 4.76 MB/sec execute 488 sec latency 21.573 ms 2024-04-05T23:44:50.031 INFO:tasks.workunit.client.0.smithi016.stdout: 1 548762 4.76 MB/sec execute 489 sec latency 570.576 ms 2024-04-05T23:44:50.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:49 smithi027 ceph-mon[78597]: pgmap v353: 97 pgs: 97 active+clean; 221 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-05T23:44:50.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:49 smithi016 ceph-mon[38248]: pgmap v353: 97 pgs: 97 active+clean; 221 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-05T23:44:50.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:49 smithi130 ceph-mon[79417]: pgmap v353: 97 pgs: 97 active+clean; 221 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-05T23:44:51.031 INFO:tasks.workunit.client.0.smithi016.stdout: 1 550602 4.77 MB/sec execute 490 sec latency 643.984 ms 2024-04-05T23:44:52.031 INFO:tasks.workunit.client.0.smithi016.stdout: 1 551682 4.77 MB/sec execute 491 sec latency 16.069 ms 2024-04-05T23:44:52.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:51 smithi027 ceph-mon[78597]: pgmap v354: 97 pgs: 97 active+clean; 225 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.0 MiB/s wr, 174 op/s 2024-04-05T23:44:52.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:51 smithi016 ceph-mon[38248]: pgmap v354: 97 pgs: 97 active+clean; 225 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.0 MiB/s wr, 174 op/s 2024-04-05T23:44:52.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:51 smithi130 ceph-mon[79417]: pgmap v354: 97 pgs: 97 active+clean; 225 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.0 MiB/s wr, 174 op/s 2024-04-05T23:44:53.031 INFO:tasks.workunit.client.0.smithi016.stdout: 1 552305 4.77 MB/sec execute 492 sec latency 626.042 ms 2024-04-05T23:44:54.031 INFO:tasks.workunit.client.0.smithi016.stdout: 1 552305 4.76 MB/sec execute 493 sec latency 1626.124 ms 2024-04-05T23:44:54.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:53 smithi027 ceph-mon[78597]: pgmap v355: 97 pgs: 97 active+clean; 225 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-04-05T23:44:54.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:53 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:44:54.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:53 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:44:54.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:53 smithi016 ceph-mon[38248]: pgmap v355: 97 pgs: 97 active+clean; 225 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-04-05T23:44:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:53 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:44:54.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:53 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:44:54.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:53 smithi130 ceph-mon[79417]: pgmap v355: 97 pgs: 97 active+clean; 225 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-04-05T23:44:54.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:53 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:44:54.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:53 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:44:55.031 INFO:tasks.workunit.client.0.smithi016.stdout: 1 552305 4.75 MB/sec execute 494 sec latency 2626.168 ms 2024-04-05T23:44:55.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:54 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:44:55.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:54 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:44:55.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:54 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:44:55.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:54 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:44:55.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:54 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:44:55.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:54 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:44:55.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:54 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:44:55.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:54 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:44:55.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:54 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:44:56.031 INFO:tasks.workunit.client.0.smithi016.stdout: 1 552305 4.74 MB/sec execute 495 sec latency 3626.259 ms 2024-04-05T23:44:56.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:56 smithi016 ceph-mon[38248]: pgmap v356: 97 pgs: 97 active+clean; 225 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-04-05T23:44:56.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:56 smithi130 ceph-mon[79417]: pgmap v356: 97 pgs: 97 active+clean; 225 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-04-05T23:44:56.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:56 smithi027 ceph-mon[78597]: pgmap v356: 97 pgs: 97 active+clean; 225 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.3 MiB/s wr, 128 op/s 2024-04-05T23:44:57.031 INFO:tasks.workunit.client.0.smithi016.stdout: 1 552305 4.73 MB/sec execute 496 sec latency 4626.391 ms 2024-04-05T23:44:58.031 INFO:tasks.workunit.client.0.smithi016.stdout: 1 553908 4.74 MB/sec execute 497 sec latency 4816.363 ms 2024-04-05T23:44:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.246+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3419447957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c8101040 msgr2=0x7fc8c8101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.246+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3419447957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c8101040 0x7fc8c8101420 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80098f0 tx=0x7fc8b8009c00 comp rx=0 tx=0).stop 2024-04-05T23:44:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.246+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3419447957 shutdown_connections 2024-04-05T23:44:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.246+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3419447957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c8101040 0x7fc8c8101420 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.246+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3419447957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c8106160 0x7fc8c810dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.246+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3419447957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8c81019f0 0x7fc8c8105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.246+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3419447957 >> 172.21.15.16:0/3419447957 conn(0x7fc8c80fc8e0 msgr2=0x7fc8c80fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.246+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3419447957 shutdown_connections 2024-04-05T23:44:58.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.246+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3419447957 wait complete. 2024-04-05T23:44:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.247+0000 7fc8ccc00700 1 Processor -- start 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.247+0000 7fc8ccc00700 1 -- start start 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8ccc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8c8101040 0x7fc8c8197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8ccc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c8106160 0x7fc8c81978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8ccc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c819bc20 0x7fc8c819c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8ccc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c819c6a0 con 0x7fc8c819bc20 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8ccc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c819c810 con 0x7fc8c8106160 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8ccc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c819c9d0 con 0x7fc8c8101040 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c8106160 0x7fc8c81978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8c8101040 0x7fc8c8197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c8106160 0x7fc8c81978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51030/0 (socket says 172.21.15.16:51030) 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8c8101040 0x7fc8c8197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:36224/0 (socket says 172.21.15.16:36224) 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5400700 1 -- 172.21.15.16:0/3698138436 learned_addr learned my addr 172.21.15.16:0/3698138436 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c6800700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c819bc20 0x7fc8c819c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5400700 1 -- 172.21.15.16:0/3698138436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8c8101040 msgr2=0x7fc8c8197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5400700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8c8101040 0x7fc8c8197390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5400700 1 -- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c819bc20 msgr2=0x7fc8c819c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5400700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c819bc20 0x7fc8c819c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c5400700 1 -- 172.21.15.16:0/3698138436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c0009d00 con 0x7fc8c8106160 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.248+0000 7fc8c6800700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c819bc20 0x7fc8c819c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.249+0000 7fc8c5400700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c8106160 0x7fc8c81978d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400e970 tx=0x7fc8b400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.249+0000 7fc8bea00700 1 -- 172.21.15.16:0/3698138436 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4009790 con 0x7fc8c8106160 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.249+0000 7fc8bea00700 1 -- 172.21.15.16:0/3698138436 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8b400eec0 con 0x7fc8c8106160 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.249+0000 7fc8bea00700 1 -- 172.21.15.16:0/3698138436 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4005590 con 0x7fc8c8106160 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.249+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3698138436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b8009520 con 0x7fc8c8106160 2024-04-05T23:44:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.249+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3698138436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c80734f0 con 0x7fc8c8106160 2024-04-05T23:44:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.252+0000 7fc8bea00700 1 -- 172.21.15.16:0/3698138436 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fc8b400cbd0 con 0x7fc8c8106160 2024-04-05T23:44:58.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.254+0000 7fc8e07cdb80 1 Processor -- start 2024-04-05T23:44:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.254+0000 7fc8e07cdb80 1 -- start start 2024-04-05T23:44:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.254+0000 7fc8bea00700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc8ac0a1310 0x7fc8ac0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8bea00700 1 -- 172.21.15.16:0/3698138436 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fc8b4014070 con 0x7fc8c8106160 2024-04-05T23:44:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c6800700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc8ac0a1310 0x7fc8ac0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 0x563d5f053f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f04e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f04ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5efb4b80 con 0x563d5f053b80 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5efa5930 con 0x563d5f04e5c0 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5efa32f0 con 0x563d5f0544a0 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 0x563d5f053f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f04e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f04e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:36226/0 (socket says 172.21.15.16:36226) 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 0x563d5f053f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33564/0 (socket says 172.21.15.16:33564) 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c5400700 1 -- 172.21.15.16:0/3046662939 learned_addr learned my addr 172.21.15.16:0/3046662939 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c6800700 1 --2- 172.21.15.16:0/3046662939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f04ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c5400700 1 -- 172.21.15.16:0/3046662939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 msgr2=0x563d5f04ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c5400700 1 --2- 172.21.15.16:0/3046662939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f04ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c5400700 1 -- 172.21.15.16:0/3046662939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 msgr2=0x563d5f053f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.255+0000 7fc8c5400700 1 --2- 172.21.15.16:0/3046662939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 0x563d5f053f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.256+0000 7fc8c5400700 1 -- 172.21.15.16:0/3046662939 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d5ee20350 con 0x563d5f0544a0 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.256+0000 7fc8c6800700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc8ac0a1310 0x7fc8ac0a37d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009fd0 tx=0x7fc8c0007ca0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.256+0000 7fc8c5400700 1 --2- 172.21.15.16:0/3046662939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f04e030 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4096a80 tx=0x7fc8b409a790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35d69dde62d33857 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.256+0000 7fc8b3000700 1 -- 172.21.15.16:0/3046662939 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b40a2490 con 0x563d5f0544a0 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.256+0000 7fc8b3000700 1 -- 172.21.15.16:0/3046662939 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8b4096720 con 0x563d5f0544a0 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.256+0000 7fc8b3000700 1 -- 172.21.15.16:0/3046662939 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4099700 con 0x563d5f0544a0 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/3046662939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 msgr2=0x563d5f04e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/3046662939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f04e030 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4096a80 tx=0x7fc8b409a790 comp rx=0 tx=0).stop 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/3046662939 shutdown_connections 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/3046662939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 0x563d5f053f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/3046662939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f04ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/3046662939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f04e030 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/3046662939 >> 172.21.15.16:0/3046662939 conn(0x563d5efc21c0 msgr2=0x563d5f048be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/3046662939 shutdown_connections 2024-04-05T23:44:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/3046662939 wait complete. 2024-04-05T23:44:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.257+0000 7fc8e07cdb80 1 -- start start 2024-04-05T23:44:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8e07cdb80 1 Processor -- start 2024-04-05T23:44:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f053b80 0x563d5f057800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f0544a0 0x563d5f057f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5ef9ebe0 con 0x563d5f0544a0 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5ef3ad20 con 0x563d5f04e5c0 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5ef3a9c0 con 0x563d5f053b80 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8e07cdb80 10 client.0 _ll_register_callbacks cb 0x563d5eda6340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51054/0 (socket says 172.21.15.16:51054) 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8c5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f053b80 0x563d5f057800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8c5e00700 1 -- 172.21.15.16:0/583118604 learned_addr learned my addr 172.21.15.16:0/583118604 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.258+0000 7fc8c6800700 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f0544a0 0x563d5f057f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8c5400700 1 -- 172.21.15.16:0/583118604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 msgr2=0x563d5f05a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8c5400700 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8c5400700 1 -- 172.21.15.16:0/583118604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f0544a0 msgr2=0x563d5f057f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8c5400700 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f0544a0 0x563d5f057f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8c5400700 1 -- 172.21.15.16:0/583118604 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c80734f0 con 0x563d5f053b80 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8c5e00700 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8c5400700 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f053b80 0x563d5f057800 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4099a70 tx=0x7fc8b4005910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68f7877a81e28cd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8a7400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8a7400700 1 -- 172.21.15.16:0/583118604 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b40a2490 con 0x563d5f053b80 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8a7400700 1 -- 172.21.15.16:0/583118604 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fc8b40a2210 con 0x563d5f053b80 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/583118604 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d5f0411e0 con 0x563d5f053b80 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8a7400700 1 -- 172.21.15.16:0/583118604 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8b40b2bc0 con 0x563d5f053b80 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.259+0000 7fc8e07cdb80 10 client.26791 fetch_fsmap learned FSMap version 71 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/583118604 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) v3 -- 0x563d5edccb00 con 0x563d5f053b80 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8a7400700 1 -- 172.21.15.16:0/583118604 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563d5f0411e0 con 0x563d5f053b80 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8a7400700 1 -- 172.21.15.16:0/583118604 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 71) v1 ==== 7518+0+0 (secure 0 0 0) 0x7fc8b4098f70 con 0x563d5f053b80 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8e07cdb80 10 client.26791 fetch_fsmap finished waiting for FSMap version 71 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8e07cdb80 10 client.26791 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8e07cdb80 20 client.26791 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563d5f067ba0 0x563d5f069fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8e07cdb80 4 client.26791 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/583118604 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d5ef976a0 con 0x563d5f067ba0 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.260+0000 7fc8c5e00700 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563d5f067ba0 0x563d5f069fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.261+0000 7fc8c5e00700 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563d5f067ba0 0x563d5f069fc0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c95f86141747221 server_cookie=a27be857dfb8a3a6 in_seq=0 out_seq=0 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.261+0000 7fc8a7400700 10 client.26791 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.261+0000 7fc8a7400700 1 -- 172.21.15.16:0/583118604 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563d5ef976a0 con 0x563d5f067ba0 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.261+0000 7fc8a7400700 10 client.26791 handle_command_reply: tid=0 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.261+0000 7fc8e07cdb80 1 client.26791 shutdown 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/583118604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f053b80 msgr2=0x563d5f057800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f053b80 0x563d5f057800 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4099a70 tx=0x7fc8b4005910 comp rx=0 tx=0).stop 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/583118604 shutdown_connections 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563d5f067ba0 0x563d5f069fc0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f0544a0 0x563d5f057f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/583118604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f053b80 0x563d5f057800 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8a7400700 0 client.26791 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/583118604 >> 172.21.15.16:0/583118604 conn(0x563d5efc21c0 msgr2=0x563d5f048280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/583118604 shutdown_connections 2024-04-05T23:44:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/583118604 wait complete. 2024-04-05T23:44:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.262+0000 7fc8e07cdb80 20 client.26791 trim_cache size 0 max 16384 2024-04-05T23:44:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.277+0000 7fc8e07cdb80 1 Processor -- start 2024-04-05T23:44:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.277+0000 7fc8e07cdb80 1 -- start start 2024-04-05T23:44:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f04e5c0 0x563d5f04e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f053b80 0x563d5f05ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f048a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5ee22640 con 0x563d5f04e5c0 2024-04-05T23:44:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5ef976a0 con 0x563d5f053b80 2024-04-05T23:44:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5f058730 con 0x563d5f0544a0 2024-04-05T23:44:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f053b80 0x563d5f05ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f04e5c0 0x563d5f04e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f053b80 0x563d5f05ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51060/0 (socket says 172.21.15.16:51060) 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5400700 1 -- 172.21.15.16:0/257066807 learned_addr learned my addr 172.21.15.16:0/257066807 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f04e5c0 0x563d5f04e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33580/0 (socket says 172.21.15.16:33580) 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c6800700 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f048a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5e00700 1 -- 172.21.15.16:0/257066807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 msgr2=0x563d5f048a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5e00700 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f048a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5e00700 1 -- 172.21.15.16:0/257066807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f053b80 msgr2=0x563d5f05ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5e00700 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f053b80 0x563d5f05ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5e00700 1 -- 172.21.15.16:0/257066807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d5ef64e40 con 0x563d5f04e5c0 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c5400700 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f053b80 0x563d5f05ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.278+0000 7fc8c6800700 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f048a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:44:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.279+0000 7fc8c5e00700 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f04e5c0 0x563d5f04e030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x563d5f06a890 tx=0x7fc8b8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7040fadc3328d2fb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.279+0000 7fc8b3a00700 1 -- 172.21.15.16:0/257066807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800f840 con 0x563d5f04e5c0 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.279+0000 7fc8b3a00700 1 -- 172.21.15.16:0/257066807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8b8021040 con 0x563d5f04e5c0 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.279+0000 7fc8b3a00700 1 -- 172.21.15.16:0/257066807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b801ba70 con 0x563d5f04e5c0 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/257066807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f04e5c0 msgr2=0x563d5f04e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f04e5c0 0x563d5f04e030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x563d5f06a890 tx=0x7fc8b8005dc0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/257066807 shutdown_connections 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f04e5c0 0x563d5f04e030 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f048a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/257066807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f053b80 0x563d5f05ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/257066807 >> 172.21.15.16:0/257066807 conn(0x563d5efc21c0 msgr2=0x563d5f0519a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:58.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/257066807 shutdown_connections 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/257066807 wait complete. 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 -- start start 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.280+0000 7fc8e07cdb80 1 Processor -- start 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 0x563d5f05d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8e07cdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f0ba990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5f05ddc0 con 0x563d5f053b80 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5f0baed0 con 0x563d5f04e5c0 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8e07cdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5f0bb010 con 0x563d5f0544a0 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8e07cdb80 10 client.0 _ll_register_callbacks cb 0x563d5eda6340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8c5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51074/0 (socket says 172.21.15.16:51074) 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8c5e00700 1 -- 172.21.15.16:0/2001285331 learned_addr learned my addr 172.21.15.16:0/2001285331 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8c6800700 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f0ba990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.281+0000 7fc8c5400700 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 0x563d5f05d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8c5e00700 1 -- 172.21.15.16:0/2001285331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 msgr2=0x563d5f0ba990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8c5e00700 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f0ba990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8c5e00700 1 -- 172.21.15.16:0/2001285331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 msgr2=0x563d5f05d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8c5e00700 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 0x563d5f05d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8c5e00700 1 -- 172.21.15.16:0/2001285331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d5edccb00 con 0x563d5f04e5c0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8c5e00700 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05d130 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800c010 tx=0x7fc8b800f950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5830b09be8744227 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8a7400700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8a7400700 1 -- 172.21.15.16:0/2001285331 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b802bc80 con 0x563d5f04e5c0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8a7400700 1 -- 172.21.15.16:0/2001285331 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fc8b802bde0 con 0x563d5f04e5c0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/2001285331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc8b4096720 con 0x563d5f04e5c0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8a7400700 1 -- 172.21.15.16:0/2001285331 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8b80050b0 con 0x563d5f04e5c0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8e07cdb80 10 client.26774 fetch_fsmap learned FSMap version 71 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.282+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/2001285331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) v3 -- 0x7fc8c0021040 con 0x563d5f04e5c0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.283+0000 7fc8a7400700 1 -- 172.21.15.16:0/2001285331 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8b4096720 con 0x563d5f04e5c0 2024-04-05T23:44:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.283+0000 7fc8a7400700 1 -- 172.21.15.16:0/2001285331 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 71) v1 ==== 7518+0+0 (secure 0 0 0) 0x7fc8b8009920 con 0x563d5f04e5c0 2024-04-05T23:44:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.283+0000 7fc8e07cdb80 10 client.26774 fetch_fsmap finished waiting for FSMap version 71 2024-04-05T23:44:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.283+0000 7fc8e07cdb80 10 client.26774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:44:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.283+0000 7fc8e07cdb80 20 client.26774 populate_metadata read hostname 'smithi016' 2024-04-05T23:44:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.283+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563d5f0bc500 0x563d5f0be920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:44:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.283+0000 7fc8e07cdb80 4 client.26774 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:44:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.283+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/2001285331 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563d5ededc90 con 0x563d5f0bc500 2024-04-05T23:44:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.283+0000 7fc8c5400700 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563d5f0bc500 0x563d5f0be920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:44:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.284+0000 7fc8c5400700 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563d5f0bc500 0x563d5f0be920 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b9d246a451018cc server_cookie=5f5406620d8fa85b in_seq=0 out_seq=0 2024-04-05T23:44:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.284+0000 7fc8a7400700 10 client.26774 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8a7400700 1 -- 172.21.15.16:0/2001285331 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563d5ededc90 con 0x563d5f0bc500 2024-04-05T23:44:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8a7400700 10 client.26774 handle_command_reply: tid=0 2024-04-05T23:44:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 client.26774 shutdown 2024-04-05T23:44:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/2001285331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 msgr2=0x563d5f05d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05d130 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800c010 tx=0x7fc8b800f950 comp rx=0 tx=0).stop 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/2001285331 shutdown_connections 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x563d5f0bc500 0x563d5f0be920 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d5f0544a0 0x563d5f0ba990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d5f04e5c0 0x563d5f05d130 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8a7400700 0 client.26774 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 --2- 172.21.15.16:0/2001285331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563d5f053b80 0x563d5f05d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/2001285331 >> 172.21.15.16:0/2001285331 conn(0x563d5efc21c0 msgr2=0x563d5f056d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/2001285331 shutdown_connections 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 1 -- 172.21.15.16:0/2001285331 wait complete. 2024-04-05T23:44:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.317+0000 7fc8e07cdb80 20 client.26774 trim_cache size 0 max 16384 2024-04-05T23:44:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:44:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:44:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:44:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc8ac0a1310 msgr2=0x7fc8ac0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc8ac0a1310 0x7fc8ac0a37d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009fd0 tx=0x7fc8c0007ca0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3698138436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c8106160 msgr2=0x7fc8c81978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c8106160 0x7fc8c81978d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400e970 tx=0x7fc8b400ec80 comp rx=0 tx=0).stop 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3698138436 shutdown_connections 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fc8ac0a1310 0x7fc8ac0a37d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8c819bc20 0x7fc8c819c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8c8101040 0x7fc8c8197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 --2- 172.21.15.16:0/3698138436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8c8106160 0x7fc8c81978d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3698138436 >> 172.21.15.16:0/3698138436 conn(0x7fc8c80fc8e0 msgr2=0x7fc8c80fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3698138436 shutdown_connections 2024-04-05T23:44:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:44:58.318+0000 7fc8ccc00700 1 -- 172.21.15.16:0/3698138436 wait complete. 2024-04-05T23:44:58.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:44:58 smithi016 ceph-mon[38248]: pgmap v357: 97 pgs: 97 active+clean; 227 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-04-05T23:44:58.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:44:58 smithi130 ceph-mon[79417]: pgmap v357: 97 pgs: 97 active+clean; 227 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-04-05T23:44:58.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:44:58 smithi027 ceph-mon[78597]: pgmap v357: 97 pgs: 97 active+clean; 227 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-04-05T23:44:59.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 554973 4.74 MB/sec execute 498 sec latency 28.484 ms 2024-04-05T23:45:00.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555852 4.74 MB/sec execute 499 sec latency 464.230 ms 2024-04-05T23:45:00.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:00 smithi016 ceph-mon[38248]: pgmap v358: 97 pgs: 97 active+clean; 227 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-04-05T23:45:00.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:00 smithi130 ceph-mon[79417]: pgmap v358: 97 pgs: 97 active+clean; 227 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-04-05T23:45:00.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:00 smithi027 ceph-mon[78597]: pgmap v358: 97 pgs: 97 active+clean; 227 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-04-05T23:45:01.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555852 4.73 MB/sec execute 500 sec latency 1464.368 ms 2024-04-05T23:45:02.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 555852 4.72 MB/sec execute 501 sec latency 2464.560 ms 2024-04-05T23:45:02.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:02 smithi016 ceph-mon[38248]: pgmap v359: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-04-05T23:45:02.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:02 smithi130 ceph-mon[79417]: pgmap v359: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-04-05T23:45:02.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:02 smithi027 ceph-mon[78597]: pgmap v359: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-04-05T23:45:03.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 556021 4.71 MB/sec execute 502 sec latency 2664.685 ms 2024-04-05T23:45:04.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 556021 4.70 MB/sec execute 503 sec latency 1706.044 ms 2024-04-05T23:45:04.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:04 smithi016 ceph-mon[38248]: pgmap v360: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-04-05T23:45:04.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:04 smithi130 ceph-mon[79417]: pgmap v360: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-04-05T23:45:04.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:04 smithi027 ceph-mon[78597]: pgmap v360: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-04-05T23:45:05.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 556021 4.69 MB/sec execute 504 sec latency 2706.203 ms 2024-04-05T23:45:06.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 557796 4.71 MB/sec execute 505 sec latency 2800.506 ms 2024-04-05T23:45:06.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:06 smithi016 ceph-mon[38248]: pgmap v361: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-04-05T23:45:06.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:06 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:06.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:06 smithi130 ceph-mon[79417]: pgmap v361: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-04-05T23:45:06.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:06 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:06.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:06 smithi027 ceph-mon[78597]: pgmap v361: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-04-05T23:45:06.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:06 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:07.032 INFO:tasks.workunit.client.0.smithi016.stdout: 1 558841 4.70 MB/sec execute 506 sec latency 14.897 ms 2024-04-05T23:45:08.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 559397 4.70 MB/sec execute 507 sec latency 622.428 ms 2024-04-05T23:45:08.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:08 smithi016 ceph-mon[38248]: pgmap v362: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-05T23:45:08.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:08 smithi130 ceph-mon[79417]: pgmap v362: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-05T23:45:08.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:08 smithi027 ceph-mon[78597]: pgmap v362: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-05T23:45:09.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 559397 4.69 MB/sec execute 508 sec latency 1622.570 ms 2024-04-05T23:45:09.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:09 smithi016 ceph-mon[38248]: osdmap e96: 12 total, 12 up, 12 in 2024-04-05T23:45:09.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:09 smithi130 ceph-mon[79417]: osdmap e96: 12 total, 12 up, 12 in 2024-04-05T23:45:09.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:09 smithi027 ceph-mon[78597]: osdmap e96: 12 total, 12 up, 12 in 2024-04-05T23:45:10.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 559397 4.68 MB/sec execute 509 sec latency 2622.707 ms 2024-04-05T23:45:10.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:10 smithi016 ceph-mon[38248]: pgmap v364: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.0 MiB/s wr, 125 op/s 2024-04-05T23:45:10.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:10 smithi130 ceph-mon[79417]: pgmap v364: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.0 MiB/s wr, 125 op/s 2024-04-05T23:45:10.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:10 smithi027 ceph-mon[78597]: pgmap v364: 97 pgs: 97 active+clean; 224 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.0 MiB/s wr, 125 op/s 2024-04-05T23:45:11.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 559397 4.67 MB/sec execute 510 sec latency 3622.789 ms 2024-04-05T23:45:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.482+0000 7fa51d400700 1 -- 172.21.15.16:0/244323629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5181019d0 msgr2=0x7fa518105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.482+0000 7fa51d400700 1 --2- 172.21.15.16:0/244323629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5181019d0 0x7fa518105a10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa5040098f0 tx=0x7fa504009c00 comp rx=0 tx=0).stop 2024-04-05T23:45:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.482+0000 7fa51d400700 1 -- 172.21.15.16:0/244323629 shutdown_connections 2024-04-05T23:45:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.482+0000 7fa51d400700 1 --2- 172.21.15.16:0/244323629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518106140 0x7fa51810dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.482+0000 7fa51d400700 1 --2- 172.21.15.16:0/244323629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa518101020 0x7fa518101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.482+0000 7fa51d400700 1 --2- 172.21.15.16:0/244323629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5181019d0 0x7fa518105a10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.483+0000 7fa51d400700 1 -- 172.21.15.16:0/244323629 >> 172.21.15.16:0/244323629 conn(0x7fa5180fc880 msgr2=0x7fa5180feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.483+0000 7fa51d400700 1 -- 172.21.15.16:0/244323629 shutdown_connections 2024-04-05T23:45:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.483+0000 7fa51d400700 1 -- 172.21.15.16:0/244323629 wait complete. 2024-04-05T23:45:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.484+0000 7fa51d400700 1 Processor -- start 2024-04-05T23:45:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.484+0000 7fa51d400700 1 -- start start 2024-04-05T23:45:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.484+0000 7fa51d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518101020 0x7fa518197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.484+0000 7fa51d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518106140 0x7fa518197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa51d400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa51819bbf0 0x7fa51819c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa51d400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51819c670 con 0x7fa518101020 2024-04-05T23:45:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa51d400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51819c7e0 con 0x7fa518106140 2024-04-05T23:45:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa51d400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51819c9a0 con 0x7fa51819bbf0 2024-04-05T23:45:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa516800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518101020 0x7fa518197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa516800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518101020 0x7fa518197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42102/0 (socket says 172.21.15.16:42102) 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa515e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518106140 0x7fa518197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa516800700 1 -- 172.21.15.16:0/2843436967 learned_addr learned my addr 172.21.15.16:0/2843436967 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa517200700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa51819bbf0 0x7fa51819c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa516800700 1 -- 172.21.15.16:0/2843436967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa51819bbf0 msgr2=0x7fa51819c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa516800700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa51819bbf0 0x7fa51819c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa516800700 1 -- 172.21.15.16:0/2843436967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518106140 msgr2=0x7fa518197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa516800700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518106140 0x7fa518197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa516800700 1 -- 172.21.15.16:0/2843436967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa510009d00 con 0x7fa518101020 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.485+0000 7fa515e00700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518106140 0x7fa518197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.486+0000 7fa516800700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518101020 0x7fa518197310 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa50800d950 tx=0x7fa50800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.486+0000 7fa50f400700 1 -- 172.21.15.16:0/2843436967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50800f890 con 0x7fa518101020 2024-04-05T23:45:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.486+0000 7fa50f400700 1 -- 172.21.15.16:0/2843436967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa508010460 con 0x7fa518101020 2024-04-05T23:45:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.486+0000 7fa50f400700 1 -- 172.21.15.16:0/2843436967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50800e610 con 0x7fa518101020 2024-04-05T23:45:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.486+0000 7fa51d400700 1 -- 172.21.15.16:0/2843436967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa504009520 con 0x7fa518101020 2024-04-05T23:45:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.486+0000 7fa51d400700 1 -- 172.21.15.16:0/2843436967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa518073470 con 0x7fa518101020 2024-04-05T23:45:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.489+0000 7fa50f400700 1 -- 172.21.15.16:0/2843436967 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fa5080108d0 con 0x7fa518101020 2024-04-05T23:45:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.490+0000 7fa50f400700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa4fc0a1290 0x7fa4fc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.491+0000 7fa50f400700 1 -- 172.21.15.16:0/2843436967 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fa5080ccb60 con 0x7fa518101020 2024-04-05T23:45:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.491+0000 7fa530fd6b80 1 Processor -- start 2024-04-05T23:45:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.491+0000 7fa515e00700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa4fc0a1290 0x7fa4fc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.491+0000 7fa530fd6b80 1 -- start start 2024-04-05T23:45:11.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.491+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.491+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016eb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.491+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01651b80 con 0x55ac016f14a0 2024-04-05T23:45:11.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01642930 con 0x55ac016f0b80 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac016402f0 con 0x55ac016eb5c0 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa515e00700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa4fc0a1290 0x7fa4fc0a3750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa504005e90 tx=0x7fa50401d040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa515e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa516800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa515e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016eb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42110/0 (socket says 172.21.15.16:42110) 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa515e00700 1 -- 172.21.15.16:0/1185783949 learned_addr learned my addr 172.21.15.16:0/1185783949 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa516800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:47786/0 (socket says 172.21.15.16:47786) 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa517200700 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa516800700 1 -- 172.21.15.16:0/1185783949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 msgr2=0x55ac016eba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa516800700 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa516800700 1 -- 172.21.15.16:0/1185783949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 msgr2=0x55ac016eb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa516800700 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016eb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa516800700 1 -- 172.21.15.16:0/1185783949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac014bd350 con 0x55ac016f0b80 2024-04-05T23:45:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa515e00700 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:45:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa517200700 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:45:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.492+0000 7fa516800700 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f0f60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa5080ca200 tx=0x7fa50809a2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d89a97ae3851929d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.493+0000 7fa503400700 1 -- 172.21.15.16:0/1185783949 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5080a2040 con 0x55ac016f0b80 2024-04-05T23:45:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.493+0000 7fa503400700 1 -- 172.21.15.16:0/1185783949 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5080962d0 con 0x55ac016f0b80 2024-04-05T23:45:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.493+0000 7fa503400700 1 -- 172.21.15.16:0/1185783949 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508099380 con 0x55ac016f0b80 2024-04-05T23:45:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.493+0000 7fa530fd6b80 1 -- 172.21.15.16:0/1185783949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 msgr2=0x55ac016f0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.493+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f0f60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa5080ca200 tx=0x7fa50809a2c0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.494+0000 7fa530fd6b80 1 -- 172.21.15.16:0/1185783949 shutdown_connections 2024-04-05T23:45:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.494+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.494+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f0f60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.494+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/1185783949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.494+0000 7fa530fd6b80 1 -- 172.21.15.16:0/1185783949 >> 172.21.15.16:0/1185783949 conn(0x55ac0165f1c0 msgr2=0x55ac016e5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.494+0000 7fa530fd6b80 1 -- 172.21.15.16:0/1185783949 shutdown_connections 2024-04-05T23:45:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.494+0000 7fa530fd6b80 1 -- 172.21.15.16:0/1185783949 wait complete. 2024-04-05T23:45:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.494+0000 7fa530fd6b80 1 -- start start 2024-04-05T23:45:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.495+0000 7fa530fd6b80 1 Processor -- start 2024-04-05T23:45:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.495+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac016f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.495+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.495+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f14a0 0x55ac016f5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.495+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac0163bbe0 con 0x55ac016eb5c0 2024-04-05T23:45:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.495+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac015d7d20 con 0x55ac016f0b80 2024-04-05T23:45:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.495+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac015d79c0 con 0x55ac016f14a0 2024-04-05T23:45:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa530fd6b80 10 client.0 _ll_register_callbacks cb 0x55ac01443340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:45:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa516800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac016f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa516800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac016f48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42122/0 (socket says 172.21.15.16:42122) 2024-04-05T23:45:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa516800700 1 -- 172.21.15.16:0/701439466 learned_addr learned my addr 172.21.15.16:0/701439466 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa515e00700 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa517200700 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f14a0 0x55ac016f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa516800700 1 -- 172.21.15.16:0/701439466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f14a0 msgr2=0x55ac016f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa516800700 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f14a0 0x55ac016f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa516800700 1 -- 172.21.15.16:0/701439466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 msgr2=0x55ac016f4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa516800700 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa516800700 1 -- 172.21.15.16:0/701439466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51810e530 con 0x55ac016eb5c0 2024-04-05T23:45:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa515e00700 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:45:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa516800700 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac016f48f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa508099b50 tx=0x7fa50809a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd97c9dcfa304724 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa4f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:45:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.496+0000 7fa4f7e00700 1 -- 172.21.15.16:0/701439466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5080a2040 con 0x55ac016eb5c0 2024-04-05T23:45:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa4f7e00700 1 -- 172.21.15.16:0/701439466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fa5080992c0 con 0x55ac016eb5c0 2024-04-05T23:45:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa530fd6b80 1 -- 172.21.15.16:0/701439466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ac016de1e0 con 0x55ac016eb5c0 2024-04-05T23:45:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa4f7e00700 1 -- 172.21.15.16:0/701439466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa50809abd0 con 0x55ac016eb5c0 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa530fd6b80 10 client.16737 fetch_fsmap learned FSMap version 72 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa530fd6b80 1 -- 172.21.15.16:0/701439466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) v3 -- 0x7fa518073470 con 0x55ac016eb5c0 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa4f7e00700 1 -- 172.21.15.16:0/701439466 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ac016de1e0 con 0x55ac016eb5c0 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa4f7e00700 1 -- 172.21.15.16:0/701439466 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 72) v1 ==== 7514+0+0 (secure 0 0 0) 0x7fa508098bb0 con 0x55ac016eb5c0 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa530fd6b80 10 client.16737 fetch_fsmap finished waiting for FSMap version 72 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa530fd6b80 10 client.16737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa530fd6b80 20 client.16737 populate_metadata read hostname 'smithi016' 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ac01704ad0 0x55ac01706ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa530fd6b80 4 client.16737 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.497+0000 7fa530fd6b80 1 -- 172.21.15.16:0/701439466 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ac016346a0 con 0x55ac01704ad0 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.498+0000 7fa515e00700 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ac01704ad0 0x55ac01706ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.499+0000 7fa515e00700 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ac01704ad0 0x55ac01706ef0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b64f458543803149 server_cookie=174340c323951c87 in_seq=0 out_seq=0 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.499+0000 7fa4f7e00700 10 client.16737 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.499+0000 7fa4f7e00700 1 -- 172.21.15.16:0/701439466 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ac016346a0 con 0x55ac01704ad0 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.499+0000 7fa4f7e00700 10 client.16737 handle_command_reply: tid=0 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.499+0000 7fa530fd6b80 1 client.16737 shutdown 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 -- 172.21.15.16:0/701439466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 msgr2=0x55ac016f48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac016f48f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa508099b50 tx=0x7fa50809a100 comp rx=0 tx=0).stop 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 -- 172.21.15.16:0/701439466 shutdown_connections 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ac01704ad0 0x55ac01706ef0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac016f48f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f14a0 0x55ac016f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa4f7e00700 0 client.16737 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/701439466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 -- 172.21.15.16:0/701439466 >> 172.21.15.16:0/701439466 conn(0x55ac0165f1c0 msgr2=0x55ac016e5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 -- 172.21.15.16:0/701439466 shutdown_connections 2024-04-05T23:45:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 1 -- 172.21.15.16:0/701439466 wait complete. 2024-04-05T23:45:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.500+0000 7fa530fd6b80 20 client.16737 trim_cache size 0 max 16384 2024-04-05T23:45:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.515+0000 7fa530fd6b80 1 Processor -- start 2024-04-05T23:45:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.515+0000 7fa530fd6b80 1 -- start start 2024-04-05T23:45:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.515+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.515+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016fbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.515+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016f00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.515+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac014bf640 con 0x55ac016f14a0 2024-04-05T23:45:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.515+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac016f11b0 con 0x55ac016f0b80 2024-04-05T23:45:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.515+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac016fc440 con 0x55ac016eb5c0 2024-04-05T23:45:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa517200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016f00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa516800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa517200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016f00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42138/0 (socket says 172.21.15.16:42138) 2024-04-05T23:45:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa516800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42928/0 (socket says 172.21.15.16:42928) 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa517200700 1 -- 172.21.15.16:0/3107736819 learned_addr learned my addr 172.21.15.16:0/3107736819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa515e00700 1 --2- 172.21.15.16:0/3107736819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016fbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa516800700 1 -- 172.21.15.16:0/3107736819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 msgr2=0x55ac016fbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa516800700 1 --2- 172.21.15.16:0/3107736819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016fbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa516800700 1 -- 172.21.15.16:0/3107736819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 msgr2=0x55ac016f00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa516800700 1 --2- 172.21.15.16:0/3107736819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016f00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa516800700 1 -- 172.21.15.16:0/3107736819 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac016f54e0 con 0x55ac016eb5c0 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa516800700 1 --2- 172.21.15.16:0/3107736819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eb030 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa508098fc0 tx=0x7fa50800dfb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=995662a98777aa53 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.516+0000 7fa503e00700 1 -- 172.21.15.16:0/3107736819 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5080adae0 con 0x55ac016eb5c0 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa503e00700 1 -- 172.21.15.16:0/3107736819 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa50809a2c0 con 0x55ac016eb5c0 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa503e00700 1 -- 172.21.15.16:0/3107736819 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5080a1e10 con 0x55ac016eb5c0 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa530fd6b80 1 -- 172.21.15.16:0/3107736819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 msgr2=0x55ac016eb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/3107736819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eb030 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa508098fc0 tx=0x7fa50800dfb0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa530fd6b80 1 -- 172.21.15.16:0/3107736819 shutdown_connections 2024-04-05T23:45:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/3107736819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016f14a0 0x55ac016f00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/3107736819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016eb5c0 0x55ac016eb030 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/3107736819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f0b80 0x55ac016fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa530fd6b80 1 -- 172.21.15.16:0/3107736819 >> 172.21.15.16:0/3107736819 conn(0x55ac0165f1c0 msgr2=0x55ac016ef9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa530fd6b80 1 -- 172.21.15.16:0/3107736819 shutdown_connections 2024-04-05T23:45:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.517+0000 7fa530fd6b80 1 -- 172.21.15.16:0/3107736819 wait complete. 2024-04-05T23:45:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa530fd6b80 1 -- start start 2024-04-05T23:45:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa530fd6b80 1 Processor -- start 2024-04-05T23:45:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac0174ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f0b80 0x55ac0174cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa530fd6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f14a0 0x55ac01757910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac0174d6d0 con 0x55ac016eb5c0 2024-04-05T23:45:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01757e50 con 0x55ac016f14a0 2024-04-05T23:45:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa530fd6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01757fc0 con 0x55ac016f0b80 2024-04-05T23:45:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa530fd6b80 10 client.0 _ll_register_callbacks cb 0x55ac01443340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:45:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa515e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f0b80 0x55ac0174cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.518+0000 7fa516800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac0174ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa515e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f0b80 0x55ac0174cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:42936/0 (socket says 172.21.15.16:42936) 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa515e00700 1 -- 172.21.15.16:0/276220960 learned_addr learned my addr 172.21.15.16:0/276220960 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa517200700 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f14a0 0x55ac01757910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa515e00700 1 -- 172.21.15.16:0/276220960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f14a0 msgr2=0x55ac01757910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa515e00700 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f14a0 0x55ac01757910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa515e00700 1 -- 172.21.15.16:0/276220960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 msgr2=0x55ac0174ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa515e00700 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac0174ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa515e00700 1 -- 172.21.15.16:0/276220960 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa518073470 con 0x55ac016f0b80 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa515e00700 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f0b80 0x55ac0174cfc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa504009fd0 tx=0x7fa504023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8faf67c74ad17519 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa4f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa4f7e00700 1 -- 172.21.15.16:0/276220960 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504039040 con 0x55ac016f0b80 2024-04-05T23:45:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa530fd6b80 1 -- 172.21.15.16:0/276220960 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5080962d0 con 0x55ac016f0b80 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.519+0000 7fa4f7e00700 1 -- 172.21.15.16:0/276220960 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fa5040052b0 con 0x55ac016f0b80 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa4f7e00700 1 -- 172.21.15.16:0/276220960 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa504005470 con 0x55ac016f0b80 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa530fd6b80 10 client.26815 fetch_fsmap learned FSMap version 72 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa530fd6b80 1 -- 172.21.15.16:0/276220960 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) v3 -- 0x55ac01469b00 con 0x55ac016f0b80 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa4f7e00700 1 -- 172.21.15.16:0/276220960 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5080962d0 con 0x55ac016f0b80 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa4f7e00700 1 -- 172.21.15.16:0/276220960 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 72) v1 ==== 7514+0+0 (secure 0 0 0) 0x7fa504003880 con 0x55ac016f0b80 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa530fd6b80 10 client.26815 fetch_fsmap finished waiting for FSMap version 72 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa530fd6b80 10 client.26815 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa530fd6b80 20 client.26815 populate_metadata read hostname 'smithi016' 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ac01759880 0x55ac0175bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa530fd6b80 4 client.26815 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.520+0000 7fa530fd6b80 1 -- 172.21.15.16:0/276220960 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ac0148ac90 con 0x55ac01759880 2024-04-05T23:45:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.521+0000 7fa516800700 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ac01759880 0x55ac0175bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:11.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.521+0000 7fa516800700 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ac01759880 0x55ac0175bca0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=392584dfb7d7c6ab server_cookie=535684755ccc084 in_seq=0 out_seq=0 2024-04-05T23:45:11.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.521+0000 7fa4f7e00700 10 client.26815 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.567+0000 7fa4f7e00700 1 -- 172.21.15.16:0/276220960 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ac0148ac90 con 0x55ac01759880 2024-04-05T23:45:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.567+0000 7fa4f7e00700 10 client.26815 handle_command_reply: tid=0 2024-04-05T23:45:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.567+0000 7fa530fd6b80 1 client.26815 shutdown 2024-04-05T23:45:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 -- 172.21.15.16:0/276220960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f0b80 msgr2=0x55ac0174cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f0b80 0x55ac0174cfc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa504009fd0 tx=0x7fa504023000 comp rx=0 tx=0).stop 2024-04-05T23:45:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 -- 172.21.15.16:0/276220960 shutdown_connections 2024-04-05T23:45:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55ac01759880 0x55ac0175bca0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ac016eb5c0 0x55ac0174ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ac016f14a0 0x55ac01757910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa4f7e00700 0 client.26815 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 --2- 172.21.15.16:0/276220960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ac016f0b80 0x55ac0174cfc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 -- 172.21.15.16:0/276220960 >> 172.21.15.16:0/276220960 conn(0x55ac0165f1c0 msgr2=0x55ac016f1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 -- 172.21.15.16:0/276220960 shutdown_connections 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 1 -- 172.21.15.16:0/276220960 wait complete. 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.568+0000 7fa530fd6b80 20 client.26815 trim_cache size 0 max 16384 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:45:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:45:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 -- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa4fc0a1290 msgr2=0x7fa4fc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa4fc0a1290 0x7fa4fc0a3750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa504005e90 tx=0x7fa50401d040 comp rx=0 tx=0).stop 2024-04-05T23:45:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 -- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518101020 msgr2=0x7fa518197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518101020 0x7fa518197310 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa50800d950 tx=0x7fa50800dc60 comp rx=0 tx=0).stop 2024-04-05T23:45:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 -- 172.21.15.16:0/2843436967 shutdown_connections 2024-04-05T23:45:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fa4fc0a1290 0x7fa4fc0a3750 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa518101020 0x7fa518197310 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa51819bbf0 0x7fa51819c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 --2- 172.21.15.16:0/2843436967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518106140 0x7fa518197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 -- 172.21.15.16:0/2843436967 >> 172.21.15.16:0/2843436967 conn(0x7fa5180fc880 msgr2=0x7fa5180fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 -- 172.21.15.16:0/2843436967 shutdown_connections 2024-04-05T23:45:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:11.569+0000 7fa51d400700 1 -- 172.21.15.16:0/2843436967 wait complete. 2024-04-05T23:45:12.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 559397 4.66 MB/sec execute 511 sec latency 4622.876 ms 2024-04-05T23:45:12.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:12 smithi016 ceph-mon[38248]: pgmap v365: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-05T23:45:12.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:12 smithi130 ceph-mon[79417]: pgmap v365: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-05T23:45:12.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:12 smithi027 ceph-mon[78597]: pgmap v365: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-05T23:45:13.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 561053 4.67 MB/sec execute 512 sec latency 4810.026 ms 2024-04-05T23:45:13.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:13 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:13.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:13 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:13.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:13 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:14.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 562210 4.67 MB/sec execute 513 sec latency 11.652 ms 2024-04-05T23:45:14.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:14 smithi016 ceph-mon[38248]: pgmap v366: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-05T23:45:14.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:14 smithi130 ceph-mon[79417]: pgmap v366: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-05T23:45:14.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:14 smithi027 ceph-mon[78597]: pgmap v366: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-05T23:45:15.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 562946 4.67 MB/sec execute 514 sec latency 555.250 ms 2024-04-05T23:45:15.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:15 smithi016 ceph-mon[38248]: osdmap e97: 12 total, 12 up, 12 in 2024-04-05T23:45:15.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:15 smithi130 ceph-mon[79417]: osdmap e97: 12 total, 12 up, 12 in 2024-04-05T23:45:15.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:15 smithi027 ceph-mon[78597]: osdmap e97: 12 total, 12 up, 12 in 2024-04-05T23:45:16.033 INFO:tasks.workunit.client.0.smithi016.stdout: 1 562946 4.66 MB/sec execute 515 sec latency 1555.327 ms 2024-04-05T23:45:16.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:16 smithi016 ceph-mon[38248]: pgmap v368: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-04-05T23:45:16.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:16 smithi130 ceph-mon[79417]: pgmap v368: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-04-05T23:45:16.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:16 smithi027 ceph-mon[78597]: pgmap v368: 97 pgs: 97 active+clean; 226 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-04-05T23:45:17.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 562946 4.66 MB/sec execute 516 sec latency 2555.452 ms 2024-04-05T23:45:18.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 564521 4.66 MB/sec execute 517 sec latency 2745.773 ms 2024-04-05T23:45:18.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:18 smithi130 ceph-mon[79417]: pgmap v369: 97 pgs: 97 active+clean; 225 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-05T23:45:18.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:18 smithi027 ceph-mon[78597]: pgmap v369: 97 pgs: 97 active+clean; 225 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-05T23:45:19.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 565697 4.67 MB/sec execute 518 sec latency 19.443 ms 2024-04-05T23:45:19.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:18 smithi016 ceph-mon[38248]: pgmap v369: 97 pgs: 97 active+clean; 225 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-05T23:45:20.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 566486 4.67 MB/sec execute 519 sec latency 459.308 ms 2024-04-05T23:45:20.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:20 smithi027 ceph-mon[78597]: pgmap v370: 97 pgs: 97 active+clean; 225 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-04-05T23:45:21.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 568404 4.68 MB/sec execute 520 sec latency 529.148 ms 2024-04-05T23:45:21.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:20 smithi016 ceph-mon[38248]: pgmap v370: 97 pgs: 97 active+clean; 225 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-04-05T23:45:21.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:20 smithi130 ceph-mon[79417]: pgmap v370: 97 pgs: 97 active+clean; 225 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-04-05T23:45:22.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 569324 4.68 MB/sec execute 521 sec latency 42.120 ms 2024-04-05T23:45:22.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:22 smithi027 ceph-mon[78597]: pgmap v371: 97 pgs: 97 active+clean; 230 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-05T23:45:23.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 570036 4.68 MB/sec execute 522 sec latency 554.183 ms 2024-04-05T23:45:23.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:22 smithi016 ceph-mon[38248]: pgmap v371: 97 pgs: 97 active+clean; 230 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-05T23:45:23.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:22 smithi130 ceph-mon[79417]: pgmap v371: 97 pgs: 97 active+clean; 230 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-05T23:45:24.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 570036 4.67 MB/sec execute 523 sec latency 1554.230 ms 2024-04-05T23:45:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.729+0000 7efc86200700 1 -- 172.21.15.16:0/2203679252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc80073310 msgr2=0x7efc800736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.729+0000 7efc86200700 1 --2- 172.21.15.16:0/2203679252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc80073310 0x7efc800736f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efc70009990 tx=0x7efc70009ca0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.729+0000 7efc86200700 1 -- 172.21.15.16:0/2203679252 shutdown_connections 2024-04-05T23:45:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.729+0000 7efc86200700 1 --2- 172.21.15.16:0/2203679252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc8010d600 0x7efc8010fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.729+0000 7efc86200700 1 --2- 172.21.15.16:0/2203679252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc80073310 0x7efc800736f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.729+0000 7efc86200700 1 --2- 172.21.15.16:0/2203679252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc80073c30 0x7efc8010d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.729+0000 7efc86200700 1 -- 172.21.15.16:0/2203679252 >> 172.21.15.16:0/2203679252 conn(0x7efc800fc8b0 msgr2=0x7efc800fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.729+0000 7efc86200700 1 -- 172.21.15.16:0/2203679252 shutdown_connections 2024-04-05T23:45:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.730+0000 7efc86200700 1 -- 172.21.15.16:0/2203679252 wait complete. 2024-04-05T23:45:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.730+0000 7efc86200700 1 Processor -- start 2024-04-05T23:45:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc86200700 1 -- start start 2024-04-05T23:45:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc86200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc80073310 0x7efc801994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc86200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc80073c30 0x7efc80199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc86200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc8010d600 0x7efc8019ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc86200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8019e400 con 0x7efc80073310 2024-04-05T23:45:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc86200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8019e570 con 0x7efc8010d600 2024-04-05T23:45:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc86200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8019e730 con 0x7efc80073c30 2024-04-05T23:45:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc7f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc80073310 0x7efc801994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc7ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc80073c30 0x7efc80199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc8010d600 0x7efc8019ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc7ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc80073c30 0x7efc80199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:36864/0 (socket says 172.21.15.16:36864) 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.731+0000 7efc7ea00700 1 -- 172.21.15.16:0/2051177288 learned_addr learned my addr 172.21.15.16:0/2051177288 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.732+0000 7efc7ea00700 1 -- 172.21.15.16:0/2051177288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc8010d600 msgr2=0x7efc8019ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.732+0000 7efc7ea00700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc8010d600 0x7efc8019ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.732+0000 7efc7ea00700 1 -- 172.21.15.16:0/2051177288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc80073310 msgr2=0x7efc801994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.732+0000 7efc7ea00700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc80073310 0x7efc801994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.732+0000 7efc7ea00700 1 -- 172.21.15.16:0/2051177288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc700095f0 con 0x7efc80073c30 2024-04-05T23:45:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.732+0000 7efc7fe00700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc8010d600 0x7efc8019ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:45:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.732+0000 7efc7ea00700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc80073c30 0x7efc80199a20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00d980 tx=0x7efc6c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.732+0000 7efc77e00700 1 -- 172.21.15.16:0/2051177288 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c004220 con 0x7efc80073c30 2024-04-05T23:45:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.733+0000 7efc86200700 1 -- 172.21.15.16:0/2051177288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8019ea10 con 0x7efc80073c30 2024-04-05T23:45:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.733+0000 7efc77e00700 1 -- 172.21.15.16:0/2051177288 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc6c007640 con 0x7efc80073c30 2024-04-05T23:45:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.733+0000 7efc77e00700 1 -- 172.21.15.16:0/2051177288 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c005690 con 0x7efc80073c30 2024-04-05T23:45:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.733+0000 7efc86200700 1 -- 172.21.15.16:0/2051177288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc801a6250 con 0x7efc80073c30 2024-04-05T23:45:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.736+0000 7efc77e00700 1 -- 172.21.15.16:0/2051177288 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7efc6c004750 con 0x7efc80073c30 2024-04-05T23:45:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.737+0000 7efc99cbcb80 1 Processor -- start 2024-04-05T23:45:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.737+0000 7efc99cbcb80 1 -- start start 2024-04-05T23:45:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.737+0000 7efc77e00700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7efc640a1290 0x7efc640a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 0x55c4bc201f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc77e00700 1 -- 172.21.15.16:0/2051177288 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8556+0+0 (secure 0 0 0) 0x7efc6c0ccb00 con 0x7efc80073c30 2024-04-05T23:45:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc1fc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc1fca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc162b80 con 0x55c4bc2024a0 2024-04-05T23:45:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc153930 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc1512f0 con 0x55c4bc201b80 2024-04-05T23:45:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7fe00700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7efc640a1290 0x7efc640a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc1fc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc1fc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36118/0 (socket says 172.21.15.16:36118) 2024-04-05T23:45:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7ea00700 1 -- 172.21.15.16:0/2831112247 learned_addr learned my addr 172.21.15.16:0/2831112247 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7f400700 1 --2- 172.21.15.16:0/2831112247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 0x55c4bc201f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7ea00700 1 -- 172.21.15.16:0/2831112247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 msgr2=0x55c4bc201f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7ea00700 1 --2- 172.21.15.16:0/2831112247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 0x55c4bc201f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7ea00700 1 -- 172.21.15.16:0/2831112247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc1fc5c0 msgr2=0x55c4bc1fca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7ea00700 1 --2- 172.21.15.16:0/2831112247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc1fca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.738+0000 7efc7ea00700 1 -- 172.21.15.16:0/2831112247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4bbfce350 con 0x55c4bc2024a0 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.739+0000 7efc7ea00700 1 --2- 172.21.15.16:0/2831112247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc1fc030 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efc6c096600 tx=0x7efc6c09a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206b6e5f58892903 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.739+0000 7efc7fe00700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7efc640a1290 0x7efc640a3750 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efc780075e0 tx=0x7efc780079a0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.739+0000 7efc6be00700 1 -- 172.21.15.16:0/2831112247 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c09d9c0 con 0x55c4bc2024a0 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.739+0000 7efc6be00700 1 -- 172.21.15.16:0/2831112247 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc6c0a2010 con 0x55c4bc2024a0 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.739+0000 7efc6be00700 1 -- 172.21.15.16:0/2831112247 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c099080 con 0x55c4bc2024a0 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.740+0000 7efc99cbcb80 1 -- 172.21.15.16:0/2831112247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 msgr2=0x55c4bc1fc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.740+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/2831112247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc1fc030 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efc6c096600 tx=0x7efc6c09a360 comp rx=0 tx=0).stop 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.740+0000 7efc99cbcb80 1 -- 172.21.15.16:0/2831112247 shutdown_connections 2024-04-05T23:45:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.740+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/2831112247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc1fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.740+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/2831112247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 0x55c4bc201f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.740+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/2831112247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc1fc030 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.740+0000 7efc99cbcb80 1 -- 172.21.15.16:0/2831112247 >> 172.21.15.16:0/2831112247 conn(0x55c4bc1701c0 msgr2=0x55c4bc1f6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.740+0000 7efc99cbcb80 1 -- 172.21.15.16:0/2831112247 shutdown_connections 2024-04-05T23:45:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.740+0000 7efc99cbcb80 1 -- 172.21.15.16:0/2831112247 wait complete. 2024-04-05T23:45:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.741+0000 7efc99cbcb80 1 -- start start 2024-04-05T23:45:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.741+0000 7efc99cbcb80 1 Processor -- start 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc201b80 0x55c4bc205870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 0x55c4bc205e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc14cbe0 con 0x55c4bc201b80 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc0e8d20 con 0x55c4bc2024a0 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc0e89c0 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc99cbcb80 10 client.0 _ll_register_callbacks cb 0x55c4bbf54340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc7ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc201b80 0x55c4bc205870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc7ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc201b80 0x55c4bc205870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36130/0 (socket says 172.21.15.16:36130) 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc7f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc7ea00700 1 -- 172.21.15.16:0/4014131830 learned_addr learned my addr 172.21.15.16:0/4014131830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.742+0000 7efc7fe00700 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 0x55c4bc205e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc7ea00700 1 -- 172.21.15.16:0/4014131830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 msgr2=0x55c4bc208480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc7ea00700 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc7ea00700 1 -- 172.21.15.16:0/4014131830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 msgr2=0x55c4bc205e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc7ea00700 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 0x55c4bc205e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc7ea00700 1 -- 172.21.15.16:0/4014131830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc801a6250 con 0x55c4bc201b80 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc7f400700 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc7ea00700 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc201b80 0x55c4bc205870 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0a1ae0 tx=0x7efc6c0045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835b141f93a241e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc68c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc68c00700 1 -- 172.21.15.16:0/4014131830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c097fb0 con 0x55c4bc201b80 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc68c00700 1 -- 172.21.15.16:0/4014131830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8556+0+0 (secure 0 0 0) 0x7efc6c099290 con 0x55c4bc201b80 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc99cbcb80 1 -- 172.21.15.16:0/4014131830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4bc1ef1e0 con 0x55c4bc201b80 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.743+0000 7efc68c00700 1 -- 172.21.15.16:0/4014131830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc6c099df0 con 0x55c4bc201b80 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc99cbcb80 10 client.16755 fetch_fsmap learned FSMap version 73 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc99cbcb80 1 -- 172.21.15.16:0/4014131830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) v3 -- 0x55c4bbf7ab00 con 0x55c4bc201b80 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc68c00700 1 -- 172.21.15.16:0/4014131830 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc6c07e020 con 0x55c4bc201b80 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc68c00700 1 -- 172.21.15.16:0/4014131830 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) v1 ==== 7510+0+0 (secure 0 0 0) 0x7efc6c09ac30 con 0x55c4bc201b80 2024-04-05T23:45:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc99cbcb80 10 client.16755 fetch_fsmap finished waiting for FSMap version 73 2024-04-05T23:45:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc99cbcb80 10 client.16755 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:45:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc99cbcb80 20 client.16755 populate_metadata read hostname 'smithi016' 2024-04-05T23:45:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c4bc215a10 0x55c4bc217e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc99cbcb80 4 client.16755 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:45:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc99cbcb80 1 -- 172.21.15.16:0/4014131830 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4bc1456a0 con 0x55c4bc215a10 2024-04-05T23:45:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.744+0000 7efc7f400700 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c4bc215a10 0x55c4bc217e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.745+0000 7efc7f400700 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c4bc215a10 0x55c4bc217e30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fea493ee5349a5c7 server_cookie=ab61762f69a93537 in_seq=0 out_seq=0 2024-04-05T23:45:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.745+0000 7efc68c00700 10 client.16755 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc68c00700 1 -- 172.21.15.16:0/4014131830 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c4bc1456a0 con 0x55c4bc215a10 2024-04-05T23:45:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc68c00700 10 client.16755 handle_command_reply: tid=0 2024-04-05T23:45:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 client.16755 shutdown 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 -- 172.21.15.16:0/4014131830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc201b80 msgr2=0x55c4bc205870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc201b80 0x55c4bc205870 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0a1ae0 tx=0x7efc6c0045b0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 -- 172.21.15.16:0/4014131830 shutdown_connections 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c4bc215a10 0x55c4bc217e30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 0x55c4bc205e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc68c00700 0 client.16755 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/4014131830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc201b80 0x55c4bc205870 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 -- 172.21.15.16:0/4014131830 >> 172.21.15.16:0/4014131830 conn(0x55c4bc1701c0 msgr2=0x55c4bc1f6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 -- 172.21.15.16:0/4014131830 shutdown_connections 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 1 -- 172.21.15.16:0/4014131830 wait complete. 2024-04-05T23:45:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.746+0000 7efc99cbcb80 20 client.16755 trim_cache size 0 max 16384 2024-04-05T23:45:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.761+0000 7efc99cbcb80 1 Processor -- start 2024-04-05T23:45:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc99cbcb80 1 -- start start 2024-04-05T23:45:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc211df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc201b80 0x55c4bc1fc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc205130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bbfd0640 con 0x55c4bc2024a0 2024-04-05T23:45:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc206480 con 0x55c4bc201b80 2024-04-05T23:45:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc205670 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc7ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc201b80 0x55c4bc1fc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc7fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc205130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc7ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc201b80 0x55c4bc1fc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:55024/0 (socket says 172.21.15.16:55024) 2024-04-05T23:45:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc7ea00700 1 -- 172.21.15.16:0/1493887917 learned_addr learned my addr 172.21.15.16:0/1493887917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.762+0000 7efc7f400700 1 --2- 172.21.15.16:0/1493887917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc211df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.763+0000 7efc7ea00700 1 -- 172.21.15.16:0/1493887917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 msgr2=0x55c4bc211df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.763+0000 7efc7ea00700 1 --2- 172.21.15.16:0/1493887917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc211df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.763+0000 7efc7ea00700 1 -- 172.21.15.16:0/1493887917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 msgr2=0x55c4bc205130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.763+0000 7efc7ea00700 1 --2- 172.21.15.16:0/1493887917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc205130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.763+0000 7efc7ea00700 1 -- 172.21.15.16:0/1493887917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4bc112e40 con 0x55c4bc201b80 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.763+0000 7efc7ea00700 1 --2- 172.21.15.16:0/1493887917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc201b80 0x55c4bc1fc030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00d950 tx=0x7efc6c09a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=668e727209bf1c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.763+0000 7efc75200700 1 -- 172.21.15.16:0/1493887917 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0996d0 con 0x55c4bc201b80 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.763+0000 7efc75200700 1 -- 172.21.15.16:0/1493887917 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc6c07e040 con 0x55c4bc201b80 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.763+0000 7efc75200700 1 -- 172.21.15.16:0/1493887917 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c09aba0 con 0x55c4bc201b80 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1493887917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc201b80 msgr2=0x55c4bc1fc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1493887917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc201b80 0x55c4bc1fc030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00d950 tx=0x7efc6c09a360 comp rx=0 tx=0).stop 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1493887917 shutdown_connections 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1493887917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc2024a0 0x55c4bc205130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1493887917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc211df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1493887917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc201b80 0x55c4bc1fc030 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1493887917 >> 172.21.15.16:0/1493887917 conn(0x55c4bc1701c0 msgr2=0x55c4bc208270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1493887917 shutdown_connections 2024-04-05T23:45:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1493887917 wait complete. 2024-04-05T23:45:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.764+0000 7efc99cbcb80 1 -- start start 2024-04-05T23:45:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.765+0000 7efc99cbcb80 1 Processor -- start 2024-04-05T23:45:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.765+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.765+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 0x55c4bc209320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.765+0000 7efc99cbcb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 0x55c4bc209860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.765+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc2687f0 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.765+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc268930 con 0x55c4bc2024a0 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.765+0000 7efc99cbcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bc268ac0 con 0x55c4bc201b80 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.765+0000 7efc99cbcb80 10 client.0 _ll_register_callbacks cb 0x7efc70009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36146/0 (socket says 172.21.15.16:36146) 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7f400700 1 -- 172.21.15.16:0/1200045648 learned_addr learned my addr 172.21.15.16:0/1200045648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7ea00700 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 0x55c4bc209320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7fe00700 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 0x55c4bc209860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7f400700 1 -- 172.21.15.16:0/1200045648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 msgr2=0x55c4bc209320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7f400700 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 0x55c4bc209320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7f400700 1 -- 172.21.15.16:0/1200045648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 msgr2=0x55c4bc209860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7f400700 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 0x55c4bc209860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7f400700 1 -- 172.21.15.16:0/1200045648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4bbf7ab00 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc7f400700 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208de0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efc700098e0 tx=0x7efc7000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d2e2b450fbafdd server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.766+0000 7efc68c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc68c00700 1 -- 172.21.15.16:0/1200045648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc700280b0 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc68c00700 1 -- 172.21.15.16:0/1200045648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8556+0+0 (secure 0 0 0) 0x7efc70028210 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1200045648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efc6c0a2010 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc68c00700 1 -- 172.21.15.16:0/1200045648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc700283d0 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc99cbcb80 10 client.16761 fetch_fsmap learned FSMap version 73 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1200045648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) v3 -- 0x55c4bbfd2c70 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc68c00700 1 -- 172.21.15.16:0/1200045648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc6c0a2010 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc68c00700 1 -- 172.21.15.16:0/1200045648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) v1 ==== 7510+0+0 (secure 0 0 0) 0x7efc70003c50 con 0x55c4bc1fc5c0 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc99cbcb80 10 client.16761 fetch_fsmap finished waiting for FSMap version 73 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc99cbcb80 10 client.16761 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc99cbcb80 20 client.16761 populate_metadata read hostname 'smithi016' 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.767+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c4bc26a3f0 0x55c4bc26c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.768+0000 7efc99cbcb80 4 client.16761 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.768+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1200045648 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c4bbf9bc90 con 0x55c4bc26a3f0 2024-04-05T23:45:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.768+0000 7efc7ea00700 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c4bc26a3f0 0x55c4bc26c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.768+0000 7efc68c00700 10 client.16761 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.768+0000 7efc7ea00700 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c4bc26a3f0 0x55c4bc26c810 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e315fd75e514b01 server_cookie=1157511331ba7bee in_seq=0 out_seq=0 2024-04-05T23:45:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.804+0000 7efc68c00700 1 -- 172.21.15.16:0/1200045648 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c4bbf9bc90 con 0x55c4bc26a3f0 2024-04-05T23:45:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.804+0000 7efc68c00700 10 client.16761 handle_command_reply: tid=0 2024-04-05T23:45:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 client.16761 shutdown 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1200045648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc1fc5c0 msgr2=0x55c4bc208de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208de0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efc700098e0 tx=0x7efc7000be80 comp rx=0 tx=0).stop 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1200045648 shutdown_connections 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55c4bc26a3f0 0x55c4bc26c810 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c4bc1fc5c0 0x55c4bc208de0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc68c00700 0 client.16761 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c4bc2024a0 0x55c4bc209860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 --2- 172.21.15.16:0/1200045648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4bc201b80 0x55c4bc209320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1200045648 >> 172.21.15.16:0/1200045648 conn(0x55c4bc1701c0 msgr2=0x55c4bc1fc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1200045648 shutdown_connections 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 1 -- 172.21.15.16:0/1200045648 wait complete. 2024-04-05T23:45:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.805+0000 7efc99cbcb80 20 client.16761 trim_cache size 0 max 16384 2024-04-05T23:45:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:45:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:45:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:45:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 -- 172.21.15.16:0/2051177288 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7efc640a1290 msgr2=0x7efc640a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7efc640a1290 0x7efc640a3750 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efc780075e0 tx=0x7efc780079a0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 -- 172.21.15.16:0/2051177288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc80073c30 msgr2=0x7efc80199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc80073c30 0x7efc80199a20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00d980 tx=0x7efc6c00dc90 comp rx=0 tx=0).stop 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 -- 172.21.15.16:0/2051177288 shutdown_connections 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7efc640a1290 0x7efc640a3750 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc80073310 0x7efc801994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc8010d600 0x7efc8019ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 --2- 172.21.15.16:0/2051177288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc80073c30 0x7efc80199a20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 -- 172.21.15.16:0/2051177288 >> 172.21.15.16:0/2051177288 conn(0x7efc800fc8b0 msgr2=0x7efc800fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 -- 172.21.15.16:0/2051177288 shutdown_connections 2024-04-05T23:45:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:24.806+0000 7efc86200700 1 -- 172.21.15.16:0/2051177288 wait complete. 2024-04-05T23:45:24.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:24 smithi027 ceph-mon[78597]: pgmap v372: 97 pgs: 97 active+clean; 230 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-05T23:45:25.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 570036 4.66 MB/sec execute 524 sec latency 2554.279 ms 2024-04-05T23:45:25.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:24 smithi016 ceph-mon[38248]: pgmap v372: 97 pgs: 97 active+clean; 230 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-05T23:45:25.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:24 smithi130 ceph-mon[79417]: pgmap v372: 97 pgs: 97 active+clean; 230 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-04-05T23:45:26.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 570323 4.65 MB/sec execute 525 sec latency 3418.815 ms 2024-04-05T23:45:26.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:26 smithi027 ceph-mon[78597]: pgmap v373: 97 pgs: 97 active+clean; 230 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-04-05T23:45:27.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 572113 4.67 MB/sec execute 526 sec latency 16.296 ms 2024-04-05T23:45:27.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:26 smithi016 ceph-mon[38248]: pgmap v373: 97 pgs: 97 active+clean; 230 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-04-05T23:45:27.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:26 smithi130 ceph-mon[79417]: pgmap v373: 97 pgs: 97 active+clean; 230 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-04-05T23:45:27.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:27 smithi027 ceph-mon[78597]: pgmap v374: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-04-05T23:45:28.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 573312 4.67 MB/sec execute 527 sec latency 16.301 ms 2024-04-05T23:45:28.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:27 smithi016 ceph-mon[38248]: pgmap v374: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-04-05T23:45:28.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:27 smithi130 ceph-mon[79417]: pgmap v374: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-04-05T23:45:29.034 INFO:tasks.workunit.client.0.smithi016.stdout: 1 573579 4.66 MB/sec execute 528 sec latency 797.170 ms 2024-04-05T23:45:30.035 INFO:tasks.workunit.client.0.smithi016.stdout: 1 573579 4.65 MB/sec execute 529 sec latency 1797.324 ms 2024-04-05T23:45:30.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:29 smithi027 ceph-mon[78597]: pgmap v375: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-05T23:45:30.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:29 smithi016 ceph-mon[38248]: pgmap v375: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-05T23:45:30.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:29 smithi130 ceph-mon[79417]: pgmap v375: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-05T23:45:31.035 INFO:tasks.workunit.client.0.smithi016.stdout: 1 573579 4.64 MB/sec execute 530 sec latency 2797.487 ms 2024-04-05T23:45:32.035 INFO:tasks.workunit.client.0.smithi016.stdout: 1 573579 4.63 MB/sec execute 531 sec latency 3797.626 ms 2024-04-05T23:45:32.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:31 smithi027 ceph-mon[78597]: pgmap v376: 97 pgs: 97 active+clean; 231 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-05T23:45:32.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:31 smithi016 ceph-mon[38248]: pgmap v376: 97 pgs: 97 active+clean; 231 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-05T23:45:32.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:31 smithi130 ceph-mon[79417]: pgmap v376: 97 pgs: 97 active+clean; 231 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-05T23:45:33.035 INFO:tasks.workunit.client.0.smithi016.stdout: 1 575093 4.64 MB/sec execute 532 sec latency 3985.871 ms 2024-04-05T23:45:34.035 INFO:tasks.workunit.client.0.smithi016.stdout: 1 576336 4.64 MB/sec execute 533 sec latency 13.467 ms 2024-04-05T23:45:34.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:33 smithi027 ceph-mon[78597]: pgmap v377: 97 pgs: 97 active+clean; 231 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-05T23:45:34.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:33 smithi016 ceph-mon[38248]: pgmap v377: 97 pgs: 97 active+clean; 231 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-05T23:45:34.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:33 smithi130 ceph-mon[79417]: pgmap v377: 97 pgs: 97 active+clean; 231 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-05T23:45:35.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 577126 4.64 MB/sec execute 534 sec latency 484.793 ms 2024-04-05T23:45:36.035 INFO:tasks.workunit.client.0.smithi016.stdout: 1 577126 4.63 MB/sec execute 535 sec latency 1484.930 ms 2024-04-05T23:45:36.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:35 smithi027 ceph-mon[78597]: pgmap v378: 97 pgs: 97 active+clean; 231 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-05T23:45:36.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:35 smithi016 ceph-mon[38248]: pgmap v378: 97 pgs: 97 active+clean; 231 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-05T23:45:36.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:35 smithi130 ceph-mon[79417]: pgmap v378: 97 pgs: 97 active+clean; 231 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-05T23:45:37.035 INFO:tasks.workunit.client.0.smithi016.stdout: 1 577126 4.62 MB/sec execute 536 sec latency 2485.056 ms 2024-04-05T23:45:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.966+0000 7fdffca00700 1 -- 172.21.15.16:0/2657790180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff8073310 msgr2=0x7fdff80736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.966+0000 7fdffca00700 1 --2- 172.21.15.16:0/2657790180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff8073310 0x7fdff80736f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009990 tx=0x7fdfe8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.966+0000 7fdffca00700 1 -- 172.21.15.16:0/2657790180 shutdown_connections 2024-04-05T23:45:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.966+0000 7fdffca00700 1 --2- 172.21.15.16:0/2657790180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff8073310 0x7fdff80736f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.966+0000 7fdffca00700 1 --2- 172.21.15.16:0/2657790180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdff810d610 0x7fdff810fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.966+0000 7fdffca00700 1 --2- 172.21.15.16:0/2657790180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdff8073c30 0x7fdff810d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.966+0000 7fdffca00700 1 -- 172.21.15.16:0/2657790180 >> 172.21.15.16:0/2657790180 conn(0x7fdff80fc8e0 msgr2=0x7fdff80fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.966+0000 7fdffca00700 1 -- 172.21.15.16:0/2657790180 shutdown_connections 2024-04-05T23:45:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.966+0000 7fdffca00700 1 -- 172.21.15.16:0/2657790180 wait complete. 2024-04-05T23:45:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.967+0000 7fdffca00700 1 Processor -- start 2024-04-05T23:45:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.967+0000 7fdffca00700 1 -- start start 2024-04-05T23:45:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdffca00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff8073310 0x7fdff81994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdffca00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdff8073c30 0x7fdff81999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdffca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdff810d610 0x7fdff819dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdffca00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff819e3d0 con 0x7fdff8073310 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdffca00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff819e540 con 0x7fdff810d610 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdffca00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff819e700 con 0x7fdff8073c30 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdff8073c30 0x7fdff81999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff8073310 0x7fdff81994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdff810d610 0x7fdff819dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdff8073c30 0x7fdff81999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59116/0 (socket says 172.21.15.16:59116) 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff5400700 1 -- 172.21.15.16:0/1332241035 learned_addr learned my addr 172.21.15.16:0/1332241035 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff5400700 1 -- 172.21.15.16:0/1332241035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdff810d610 msgr2=0x7fdff819dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff5400700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdff810d610 0x7fdff819dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff5400700 1 -- 172.21.15.16:0/1332241035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff8073310 msgr2=0x7fdff81994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff5400700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff8073310 0x7fdff81994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff5400700 1 -- 172.21.15.16:0/1332241035 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe80095f0 con 0x7fdff8073c30 2024-04-05T23:45:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.968+0000 7fdff6800700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdff810d610 0x7fdff819dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:45:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.969+0000 7fdff5400700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdff8073c30 0x7fdff81999f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400e9a0 tx=0x7fdfe400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.969+0000 7fdfeea00700 1 -- 172.21.15.16:0/1332241035 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4009790 con 0x7fdff8073c30 2024-04-05T23:45:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.969+0000 7fdfeea00700 1 -- 172.21.15.16:0/1332241035 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfe4010430 con 0x7fdff8073c30 2024-04-05T23:45:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.969+0000 7fdffca00700 1 -- 172.21.15.16:0/1332241035 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff819e9e0 con 0x7fdff8073c30 2024-04-05T23:45:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.969+0000 7fdfeea00700 1 -- 172.21.15.16:0/1332241035 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4018680 con 0x7fdff8073c30 2024-04-05T23:45:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.969+0000 7fdffca00700 1 -- 172.21.15.16:0/1332241035 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff81a6220 con 0x7fdff8073c30 2024-04-05T23:45:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.974+0000 7fdfeea00700 1 -- 172.21.15.16:0/1332241035 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fdfe40105a0 con 0x7fdff8073c30 2024-04-05T23:45:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.975+0000 7fe010530b80 1 Processor -- start 2024-04-05T23:45:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.975+0000 7fe010530b80 1 -- start start 2024-04-05T23:45:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.975+0000 7fe010530b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.975+0000 7fe010530b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.975+0000 7fe010530b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.975+0000 7fe010530b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c6717b80 con 0x5637c67b74a0 2024-04-05T23:45:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.975+0000 7fe010530b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c6708930 con 0x5637c67b15c0 2024-04-05T23:45:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.975+0000 7fe010530b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c67062f0 con 0x5637c67b6b80 2024-04-05T23:45:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.975+0000 7fdfeea00700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdfdc0a1290 0x7fdfdc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff6800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdfeea00700 1 -- 172.21.15.16:0/1332241035 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdfe4014070 con 0x7fdff8073c30 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42036/0 (socket says 172.21.15.16:42036) 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff6800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59128/0 (socket says 172.21.15.16:59128) 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5400700 1 -- 172.21.15.16:0/3112442972 learned_addr learned my addr 172.21.15.16:0/3112442972 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5e00700 1 --2- 172.21.15.16:0/3112442972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5400700 1 -- 172.21.15.16:0/3112442972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 msgr2=0x5637c67b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5400700 1 --2- 172.21.15.16:0/3112442972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff6800700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdfdc0a1290 0x7fdfdc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5400700 1 -- 172.21.15.16:0/3112442972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 msgr2=0x5637c67b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5400700 1 --2- 172.21.15.16:0/3112442972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67b1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5400700 1 -- 172.21.15.16:0/3112442972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637c6583350 con 0x5637c67b15c0 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5e00700 1 --2- 172.21.15.16:0/3112442972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdff5400700 1 --2- 172.21.15.16:0/3112442972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67b1a20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4000c00 tx=0x7fdfe4099ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cfff9a57ebe8bcc server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.976+0000 7fdfe3000700 1 -- 172.21.15.16:0/3112442972 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4025170 con 0x5637c67b15c0 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fdfe3000700 1 -- 172.21.15.16:0/3112442972 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfe40ade90 con 0x5637c67b15c0 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fdfe3000700 1 -- 172.21.15.16:0/3112442972 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe40ca880 con 0x5637c67b15c0 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fdff6800700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdfdc0a1290 0x7fdfdc0a3750 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdff00076a0 tx=0x7fdff000b040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fe010530b80 1 -- 172.21.15.16:0/3112442972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 msgr2=0x5637c67b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fe010530b80 1 --2- 172.21.15.16:0/3112442972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67b1a20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4000c00 tx=0x7fdfe4099ba0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fe010530b80 1 -- 172.21.15.16:0/3112442972 shutdown_connections 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fe010530b80 1 --2- 172.21.15.16:0/3112442972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67b1a20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fe010530b80 1 --2- 172.21.15.16:0/3112442972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fe010530b80 1 --2- 172.21.15.16:0/3112442972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fe010530b80 1 -- 172.21.15.16:0/3112442972 >> 172.21.15.16:0/3112442972 conn(0x5637c67251c0 msgr2=0x5637c67abbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.977+0000 7fe010530b80 1 -- 172.21.15.16:0/3112442972 shutdown_connections 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.978+0000 7fe010530b80 1 -- 172.21.15.16:0/3112442972 wait complete. 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.978+0000 7fe010530b80 1 -- start start 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.978+0000 7fe010530b80 1 Processor -- start 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fe010530b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67bd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fe010530b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67ba7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fe010530b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67bae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fe010530b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c6701be0 con 0x5637c67b74a0 2024-04-05T23:45:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fe010530b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c669dd20 con 0x5637c67b15c0 2024-04-05T23:45:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fe010530b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c669d9c0 con 0x5637c67b6b80 2024-04-05T23:45:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fe010530b80 10 client.0 _ll_register_callbacks cb 0x5637c6509340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:45:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67bd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67ba7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67bd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42038/0 (socket says 172.21.15.16:42038) 2024-04-05T23:45:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff5e00700 1 -- 172.21.15.16:0/2444040078 learned_addr learned my addr 172.21.15.16:0/2444040078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff6800700 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67bae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff5400700 1 -- 172.21.15.16:0/2444040078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 msgr2=0x5637c67bd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff5400700 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67bd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff5400700 1 -- 172.21.15.16:0/2444040078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 msgr2=0x5637c67bae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff5400700 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67bae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff5400700 1 -- 172.21.15.16:0/2444040078 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff81a6220 con 0x5637c67b6b80 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.979+0000 7fdff6800700 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67bae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fdff5400700 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67ba7c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdfe409a180 tx=0x7fdfe400c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71e54a81416b984 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fdfd7400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fdfd7400700 1 -- 172.21.15.16:0/2444040078 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe409ddf0 con 0x5637c67b6b80 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fdfd7400700 1 -- 172.21.15.16:0/2444040078 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdfe40a2440 con 0x5637c67b6b80 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fe010530b80 1 -- 172.21.15.16:0/2444040078 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5637c67a41e0 con 0x5637c67b6b80 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fdfd7400700 1 -- 172.21.15.16:0/2444040078 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfe409a430 con 0x5637c67b6b80 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fe010530b80 10 client.26845 fetch_fsmap learned FSMap version 73 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fe010530b80 1 -- 172.21.15.16:0/2444040078 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) v3 -- 0x5637c652fb00 con 0x5637c67b6b80 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fdfd7400700 1 -- 172.21.15.16:0/2444040078 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5637c67a41e0 con 0x5637c67b6b80 2024-04-05T23:45:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.980+0000 7fdfd7400700 1 -- 172.21.15.16:0/2444040078 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 73) v1 ==== 7510+0+0 (secure 0 0 0) 0x7fdfe4094bd0 con 0x5637c67b6b80 2024-04-05T23:45:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.981+0000 7fe010530b80 10 client.26845 fetch_fsmap finished waiting for FSMap version 73 2024-04-05T23:45:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.981+0000 7fe010530b80 10 client.26845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:45:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.981+0000 7fe010530b80 20 client.26845 populate_metadata read hostname 'smithi016' 2024-04-05T23:45:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.981+0000 7fe010530b80 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5637c67cad40 0x5637c67cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.981+0000 7fe010530b80 4 client.26845 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:45:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.981+0000 7fe010530b80 1 -- 172.21.15.16:0/2444040078 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5637c66fa6a0 con 0x5637c67cad40 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.981+0000 7fdff5e00700 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5637c67cad40 0x5637c67cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.981+0000 7fdff5e00700 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5637c67cad40 0x5637c67cd160 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3bec7189509dc25 server_cookie=af87b291ad9cd3cc in_seq=0 out_seq=0 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.981+0000 7fdfd7400700 10 client.26845 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fdfd7400700 1 -- 172.21.15.16:0/2444040078 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5637c66fa6a0 con 0x5637c67cad40 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fdfd7400700 10 client.26845 handle_command_reply: tid=0 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 client.26845 shutdown 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 -- 172.21.15.16:0/2444040078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 msgr2=0x5637c67ba7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67ba7c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdfe409a180 tx=0x7fdfe400c400 comp rx=0 tx=0).stop 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 -- 172.21.15.16:0/2444040078 shutdown_connections 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5637c67cad40 0x5637c67cd160 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b74a0 0x5637c67bae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b15c0 0x5637c67bd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 --2- 172.21.15.16:0/2444040078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67ba7c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fdfd7400700 0 client.26845 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 -- 172.21.15.16:0/2444040078 >> 172.21.15.16:0/2444040078 conn(0x5637c67251c0 msgr2=0x5637c67ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 -- 172.21.15.16:0/2444040078 shutdown_connections 2024-04-05T23:45:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.982+0000 7fe010530b80 1 -- 172.21.15.16:0/2444040078 wait complete. 2024-04-05T23:45:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.983+0000 7fe010530b80 20 client.26845 trim_cache size 0 max 16384 2024-04-05T23:45:37.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fe010530b80 1 Processor -- start 2024-04-05T23:45:37.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fe010530b80 1 -- start start 2024-04-05T23:45:37.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fe010530b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fe010530b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67c1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fe010530b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fe010530b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c6585640 con 0x5637c67b15c0 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fe010530b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c67bbc50 con 0x5637c67b74a0 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fe010530b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c67bb8d0 con 0x5637c67b6b80 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fdff6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:37.999+0000 7fdff5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67c1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42048/0 (socket says 172.21.15.16:42048) 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff6800700 1 -- 172.21.15.16:0/2851746990 learned_addr learned my addr 172.21.15.16:0/2851746990 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff6800700 1 -- 172.21.15.16:0/2851746990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 msgr2=0x5637c67c1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff6800700 1 --2- 172.21.15.16:0/2851746990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67c1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff6800700 1 -- 172.21.15.16:0/2851746990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 msgr2=0x5637c67b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff6800700 1 --2- 172.21.15.16:0/2851746990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff6800700 1 -- 172.21.15.16:0/2851746990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637c66c7e40 con 0x5637c67b74a0 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff5e00700 1 --2- 172.21.15.16:0/2851746990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:45:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdff6800700 1 --2- 172.21.15.16:0/2851746990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67c6d60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdff000e8e0 tx=0x7fdff0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1285009069bc6feb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.000+0000 7fdfe3a00700 1 -- 172.21.15.16:0/2851746990 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff001d040 con 0x5637c67b74a0 2024-04-05T23:45:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fdfe3a00700 1 -- 172.21.15.16:0/2851746990 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdff000f040 con 0x5637c67b74a0 2024-04-05T23:45:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fdfe3a00700 1 -- 172.21.15.16:0/2851746990 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff0007ea0 con 0x5637c67b74a0 2024-04-05T23:45:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fe010530b80 1 -- 172.21.15.16:0/2851746990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 msgr2=0x5637c67c6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fe010530b80 1 --2- 172.21.15.16:0/2851746990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67c6d60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdff000e8e0 tx=0x7fdff0017000 comp rx=0 tx=0).stop 2024-04-05T23:45:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fe010530b80 1 -- 172.21.15.16:0/2851746990 shutdown_connections 2024-04-05T23:45:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fe010530b80 1 --2- 172.21.15.16:0/2851746990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fe010530b80 1 --2- 172.21.15.16:0/2851746990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67c6d60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fe010530b80 1 --2- 172.21.15.16:0/2851746990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67c1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fe010530b80 1 -- 172.21.15.16:0/2851746990 >> 172.21.15.16:0/2851746990 conn(0x5637c67251c0 msgr2=0x5637c67c6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fe010530b80 1 -- 172.21.15.16:0/2851746990 shutdown_connections 2024-04-05T23:45:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.001+0000 7fe010530b80 1 -- 172.21.15.16:0/2851746990 wait complete. 2024-04-05T23:45:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.002+0000 7fe010530b80 1 -- start start 2024-04-05T23:45:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.002+0000 7fe010530b80 1 Processor -- start 2024-04-05T23:45:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.002+0000 7fe010530b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.002+0000 7fe010530b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.002+0000 7fe010530b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.002+0000 7fe010530b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c67b2c90 con 0x5637c67b15c0 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.002+0000 7fe010530b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c67b2e00 con 0x5637c67b74a0 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.002+0000 7fe010530b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637c67b2fc0 con 0x5637c67b6b80 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fe010530b80 10 client.0 _ll_register_callbacks cb 0x7fdff0009040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59170/0 (socket says 172.21.15.16:59170) 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36540/0 (socket says 172.21.15.16:36540) 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5400700 1 -- 172.21.15.16:0/1137522133 learned_addr learned my addr 172.21.15.16:0/1137522133 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5400700 1 -- 172.21.15.16:0/1137522133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 msgr2=0x5637c67b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5400700 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5400700 1 -- 172.21.15.16:0/1137522133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 msgr2=0x5637c67b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5400700 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5400700 1 -- 172.21.15.16:0/1137522133 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff8110740 con 0x5637c67b6b80 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5e00700 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff6800700 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdff5400700 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b3ec0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4010c30 tx=0x7fdfe40ba420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2aaf04c5e640432 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.003+0000 7fdfd7400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fdfd7400700 1 -- 172.21.15.16:0/1137522133 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4099b00 con 0x5637c67b6b80 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fdfd7400700 1 -- 172.21.15.16:0/1137522133 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdfe4063020 con 0x5637c67b6b80 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fe010530b80 1 -- 172.21.15.16:0/1137522133 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdfe40ade90 con 0x5637c67b6b80 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fdfd7400700 1 -- 172.21.15.16:0/1137522133 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfe4099fc0 con 0x5637c67b6b80 2024-04-05T23:45:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fe010530b80 10 client.26851 fetch_fsmap learned FSMap version 73 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fe010530b80 1 -- 172.21.15.16:0/1137522133 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) v3 -- 0x7fdff000e510 con 0x5637c67b6b80 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fdfd7400700 1 -- 172.21.15.16:0/1137522133 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfe40ade90 con 0x5637c67b6b80 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fdfd7400700 1 -- 172.21.15.16:0/1137522133 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 73) v1 ==== 7510+0+0 (secure 0 0 0) 0x7fdfe40a1fb0 con 0x5637c67b6b80 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fe010530b80 10 client.26851 fetch_fsmap finished waiting for FSMap version 73 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fe010530b80 10 client.26851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fe010530b80 20 client.26851 populate_metadata read hostname 'smithi016' 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fe010530b80 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5637c681f850 0x5637c6821c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fe010530b80 4 client.26851 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.004+0000 7fe010530b80 1 -- 172.21.15.16:0/1137522133 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5637c6550c90 con 0x5637c681f850 2024-04-05T23:45:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.005+0000 7fdff5e00700 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5637c681f850 0x5637c6821c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.005+0000 7fdff5e00700 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5637c681f850 0x5637c6821c70 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1560e836c4d79c7b server_cookie=a5b352dea94086f9 in_seq=0 out_seq=0 2024-04-05T23:45:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.005+0000 7fdfd7400700 10 client.26851 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fdfd7400700 1 -- 172.21.15.16:0/1137522133 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5637c6550c90 con 0x5637c681f850 2024-04-05T23:45:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fdfd7400700 10 client.26851 handle_command_reply: tid=0 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 client.26851 shutdown 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 -- 172.21.15.16:0/1137522133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 msgr2=0x5637c67b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b3ec0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4010c30 tx=0x7fdfe40ba420 comp rx=0 tx=0).stop 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 -- 172.21.15.16:0/1137522133 shutdown_connections 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5637c681f850 0x5637c6821c70 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5637c67b15c0 0x5637c67b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5637c67b74a0 0x5637c67b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fdfd7400700 0 client.26851 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 --2- 172.21.15.16:0/1137522133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637c67b6b80 0x5637c67b3ec0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 -- 172.21.15.16:0/1137522133 >> 172.21.15.16:0/1137522133 conn(0x5637c67251c0 msgr2=0x5637c67c6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 -- 172.21.15.16:0/1137522133 shutdown_connections 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.038+0000 7fe010530b80 1 -- 172.21.15.16:0/1137522133 wait complete. 2024-04-05T23:45:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fe010530b80 20 client.26851 trim_cache size 0 max 16384 2024-04-05T23:45:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:45:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:45:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:45:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 -- 172.21.15.16:0/1332241035 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdfdc0a1290 msgr2=0x7fdfdc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdfdc0a1290 0x7fdfdc0a3750 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdff00076a0 tx=0x7fdff000b040 comp rx=0 tx=0).stop 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 -- 172.21.15.16:0/1332241035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdff8073c30 msgr2=0x7fdff81999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdff8073c30 0x7fdff81999f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400e9a0 tx=0x7fdfe400ecb0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 -- 172.21.15.16:0/1332241035 shutdown_connections 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdfdc0a1290 0x7fdfdc0a3750 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff8073310 0x7fdff81994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdff810d610 0x7fdff819dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 --2- 172.21.15.16:0/1332241035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdff8073c30 0x7fdff81999f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.039+0000 7fdffca00700 1 -- 172.21.15.16:0/1332241035 >> 172.21.15.16:0/1332241035 conn(0x7fdff80fc8e0 msgr2=0x7fdff80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.040+0000 7fdffca00700 1 -- 172.21.15.16:0/1332241035 shutdown_connections 2024-04-05T23:45:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:38.040+0000 7fdffca00700 1 -- 172.21.15.16:0/1332241035 wait complete. 2024-04-05T23:45:38.035 INFO:tasks.workunit.client.0.smithi016.stdout: 1 578643 4.63 MB/sec execute 537 sec latency 2674.067 ms 2024-04-05T23:45:38.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:37 smithi027 ceph-mon[78597]: pgmap v379: 97 pgs: 97 active+clean; 234 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-05T23:45:38.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:37 smithi016 ceph-mon[38248]: pgmap v379: 97 pgs: 97 active+clean; 234 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-05T23:45:38.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:37 smithi130 ceph-mon[79417]: pgmap v379: 97 pgs: 97 active+clean; 234 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-05T23:45:39.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 579804 4.63 MB/sec execute 538 sec latency 13.773 ms 2024-04-05T23:45:40.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 580669 4.63 MB/sec execute 539 sec latency 460.287 ms 2024-04-05T23:45:40.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:40 smithi016 ceph-mon[38248]: pgmap v380: 97 pgs: 97 active+clean; 234 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 980 KiB/s wr, 104 op/s 2024-04-05T23:45:40.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:40 smithi130 ceph-mon[79417]: pgmap v380: 97 pgs: 97 active+clean; 234 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 980 KiB/s wr, 104 op/s 2024-04-05T23:45:40.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:40 smithi027 ceph-mon[78597]: pgmap v380: 97 pgs: 97 active+clean; 234 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 980 KiB/s wr, 104 op/s 2024-04-05T23:45:41.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 582511 4.65 MB/sec execute 540 sec latency 527.285 ms 2024-04-05T23:45:42.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 583507 4.64 MB/sec execute 541 sec latency 16.301 ms 2024-04-05T23:45:42.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:42 smithi027 ceph-mon[78597]: pgmap v381: 97 pgs: 97 active+clean; 235 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-05T23:45:42.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:42 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:43.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 584222 4.64 MB/sec execute 542 sec latency 570.381 ms 2024-04-05T23:45:43.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:42 smithi016 ceph-mon[38248]: pgmap v381: 97 pgs: 97 active+clean; 235 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-05T23:45:43.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:42 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:43.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:42 smithi130 ceph-mon[79417]: pgmap v381: 97 pgs: 97 active+clean; 235 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-05T23:45:43.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:42 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:43.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:43 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:44.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 584222 4.64 MB/sec execute 543 sec latency 1570.555 ms 2024-04-05T23:45:44.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:43 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:44.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:43 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:44.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:44 smithi027 ceph-mon[78597]: pgmap v382: 97 pgs: 97 active+clean; 235 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-05T23:45:45.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 584222 4.63 MB/sec execute 544 sec latency 2570.712 ms 2024-04-05T23:45:45.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:44 smithi016 ceph-mon[38248]: pgmap v382: 97 pgs: 97 active+clean; 235 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-05T23:45:45.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:44 smithi130 ceph-mon[79417]: pgmap v382: 97 pgs: 97 active+clean; 235 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-05T23:45:46.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 584509 4.62 MB/sec execute 545 sec latency 3436.632 ms 2024-04-05T23:45:46.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:46 smithi027 ceph-mon[78597]: pgmap v383: 97 pgs: 97 active+clean; 235 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-05T23:45:47.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 586335 4.63 MB/sec execute 546 sec latency 16.976 ms 2024-04-05T23:45:47.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:46 smithi016 ceph-mon[38248]: pgmap v383: 97 pgs: 97 active+clean; 235 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-05T23:45:47.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:46 smithi130 ceph-mon[79417]: pgmap v383: 97 pgs: 97 active+clean; 235 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-05T23:45:47.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:47 smithi027 ceph-mon[78597]: pgmap v384: 97 pgs: 97 active+clean; 238 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.6 MiB/s wr, 162 op/s 2024-04-05T23:45:48.036 INFO:tasks.workunit.client.0.smithi016.stdout: 1 587577 4.64 MB/sec execute 547 sec latency 16.974 ms 2024-04-05T23:45:48.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:47 smithi016 ceph-mon[38248]: pgmap v384: 97 pgs: 97 active+clean; 238 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.6 MiB/s wr, 162 op/s 2024-04-05T23:45:48.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:47 smithi130 ceph-mon[79417]: pgmap v384: 97 pgs: 97 active+clean; 238 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.6 MiB/s wr, 162 op/s 2024-04-05T23:45:49.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 587763 4.63 MB/sec execute 548 sec latency 870.286 ms 2024-04-05T23:45:50.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 587763 4.62 MB/sec execute 549 sec latency 1870.412 ms 2024-04-05T23:45:50.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:49 smithi027 ceph-mon[78597]: pgmap v385: 97 pgs: 97 active+clean; 238 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-05T23:45:50.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:49 smithi016 ceph-mon[38248]: pgmap v385: 97 pgs: 97 active+clean; 238 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-05T23:45:50.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:49 smithi130 ceph-mon[79417]: pgmap v385: 97 pgs: 97 active+clean; 238 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-05T23:45:51.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 587763 4.61 MB/sec execute 550 sec latency 2870.533 ms 2024-04-05T23:45:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.203+0000 7f929de00700 1 -- 172.21.15.16:0/2890360994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 msgr2=0x7f9298107320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.203+0000 7f929de00700 1 --2- 172.21.15.16:0/2890360994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 0x7f9298107320 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9288009990 tx=0x7f9288009ca0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.203+0000 7f929de00700 1 -- 172.21.15.16:0/2890360994 shutdown_connections 2024-04-05T23:45:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.203+0000 7f929de00700 1 --2- 172.21.15.16:0/2890360994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 0x7f9298107320 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.203+0000 7f929de00700 1 --2- 172.21.15.16:0/2890360994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9298068be0 0x7f929810dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.203+0000 7f929de00700 1 --2- 172.21.15.16:0/2890360994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9298068220 0x7f92980686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.203+0000 7f929de00700 1 -- 172.21.15.16:0/2890360994 >> 172.21.15.16:0/2890360994 conn(0x7f9298076e50 msgr2=0x7f9298077260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.204+0000 7f929de00700 1 -- 172.21.15.16:0/2890360994 shutdown_connections 2024-04-05T23:45:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.204+0000 7f929de00700 1 -- 172.21.15.16:0/2890360994 wait complete. 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.205+0000 7f929de00700 1 Processor -- start 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.205+0000 7f929de00700 1 -- start start 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.205+0000 7f929de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9298068220 0x7f92980ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.205+0000 7f929de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9298068be0 0x7f9298100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.205+0000 7f929de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 0x7f92981007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.205+0000 7f929de00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298101d20 con 0x7f9298106f40 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.205+0000 7f929de00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92981a9bb0 con 0x7f9298068220 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.205+0000 7f929de00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92981a9d70 con 0x7f9298068be0 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.205+0000 7f9297400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 0x7f92981007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9297400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 0x7f92981007d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41282/0 (socket says 172.21.15.16:41282) 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9296a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9298068220 0x7f92980ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9297400700 1 -- 172.21.15.16:0/233872171 learned_addr learned my addr 172.21.15.16:0/233872171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9296000700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9298068be0 0x7f9298100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9297400700 1 -- 172.21.15.16:0/233872171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9298068be0 msgr2=0x7f9298100290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9297400700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9298068be0 0x7f9298100290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9297400700 1 -- 172.21.15.16:0/233872171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9298068220 msgr2=0x7f92980ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9297400700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9298068220 0x7f92980ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9297400700 1 -- 172.21.15.16:0/233872171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92880095f0 con 0x7f9298106f40 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f9297400700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 0x7f92981007d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f929000ef30 tx=0x7f929000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f929ca00700 1 -- 172.21.15.16:0/233872171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9290018070 con 0x7f9298106f40 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.206+0000 7f929ca00700 1 -- 172.21.15.16:0/233872171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f929000f040 con 0x7f9298106f40 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.207+0000 7f929de00700 1 -- 172.21.15.16:0/233872171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92981a9ff0 con 0x7f9298106f40 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.207+0000 7f929ca00700 1 -- 172.21.15.16:0/233872171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929000cd00 con 0x7f9298106f40 2024-04-05T23:45:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.207+0000 7f929de00700 1 -- 172.21.15.16:0/233872171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92981aa4d0 con 0x7f9298106f40 2024-04-05T23:45:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.210+0000 7f929ca00700 1 -- 172.21.15.16:0/233872171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f9290003a00 con 0x7f9298106f40 2024-04-05T23:45:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.212+0000 7f92b1934b80 1 Processor -- start 2024-04-05T23:45:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.212+0000 7f929ca00700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f927c0a1290 0x7f927c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.212+0000 7f92b1934b80 1 -- start start 2024-04-05T23:45:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.212+0000 7f929ca00700 1 -- 172.21.15.16:0/233872171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f92900cc820 con 0x7f9298106f40 2024-04-05T23:45:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.212+0000 7f9296000700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f927c0a1290 0x7f927c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f408f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f403030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 0x555a1f403a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f92b1934b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f369b80 con 0x555a1f408b80 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f92b1934b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f35a930 con 0x555a1f4094a0 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f92b1934b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f3582f0 con 0x555a1f4035c0 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f408f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f408f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41298/0 (socket says 172.21.15.16:41298) 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296a00700 1 -- 172.21.15.16:0/1703858148 learned_addr learned my addr 172.21.15.16:0/1703858148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9297400700 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 0x555a1f403a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296000700 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f403030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296000700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f927c0a1290 0x7f927c0a3750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9298101480 tx=0x7f9284009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296a00700 1 -- 172.21.15.16:0/1703858148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 msgr2=0x555a1f403a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296a00700 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 0x555a1f403a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296a00700 1 -- 172.21.15.16:0/1703858148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 msgr2=0x555a1f403030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296a00700 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f403030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296a00700 1 -- 172.21.15.16:0/1703858148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a1f1d5350 con 0x555a1f408b80 2024-04-05T23:45:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.213+0000 7f9296000700 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f403030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:45:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.214+0000 7f9296a00700 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f408f60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9288005ec0 tx=0x7f9288005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc20926e0c5c728d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.214+0000 7f9283e00700 1 -- 172.21.15.16:0/1703858148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928801d070 con 0x555a1f408b80 2024-04-05T23:45:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.214+0000 7f9283e00700 1 -- 172.21.15.16:0/1703858148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f928801e040 con 0x555a1f408b80 2024-04-05T23:45:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.214+0000 7f9283e00700 1 -- 172.21.15.16:0/1703858148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92880047f0 con 0x555a1f408b80 2024-04-05T23:45:51.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 -- 172.21.15.16:0/1703858148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 msgr2=0x555a1f408f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f408f60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9288005ec0 tx=0x7f9288005ef0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 -- 172.21.15.16:0/1703858148 shutdown_connections 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f408f60 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 0x555a1f403a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 --2- 172.21.15.16:0/1703858148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f403030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 -- 172.21.15.16:0/1703858148 >> 172.21.15.16:0/1703858148 conn(0x555a1f3771c0 msgr2=0x555a1f3fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 -- 172.21.15.16:0/1703858148 shutdown_connections 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 -- 172.21.15.16:0/1703858148 wait complete. 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.215+0000 7f92b1934b80 1 -- start start 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f92b1934b80 1 Processor -- start 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4035c0 0x555a1f40f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f40c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 0x555a1f40d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f92b1934b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f353be0 con 0x555a1f408b80 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f92b1934b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f2efd20 con 0x555a1f4035c0 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f92b1934b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f2ef9c0 con 0x555a1f4094a0 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f92b1934b80 10 client.0 _ll_register_callbacks cb 0x555a1f15b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f9296a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4035c0 0x555a1f40f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f9296000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f40c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f9296a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4035c0 0x555a1f40f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:52588/0 (socket says 172.21.15.16:52588) 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f9296a00700 1 -- 172.21.15.16:0/726105866 learned_addr learned my addr 172.21.15.16:0/726105866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.216+0000 7f9297400700 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 0x555a1f40d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f9296a00700 1 -- 172.21.15.16:0/726105866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 msgr2=0x555a1f40d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f9296a00700 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 0x555a1f40d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f9296a00700 1 -- 172.21.15.16:0/726105866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 msgr2=0x555a1f40c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f9296a00700 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f40c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f9296a00700 1 -- 172.21.15.16:0/726105866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929810e570 con 0x555a1f4035c0 2024-04-05T23:45:51.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f9296a00700 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4035c0 0x555a1f40f5a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f928800bed0 tx=0x7f9288005c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e1265ba5b61a0f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f9280c00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f9280c00700 1 -- 172.21.15.16:0/726105866 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928801d070 con 0x555a1f4035c0 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f9280c00700 1 -- 172.21.15.16:0/726105866 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f9288004500 con 0x555a1f4035c0 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.217+0000 7f92b1934b80 1 -- 172.21.15.16:0/726105866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a1f3f61e0 con 0x555a1f4035c0 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f9280c00700 1 -- 172.21.15.16:0/726105866 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92880046c0 con 0x555a1f4035c0 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f92b1934b80 10 client.26837 fetch_fsmap learned FSMap version 75 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f92b1934b80 1 -- 172.21.15.16:0/726105866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) v3 -- 0x555a1f181b00 con 0x555a1f4035c0 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f9280c00700 1 -- 172.21.15.16:0/726105866 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f928803a020 con 0x555a1f4035c0 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f9280c00700 1 -- 172.21.15.16:0/726105866 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 75) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f928800f950 con 0x555a1f4035c0 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f92b1934b80 10 client.26837 fetch_fsmap finished waiting for FSMap version 75 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f92b1934b80 10 client.26837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f92b1934b80 20 client.26837 populate_metadata read hostname 'smithi016' 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f92b1934b80 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555a1f41cb30 0x555a1f41ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f92b1934b80 4 client.26837 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.218+0000 7f92b1934b80 1 -- 172.21.15.16:0/726105866 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a1f34c6a0 con 0x555a1f41cb30 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.219+0000 7f9296000700 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555a1f41cb30 0x555a1f41ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.219+0000 7f9296000700 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555a1f41cb30 0x555a1f41ef50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bf892e24904bb51 server_cookie=9dcf3446a5a7def1 in_seq=0 out_seq=0 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.219+0000 7f9280c00700 10 client.26837 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f9280c00700 1 -- 172.21.15.16:0/726105866 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555a1f34c6a0 con 0x555a1f41cb30 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f9280c00700 10 client.26837 handle_command_reply: tid=0 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 client.26837 shutdown 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 -- 172.21.15.16:0/726105866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4035c0 msgr2=0x555a1f40f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4035c0 0x555a1f40f5a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f928800bed0 tx=0x7f9288005c30 comp rx=0 tx=0).stop 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 -- 172.21.15.16:0/726105866 shutdown_connections 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555a1f41cb30 0x555a1f41ef50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 0x555a1f40d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4035c0 0x555a1f40f5a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 --2- 172.21.15.16:0/726105866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f40c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f9280c00700 0 client.26837 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 -- 172.21.15.16:0/726105866 >> 172.21.15.16:0/726105866 conn(0x555a1f3771c0 msgr2=0x555a1f3fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 -- 172.21.15.16:0/726105866 shutdown_connections 2024-04-05T23:45:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 1 -- 172.21.15.16:0/726105866 wait complete. 2024-04-05T23:45:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.220+0000 7f92b1934b80 20 client.26837 trim_cache size 0 max 16384 2024-04-05T23:45:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f92b1934b80 1 Processor -- start 2024-04-05T23:45:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f92b1934b80 1 -- start start 2024-04-05T23:45:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f4035c0 0x555a1f403030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f408b80 0x555a1f40c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 0x555a1f3fda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f92b1934b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f1d7640 con 0x555a1f4035c0 2024-04-05T23:45:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f92b1934b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f40de90 con 0x555a1f408b80 2024-04-05T23:45:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f92b1934b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f4091e0 con 0x555a1f4094a0 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f9296a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f4035c0 0x555a1f403030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f9296a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f4035c0 0x555a1f403030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41314/0 (socket says 172.21.15.16:41314) 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f9296a00700 1 -- 172.21.15.16:0/1105900566 learned_addr learned my addr 172.21.15.16:0/1105900566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f9296000700 1 --2- 172.21.15.16:0/1105900566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f408b80 0x555a1f40c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.236+0000 7f9297400700 1 --2- 172.21.15.16:0/1105900566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 0x555a1f3fda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.237+0000 7f9296a00700 1 -- 172.21.15.16:0/1105900566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 msgr2=0x555a1f3fda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.237+0000 7f9296a00700 1 --2- 172.21.15.16:0/1105900566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 0x555a1f3fda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.237+0000 7f9296a00700 1 -- 172.21.15.16:0/1105900566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f408b80 msgr2=0x555a1f40c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.237+0000 7f9296a00700 1 --2- 172.21.15.16:0/1105900566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f408b80 0x555a1f40c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.237+0000 7f9296a00700 1 -- 172.21.15.16:0/1105900566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a1f319e40 con 0x555a1f4035c0 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.237+0000 7f9296a00700 1 --2- 172.21.15.16:0/1105900566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f4035c0 0x555a1f403030 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9288009350 tx=0x7f928802b3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930b343dbcfb364c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.237+0000 7f928d200700 1 -- 172.21.15.16:0/1105900566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928801d040 con 0x555a1f4035c0 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.237+0000 7f928d200700 1 -- 172.21.15.16:0/1105900566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f928803a070 con 0x555a1f4035c0 2024-04-05T23:45:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.237+0000 7f928d200700 1 -- 172.21.15.16:0/1105900566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92880049f0 con 0x555a1f4035c0 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 -- 172.21.15.16:0/1105900566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f4035c0 msgr2=0x555a1f403030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 --2- 172.21.15.16:0/1105900566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f4035c0 0x555a1f403030 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9288009350 tx=0x7f928802b3c0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 -- 172.21.15.16:0/1105900566 shutdown_connections 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 --2- 172.21.15.16:0/1105900566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f4035c0 0x555a1f403030 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 --2- 172.21.15.16:0/1105900566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4094a0 0x555a1f3fda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 --2- 172.21.15.16:0/1105900566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f408b80 0x555a1f40c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 -- 172.21.15.16:0/1105900566 >> 172.21.15.16:0/1105900566 conn(0x555a1f3771c0 msgr2=0x555a1f40c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 -- 172.21.15.16:0/1105900566 shutdown_connections 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 -- 172.21.15.16:0/1105900566 wait complete. 2024-04-05T23:45:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.238+0000 7f92b1934b80 1 -- start start 2024-04-05T23:45:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.239+0000 7f92b1934b80 1 Processor -- start 2024-04-05T23:45:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.239+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 0x555a1f411ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.239+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f412420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.239+0000 7f92b1934b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f412980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.239+0000 7f92b1934b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f46fb20 con 0x555a1f408b80 2024-04-05T23:45:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f92b1934b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f46fc90 con 0x555a1f4094a0 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f92b1934b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1f46fe50 con 0x555a1f4035c0 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f92b1934b80 10 client.0 _ll_register_callbacks cb 0x7f9290009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f412420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f412420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41328/0 (socket says 172.21.15.16:41328) 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 0x555a1f411ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296000700 1 -- 172.21.15.16:0/2574537313 learned_addr learned my addr 172.21.15.16:0/2574537313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9297400700 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f412980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296000700 1 -- 172.21.15.16:0/2574537313 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 msgr2=0x555a1f411ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296000700 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 0x555a1f411ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296000700 1 -- 172.21.15.16:0/2574537313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 msgr2=0x555a1f412980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296000700 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f412980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296000700 1 -- 172.21.15.16:0/2574537313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a1f181b00 con 0x555a1f408b80 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9296a00700 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 0x555a1f411ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.240+0000 7f9297400700 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f412980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.241+0000 7f9296000700 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f412420 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x555a1f409900 tx=0x7f92840047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100a1713ccba39e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.241+0000 7f9280c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.241+0000 7f9280c00700 1 -- 172.21.15.16:0/2574537313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9284021030 con 0x555a1f408b80 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.241+0000 7f9280c00700 1 -- 172.21.15.16:0/2574537313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f928402a420 con 0x555a1f408b80 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.241+0000 7f92b1934b80 1 -- 172.21.15.16:0/2574537313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f928801e040 con 0x555a1f408b80 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.241+0000 7f9280c00700 1 -- 172.21.15.16:0/2574537313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9284004830 con 0x555a1f408b80 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.241+0000 7f92b1934b80 10 client.16797 fetch_fsmap learned FSMap version 75 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.241+0000 7f92b1934b80 1 -- 172.21.15.16:0/2574537313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) v3 -- 0x555a1f1d9c70 con 0x555a1f408b80 2024-04-05T23:45:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.241+0000 7f9280c00700 1 -- 172.21.15.16:0/2574537313 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f928403e080 con 0x555a1f408b80 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.242+0000 7f9280c00700 1 -- 172.21.15.16:0/2574537313 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 75) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f9284003680 con 0x555a1f408b80 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.242+0000 7f92b1934b80 10 client.16797 fetch_fsmap finished waiting for FSMap version 75 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.242+0000 7f92b1934b80 10 client.16797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.242+0000 7f92b1934b80 20 client.16797 populate_metadata read hostname 'smithi016' 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.242+0000 7f92b1934b80 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555a1f471680 0x555a1f473aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.242+0000 7f92b1934b80 4 client.16797 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.242+0000 7f92b1934b80 1 -- 172.21.15.16:0/2574537313 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555a1f1a2c90 con 0x555a1f471680 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.242+0000 7f9296a00700 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555a1f471680 0x555a1f473aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.243+0000 7f9296a00700 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555a1f471680 0x555a1f473aa0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aefbed707f55f4b3 server_cookie=6cbee5c11ec1e494 in_seq=0 out_seq=0 2024-04-05T23:45:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.243+0000 7f9280c00700 10 client.16797 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f9280c00700 1 -- 172.21.15.16:0/2574537313 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555a1f1a2c90 con 0x555a1f471680 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f9280c00700 10 client.16797 handle_command_reply: tid=0 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 client.16797 shutdown 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 -- 172.21.15.16:0/2574537313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 msgr2=0x555a1f412420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f412420 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x555a1f409900 tx=0x7f92840047c0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 -- 172.21.15.16:0/2574537313 shutdown_connections 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x555a1f471680 0x555a1f473aa0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555a1f4094a0 0x555a1f412980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555a1f4035c0 0x555a1f411ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 --2- 172.21.15.16:0/2574537313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a1f408b80 0x555a1f412420 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f9280c00700 0 client.16797 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:45:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 -- 172.21.15.16:0/2574537313 >> 172.21.15.16:0/2574537313 conn(0x555a1f3771c0 msgr2=0x555a1f40b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 -- 172.21.15.16:0/2574537313 shutdown_connections 2024-04-05T23:45:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.277+0000 7f92b1934b80 1 -- 172.21.15.16:0/2574537313 wait complete. 2024-04-05T23:45:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f92b1934b80 20 client.16797 trim_cache size 0 max 16384 2024-04-05T23:45:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:45:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:45:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:45:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:45:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 -- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f927c0a1290 msgr2=0x7f927c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f927c0a1290 0x7f927c0a3750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9298101480 tx=0x7f9284009260 comp rx=0 tx=0).stop 2024-04-05T23:45:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 -- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 msgr2=0x7f92981007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:45:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 0x7f92981007d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f929000ef30 tx=0x7f929000c3a0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 -- 172.21.15.16:0/233872171 shutdown_connections 2024-04-05T23:45:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f927c0a1290 0x7f927c0a3750 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9298106f40 0x7f92981007d0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9298068220 0x7f92980ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 --2- 172.21.15.16:0/233872171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9298068be0 0x7f9298100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:45:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 -- 172.21.15.16:0/233872171 >> 172.21.15.16:0/233872171 conn(0x7f9298076e50 msgr2=0x7f92980feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:45:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.278+0000 7f929de00700 1 -- 172.21.15.16:0/233872171 shutdown_connections 2024-04-05T23:45:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:45:51.279+0000 7f929de00700 1 -- 172.21.15.16:0/233872171 wait complete. 2024-04-05T23:45:52.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 587763 4.60 MB/sec execute 551 sec latency 3870.608 ms 2024-04-05T23:45:52.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:51 smithi027 ceph-mon[78597]: pgmap v386: 97 pgs: 97 active+clean; 241 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-05T23:45:52.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:51 smithi016 ceph-mon[38248]: pgmap v386: 97 pgs: 97 active+clean; 241 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-05T23:45:52.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:51 smithi130 ceph-mon[79417]: pgmap v386: 97 pgs: 97 active+clean; 241 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-05T23:45:53.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 589411 4.61 MB/sec execute 552 sec latency 4057.520 ms 2024-04-05T23:45:54.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 590533 4.61 MB/sec execute 553 sec latency 13.845 ms 2024-04-05T23:45:54.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:53 smithi027 ceph-mon[78597]: pgmap v387: 97 pgs: 97 active+clean; 241 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 962 KiB/s wr, 100 op/s 2024-04-05T23:45:54.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:53 smithi016 ceph-mon[38248]: pgmap v387: 97 pgs: 97 active+clean; 241 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 962 KiB/s wr, 100 op/s 2024-04-05T23:45:54.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:53 smithi130 ceph-mon[79417]: pgmap v387: 97 pgs: 97 active+clean; 241 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 962 KiB/s wr, 100 op/s 2024-04-05T23:45:55.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 591312 4.61 MB/sec execute 554 sec latency 498.835 ms 2024-04-05T23:45:55.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:54 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:45:55.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:54 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:45:55.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:54 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:45:56.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 591312 4.60 MB/sec execute 555 sec latency 1498.912 ms 2024-04-05T23:45:56.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:55 smithi027 ceph-mon[78597]: pgmap v388: 97 pgs: 97 active+clean; 241 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 962 KiB/s wr, 100 op/s 2024-04-05T23:45:56.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:55 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:45:56.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:55 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:45:56.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:55 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:45:56.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:55 smithi016 ceph-mon[38248]: pgmap v388: 97 pgs: 97 active+clean; 241 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 962 KiB/s wr, 100 op/s 2024-04-05T23:45:56.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:55 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:45:56.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:55 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:45:56.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:55 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:45:56.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:55 smithi130 ceph-mon[79417]: pgmap v388: 97 pgs: 97 active+clean; 241 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 962 KiB/s wr, 100 op/s 2024-04-05T23:45:56.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:55 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:45:56.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:55 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:45:56.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:55 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:45:57.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 591312 4.60 MB/sec execute 556 sec latency 2499.031 ms 2024-04-05T23:45:58.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 592982 4.60 MB/sec execute 557 sec latency 2687.377 ms 2024-04-05T23:45:58.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:57 smithi027 ceph-mon[78597]: pgmap v389: 97 pgs: 97 active+clean; 240 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-04-05T23:45:58.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:57 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:58.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:57 smithi016 ceph-mon[38248]: pgmap v389: 97 pgs: 97 active+clean; 240 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-04-05T23:45:58.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:57 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:58.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:57 smithi130 ceph-mon[79417]: pgmap v389: 97 pgs: 97 active+clean; 240 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-04-05T23:45:58.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:57 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:45:59.037 INFO:tasks.workunit.client.0.smithi016.stdout: 1 594224 4.61 MB/sec execute 558 sec latency 12.438 ms 2024-04-05T23:46:00.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 594856 4.61 MB/sec execute 559 sec latency 618.149 ms 2024-04-05T23:46:00.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:45:59 smithi027 ceph-mon[78597]: pgmap v390: 97 pgs: 97 active+clean; 240 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.5 MiB/s wr, 125 op/s 2024-04-05T23:46:00.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:45:59 smithi016 ceph-mon[38248]: pgmap v390: 97 pgs: 97 active+clean; 240 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.5 MiB/s wr, 125 op/s 2024-04-05T23:46:00.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:45:59 smithi130 ceph-mon[79417]: pgmap v390: 97 pgs: 97 active+clean; 240 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.5 MiB/s wr, 125 op/s 2024-04-05T23:46:01.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 596416 4.61 MB/sec execute 560 sec latency 686.521 ms 2024-04-05T23:46:02.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 597393 4.61 MB/sec execute 561 sec latency 24.945 ms 2024-04-05T23:46:02.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:01 smithi027 ceph-mon[78597]: pgmap v391: 97 pgs: 97 active+clean; 244 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.2 MiB/s wr, 206 op/s 2024-04-05T23:46:02.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:01 smithi016 ceph-mon[38248]: pgmap v391: 97 pgs: 97 active+clean; 244 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.2 MiB/s wr, 206 op/s 2024-04-05T23:46:02.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:01 smithi130 ceph-mon[79417]: pgmap v391: 97 pgs: 97 active+clean; 244 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.2 MiB/s wr, 206 op/s 2024-04-05T23:46:03.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 598403 4.61 MB/sec execute 562 sec latency 322.326 ms 2024-04-05T23:46:03.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:02 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:03.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:02 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:03.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:02 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:04.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 598403 4.61 MB/sec execute 563 sec latency 1322.429 ms 2024-04-05T23:46:04.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:03 smithi027 ceph-mon[78597]: pgmap v392: 97 pgs: 97 active+clean; 244 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-04-05T23:46:04.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:03 smithi027 ceph-mon[78597]: osdmap e98: 12 total, 12 up, 12 in 2024-04-05T23:46:04.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:03 smithi016 ceph-mon[38248]: pgmap v392: 97 pgs: 97 active+clean; 244 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-04-05T23:46:04.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:03 smithi016 ceph-mon[38248]: osdmap e98: 12 total, 12 up, 12 in 2024-04-05T23:46:04.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:03 smithi130 ceph-mon[79417]: pgmap v392: 97 pgs: 97 active+clean; 244 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-04-05T23:46:04.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:03 smithi130 ceph-mon[79417]: osdmap e98: 12 total, 12 up, 12 in 2024-04-05T23:46:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.438+0000 7f137ba00700 1 -- 172.21.15.16:0/1486644272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374111dd0 msgr2=0x7f13741141c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.438+0000 7f137ba00700 1 --2- 172.21.15.16:0/1486644272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374111dd0 0x7f13741141c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f136c009f90 tx=0x7f136c00b330 comp rx=0 tx=0).stop 2024-04-05T23:46:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.439+0000 7f137ba00700 1 -- 172.21.15.16:0/1486644272 shutdown_connections 2024-04-05T23:46:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.439+0000 7f137ba00700 1 --2- 172.21.15.16:0/1486644272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374111dd0 0x7f13741141c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.439+0000 7f137ba00700 1 --2- 172.21.15.16:0/1486644272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1374073280 0x7f1374073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.439+0000 7f137ba00700 1 --2- 172.21.15.16:0/1486644272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1374073ba0 0x7f1374111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.439+0000 7f137ba00700 1 -- 172.21.15.16:0/1486644272 >> 172.21.15.16:0/1486644272 conn(0x7f13740fc8b0 msgr2=0x7f13740fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.439+0000 7f137ba00700 1 -- 172.21.15.16:0/1486644272 shutdown_connections 2024-04-05T23:46:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.439+0000 7f137ba00700 1 -- 172.21.15.16:0/1486644272 wait complete. 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.440+0000 7f137ba00700 1 Processor -- start 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.440+0000 7f137ba00700 1 -- start start 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.440+0000 7f137ba00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1374073280 0x7f13741a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.440+0000 7f137ba00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374073ba0 0x7f13741a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.440+0000 7f137ba00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1374111dd0 0x7f137419d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.440+0000 7f137ba00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137419def0 con 0x7f1374073ba0 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.440+0000 7f137ba00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137419e060 con 0x7f1374111dd0 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.440+0000 7f137ba00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137419e220 con 0x7f1374073280 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1373e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374073ba0 0x7f13741a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1379a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1374111dd0 0x7f137419d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1373e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374073ba0 0x7f13741a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36858/0 (socket says 172.21.15.16:36858) 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1373e00700 1 -- 172.21.15.16:0/1814029606 learned_addr learned my addr 172.21.15.16:0/1814029606 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1379000700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1374073280 0x7f13741a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1373e00700 1 -- 172.21.15.16:0/1814029606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1374073280 msgr2=0x7f13741a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1373e00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1374073280 0x7f13741a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1373e00700 1 -- 172.21.15.16:0/1814029606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1374111dd0 msgr2=0x7f137419d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1373e00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1374111dd0 0x7f137419d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1373e00700 1 -- 172.21.15.16:0/1814029606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13680095f0 con 0x7f1374073ba0 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1379a00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1374111dd0 0x7f137419d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.441+0000 7f1373e00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374073ba0 0x7f13741a3c40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f136000e9a0 tx=0x7f136000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.442+0000 7f1371600700 1 -- 172.21.15.16:0/1814029606 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360009790 con 0x7f1374073ba0 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.442+0000 7f1371600700 1 -- 172.21.15.16:0/1814029606 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1360010430 con 0x7f1374073ba0 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.442+0000 7f137ba00700 1 -- 172.21.15.16:0/1814029606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136c009c40 con 0x7f1374073ba0 2024-04-05T23:46:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.442+0000 7f1371600700 1 -- 172.21.15.16:0/1814029606 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360018680 con 0x7f1374073ba0 2024-04-05T23:46:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.442+0000 7f137ba00700 1 -- 172.21.15.16:0/1814029606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13741aa510 con 0x7f1374073ba0 2024-04-05T23:46:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.445+0000 7f1371600700 1 -- 172.21.15.16:0/1814029606 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f13600105a0 con 0x7f1374073ba0 2024-04-05T23:46:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f138f540b80 1 Processor -- start 2024-04-05T23:46:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1371600700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f135c0a1290 0x7f135c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f138f540b80 1 -- start start 2024-04-05T23:46:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1371600700 1 -- 172.21.15.16:0/1814029606 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8600+0+0 (secure 0 0 0) 0x7f1360014070 con 0x7f1374073ba0 2024-04-05T23:46:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379a00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f135c0a1290 0x7f135c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f138f540b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699eef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f138f540b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699e9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f138f540b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699e9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f138f540b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a6994fb80 con 0x561a699eeb80 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f138f540b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a69940930 con 0x561a699e95c0 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f138f540b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a6993e2f0 con 0x561a699ef4a0 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699eef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699eef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36860/0 (socket says 172.21.15.16:36860) 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379000700 1 -- 172.21.15.16:0/2999480008 learned_addr learned my addr 172.21.15.16:0/2999480008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1373e00700 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379a00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f135c0a1290 0x7f135c0a3750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f13741a4400 tx=0x7f136c009700 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379a00700 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699e9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379000700 1 -- 172.21.15.16:0/2999480008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 msgr2=0x561a699e9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379000700 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699e9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379000700 1 -- 172.21.15.16:0/2999480008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 msgr2=0x561a699e9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379000700 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699e9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379000700 1 -- 172.21.15.16:0/2999480008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a697bb350 con 0x561a699eeb80 2024-04-05T23:46:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1379a00700 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699e9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:46:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.447+0000 7f1373e00700 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:46:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.448+0000 7f1379000700 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699eef60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1368009fc0 tx=0x7f1368011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb869a35f5de6fc server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.448+0000 7f1366000700 1 -- 172.21.15.16:0/2999480008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136800e670 con 0x561a699eeb80 2024-04-05T23:46:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.448+0000 7f1366000700 1 -- 172.21.15.16:0/2999480008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f136800ec90 con 0x561a699eeb80 2024-04-05T23:46:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.448+0000 7f1366000700 1 -- 172.21.15.16:0/2999480008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368019d30 con 0x561a699eeb80 2024-04-05T23:46:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 -- 172.21.15.16:0/2999480008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 msgr2=0x561a699eef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699eef60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1368009fc0 tx=0x7f1368011720 comp rx=0 tx=0).stop 2024-04-05T23:46:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 -- 172.21.15.16:0/2999480008 shutdown_connections 2024-04-05T23:46:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699eef60 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699e9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 --2- 172.21.15.16:0/2999480008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 -- 172.21.15.16:0/2999480008 >> 172.21.15.16:0/2999480008 conn(0x561a6995d1c0 msgr2=0x561a699e3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 -- 172.21.15.16:0/2999480008 shutdown_connections 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 -- 172.21.15.16:0/2999480008 wait complete. 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 -- start start 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.449+0000 7f138f540b80 1 Processor -- start 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f138f540b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 0x561a699f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f138f540b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f138f540b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699ef4a0 0x561a699f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f138f540b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a69939be0 con 0x561a699eeb80 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f138f540b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a698d5d20 con 0x561a699ef4a0 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f138f540b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a698d59c0 con 0x561a699e95c0 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f138f540b80 10 client.0 _ll_register_callbacks cb 0x561a69741340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f1373e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f1379a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699ef4a0 0x561a699f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f1379000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 0x561a699f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f1373e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36864/0 (socket says 172.21.15.16:36864) 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.450+0000 7f1373e00700 1 -- 172.21.15.16:0/3378343091 learned_addr learned my addr 172.21.15.16:0/3378343091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f1379a00700 1 -- 172.21.15.16:0/3378343091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 msgr2=0x561a699f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f1379a00700 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 0x561a699f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f1379a00700 1 -- 172.21.15.16:0/3378343091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 msgr2=0x561a699f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f1379a00700 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f1379a00700 1 -- 172.21.15.16:0/3378343091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1368009c20 con 0x561a699ef4a0 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f1373e00700 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f1379a00700 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699ef4a0 0x561a699f2f60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f136c009790 tx=0x7f136c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a964ce430cb2a350 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f135aa00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f135aa00700 1 -- 172.21.15.16:0/3378343091 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c023040 con 0x561a699ef4a0 2024-04-05T23:46:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f135aa00700 1 -- 172.21.15.16:0/3378343091 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8600+0+0 (secure 0 0 0) 0x7f136c007e30 con 0x561a699ef4a0 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.451+0000 7f138f540b80 1 -- 172.21.15.16:0/3378343091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a699dc1e0 con 0x561a699ef4a0 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f135aa00700 1 -- 172.21.15.16:0/3378343091 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f136c00e040 con 0x561a699ef4a0 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f138f540b80 10 client.26852 fetch_fsmap learned FSMap version 77 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f138f540b80 1 -- 172.21.15.16:0/3378343091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=4+}) v3 -- 0x561a69767b00 con 0x561a699ef4a0 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f135aa00700 1 -- 172.21.15.16:0/3378343091 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561a699dc1e0 con 0x561a699ef4a0 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f135aa00700 1 -- 172.21.15.16:0/3378343091 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 77) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f136c029920 con 0x561a699ef4a0 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f138f540b80 10 client.26852 fetch_fsmap finished waiting for FSMap version 77 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f138f540b80 10 client.26852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f138f540b80 20 client.26852 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f138f540b80 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561a69a02f00 0x561a69a05320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f138f540b80 4 client.26852 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.452+0000 7f138f540b80 1 -- 172.21.15.16:0/3378343091 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a699326a0 con 0x561a69a02f00 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.453+0000 7f1379000700 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561a69a02f00 0x561a69a05320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.453+0000 7f1379000700 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561a69a02f00 0x561a69a05320 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c1e751d96bd47b4 server_cookie=bb737983c69279a0 in_seq=0 out_seq=0 2024-04-05T23:46:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.453+0000 7f135aa00700 10 client.26852 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f135aa00700 1 -- 172.21.15.16:0/3378343091 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561a699326a0 con 0x561a69a02f00 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f135aa00700 10 client.26852 handle_command_reply: tid=0 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 client.26852 shutdown 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 -- 172.21.15.16:0/3378343091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699ef4a0 msgr2=0x561a699f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699ef4a0 0x561a699f2f60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f136c009790 tx=0x7f136c00b330 comp rx=0 tx=0).stop 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 -- 172.21.15.16:0/3378343091 shutdown_connections 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561a69a02f00 0x561a69a05320 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699ef4a0 0x561a699f2f60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 0x561a699f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f135aa00700 0 client.26852 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 --2- 172.21.15.16:0/3378343091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 -- 172.21.15.16:0/3378343091 >> 172.21.15.16:0/3378343091 conn(0x561a6995d1c0 msgr2=0x561a699e3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 -- 172.21.15.16:0/3378343091 shutdown_connections 2024-04-05T23:46:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 1 -- 172.21.15.16:0/3378343091 wait complete. 2024-04-05T23:46:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.454+0000 7f138f540b80 20 client.26852 trim_cache size 0 max 16384 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.470+0000 7f138f540b80 1 Processor -- start 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.470+0000 7f138f540b80 1 -- start start 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.470+0000 7f138f540b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 0x561a699fef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.470+0000 7f138f540b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699eeb80 0x561a699dfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.470+0000 7f138f540b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699ef4a0 0x561a699f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.470+0000 7f138f540b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a697bd640 con 0x561a699ef4a0 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.470+0000 7f138f540b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a699f2610 con 0x561a699eeb80 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.470+0000 7f138f540b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a699ed520 con 0x561a699e95c0 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1373e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699eeb80 0x561a699dfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1373e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699eeb80 0x561a699dfef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35114/0 (socket says 172.21.15.16:35114) 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1373e00700 1 -- 172.21.15.16:0/915557378 learned_addr learned my addr 172.21.15.16:0/915557378 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1379a00700 1 --2- 172.21.15.16:0/915557378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699ef4a0 0x561a699f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1379000700 1 --2- 172.21.15.16:0/915557378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 0x561a699fef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1373e00700 1 -- 172.21.15.16:0/915557378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 msgr2=0x561a699fef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1373e00700 1 --2- 172.21.15.16:0/915557378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 0x561a699fef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1373e00700 1 -- 172.21.15.16:0/915557378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699ef4a0 msgr2=0x561a699f20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1373e00700 1 --2- 172.21.15.16:0/915557378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699ef4a0 0x561a699f20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1373e00700 1 -- 172.21.15.16:0/915557378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a698ffe40 con 0x561a699eeb80 2024-04-05T23:46:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1373e00700 1 --2- 172.21.15.16:0/915557378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699eeb80 0x561a699dfef0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1360099c60 tx=0x7f136000c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac4b13bc38cfa65 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.471+0000 7f1366a00700 1 -- 172.21.15.16:0/915557378 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360099f60 con 0x561a699eeb80 2024-04-05T23:46:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f1366a00700 1 -- 172.21.15.16:0/915557378 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f136009dd90 con 0x561a699eeb80 2024-04-05T23:46:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f1366a00700 1 -- 172.21.15.16:0/915557378 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13600996b0 con 0x561a699eeb80 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f138f540b80 1 -- 172.21.15.16:0/915557378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699eeb80 msgr2=0x561a699dfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f138f540b80 1 --2- 172.21.15.16:0/915557378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699eeb80 0x561a699dfef0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1360099c60 tx=0x7f136000c480 comp rx=0 tx=0).stop 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f138f540b80 1 -- 172.21.15.16:0/915557378 shutdown_connections 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f138f540b80 1 --2- 172.21.15.16:0/915557378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699ef4a0 0x561a699f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f138f540b80 1 --2- 172.21.15.16:0/915557378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699e95c0 0x561a699fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f138f540b80 1 --2- 172.21.15.16:0/915557378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699eeb80 0x561a699dfef0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f138f540b80 1 -- 172.21.15.16:0/915557378 >> 172.21.15.16:0/915557378 conn(0x561a6995d1c0 msgr2=0x561a69a09c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f138f540b80 1 -- 172.21.15.16:0/915557378 shutdown_connections 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.472+0000 7f138f540b80 1 -- 172.21.15.16:0/915557378 wait complete. 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.473+0000 7f138f540b80 1 -- start start 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.473+0000 7f138f540b80 1 Processor -- start 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.473+0000 7f138f540b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.473+0000 7f138f540b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.473+0000 7f138f540b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.473+0000 7f138f540b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a69a55e80 con 0x561a699eeb80 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.473+0000 7f138f540b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a69a55ff0 con 0x561a699e95c0 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.473+0000 7f138f540b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a69a561b0 con 0x561a699ef4a0 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.473+0000 7f138f540b80 10 client.0 _ll_register_callbacks cb 0x7f136c003db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699f5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35120/0 (socket says 172.21.15.16:35120) 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1373e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379000700 1 -- 172.21.15.16:0/124764285 learned_addr learned my addr 172.21.15.16:0/124764285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379a00700 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379000700 1 -- 172.21.15.16:0/124764285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 msgr2=0x561a699f69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379000700 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699f69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379000700 1 -- 172.21.15.16:0/124764285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 msgr2=0x561a699f6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379000700 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379000700 1 -- 172.21.15.16:0/124764285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13741aa510 con 0x561a699e95c0 2024-04-05T23:46:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379a00700 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:46:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f1379000700 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699f5f30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1368011b20 tx=0x7f1368019870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf2dc3ffaea2179 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f135aa00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:46:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.474+0000 7f135aa00700 1 -- 172.21.15.16:0/124764285 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136802a390 con 0x561a699e95c0 2024-04-05T23:46:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f138f540b80 1 -- 172.21.15.16:0/124764285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f136800ec90 con 0x561a699e95c0 2024-04-05T23:46:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f135aa00700 1 -- 172.21.15.16:0/124764285 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8600+0+0 (secure 0 0 0) 0x7f136802a4f0 con 0x561a699e95c0 2024-04-05T23:46:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f135aa00700 1 -- 172.21.15.16:0/124764285 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13680198e0 con 0x561a699e95c0 2024-04-05T23:46:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f138f540b80 10 client.26858 fetch_fsmap learned FSMap version 77 2024-04-05T23:46:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f138f540b80 1 -- 172.21.15.16:0/124764285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=4+}) v3 -- 0x561a697bfc70 con 0x561a699e95c0 2024-04-05T23:46:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f135aa00700 1 -- 172.21.15.16:0/124764285 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f136800ec90 con 0x561a699e95c0 2024-04-05T23:46:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f135aa00700 1 -- 172.21.15.16:0/124764285 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 77) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f1368005040 con 0x561a699e95c0 2024-04-05T23:46:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f138f540b80 10 client.26858 fetch_fsmap finished waiting for FSMap version 77 2024-04-05T23:46:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f138f540b80 10 client.26858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f138f540b80 20 client.26858 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f138f540b80 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561a69a57940 0x561a69a59d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f138f540b80 4 client.26858 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:46:04.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.475+0000 7f138f540b80 1 -- 172.21.15.16:0/124764285 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561a69788c90 con 0x561a69a57940 2024-04-05T23:46:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.476+0000 7f1373e00700 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561a69a57940 0x561a69a59d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.476+0000 7f1373e00700 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561a69a57940 0x561a69a59d60 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=535e6155e51880f9 server_cookie=6864fba4d9f79e60 in_seq=0 out_seq=0 2024-04-05T23:46:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.476+0000 7f135aa00700 10 client.26858 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f135aa00700 1 -- 172.21.15.16:0/124764285 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561a69788c90 con 0x561a69a57940 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f135aa00700 10 client.26858 handle_command_reply: tid=0 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 client.26858 shutdown 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 -- 172.21.15.16:0/124764285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 msgr2=0x561a699f5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699f5f30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1368011b20 tx=0x7f1368019870 comp rx=0 tx=0).stop 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 -- 172.21.15.16:0/124764285 shutdown_connections 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561a69a57940 0x561a69a59d60 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a699ef4a0 0x561a699f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a699e95c0 0x561a699f5f30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 --2- 172.21.15.16:0/124764285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561a699eeb80 0x561a699f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f135aa00700 0 client.26858 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 -- 172.21.15.16:0/124764285 >> 172.21.15.16:0/124764285 conn(0x561a6995d1c0 msgr2=0x561a699e99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 -- 172.21.15.16:0/124764285 shutdown_connections 2024-04-05T23:46:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 1 -- 172.21.15.16:0/124764285 wait complete. 2024-04-05T23:46:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f138f540b80 20 client.26858 trim_cache size 0 max 16384 2024-04-05T23:46:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:46:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:46:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:46:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:46:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 -- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f135c0a1290 msgr2=0x7f135c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f135c0a1290 0x7f135c0a3750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f13741a4400 tx=0x7f136c009700 comp rx=0 tx=0).stop 2024-04-05T23:46:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 -- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374073ba0 msgr2=0x7f13741a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374073ba0 0x7f13741a3c40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f136000e9a0 tx=0x7f136000ecb0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 -- 172.21.15.16:0/1814029606 shutdown_connections 2024-04-05T23:46:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f135c0a1290 0x7f135c0a3750 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1374111dd0 0x7f137419d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1374073280 0x7f13741a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 --2- 172.21.15.16:0/1814029606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1374073ba0 0x7f13741a3c40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 -- 172.21.15.16:0/1814029606 >> 172.21.15.16:0/1814029606 conn(0x7f13740fc8b0 msgr2=0x7f13740fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 -- 172.21.15.16:0/1814029606 shutdown_connections 2024-04-05T23:46:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:04.508+0000 7f137ba00700 1 -- 172.21.15.16:0/1814029606 wait complete. 2024-04-05T23:46:05.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 598403 4.60 MB/sec execute 564 sec latency 2322.530 ms 2024-04-05T23:46:06.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 598690 4.59 MB/sec execute 565 sec latency 3181.620 ms 2024-04-05T23:46:06.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:05 smithi027 ceph-mon[78597]: pgmap v394: 97 pgs: 97 active+clean; 244 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-04-05T23:46:06.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:05 smithi016 ceph-mon[38248]: pgmap v394: 97 pgs: 97 active+clean; 244 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-04-05T23:46:06.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:05 smithi130 ceph-mon[79417]: pgmap v394: 97 pgs: 97 active+clean; 244 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-04-05T23:46:07.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 600554 4.61 MB/sec execute 566 sec latency 16.974 ms 2024-04-05T23:46:08.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 601743 4.61 MB/sec execute 567 sec latency 27.539 ms 2024-04-05T23:46:08.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:07 smithi027 ceph-mon[78597]: pgmap v395: 97 pgs: 97 active+clean; 242 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.3 MiB/s wr, 178 op/s 2024-04-05T23:46:08.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:07 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:08.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:07 smithi016 ceph-mon[38248]: pgmap v395: 97 pgs: 97 active+clean; 242 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.3 MiB/s wr, 178 op/s 2024-04-05T23:46:08.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:07 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:08.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:07 smithi130 ceph-mon[79417]: pgmap v395: 97 pgs: 97 active+clean; 242 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.3 MiB/s wr, 178 op/s 2024-04-05T23:46:08.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:07 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:09.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 601947 4.60 MB/sec execute 568 sec latency 849.605 ms 2024-04-05T23:46:10.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 601947 4.59 MB/sec execute 569 sec latency 1849.660 ms 2024-04-05T23:46:10.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:10 smithi027 ceph-mon[78597]: pgmap v396: 97 pgs: 97 active+clean; 242 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.3 MiB/s wr, 178 op/s 2024-04-05T23:46:10.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:10 smithi027 ceph-mon[78597]: osdmap e99: 12 total, 12 up, 12 in 2024-04-05T23:46:11.038 INFO:tasks.workunit.client.0.smithi016.stdout: 1 601947 4.58 MB/sec execute 570 sec latency 2849.780 ms 2024-04-05T23:46:11.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:10 smithi016 ceph-mon[38248]: pgmap v396: 97 pgs: 97 active+clean; 242 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.3 MiB/s wr, 178 op/s 2024-04-05T23:46:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:10 smithi016 ceph-mon[38248]: osdmap e99: 12 total, 12 up, 12 in 2024-04-05T23:46:11.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:10 smithi130 ceph-mon[79417]: pgmap v396: 97 pgs: 97 active+clean; 242 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.3 MiB/s wr, 178 op/s 2024-04-05T23:46:11.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:10 smithi130 ceph-mon[79417]: osdmap e99: 12 total, 12 up, 12 in 2024-04-05T23:46:12.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 601947 4.58 MB/sec execute 571 sec latency 3849.916 ms 2024-04-05T23:46:12.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:12 smithi027 ceph-mon[78597]: pgmap v398: 97 pgs: 97 active+clean; 245 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-04-05T23:46:13.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 603455 4.58 MB/sec execute 572 sec latency 4033.231 ms 2024-04-05T23:46:13.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:12 smithi016 ceph-mon[38248]: pgmap v398: 97 pgs: 97 active+clean; 245 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-04-05T23:46:13.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:12 smithi130 ceph-mon[79417]: pgmap v398: 97 pgs: 97 active+clean; 245 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-04-05T23:46:14.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 604677 4.58 MB/sec execute 573 sec latency 15.368 ms 2024-04-05T23:46:14.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:14 smithi027 ceph-mon[78597]: pgmap v399: 97 pgs: 97 active+clean; 245 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-05T23:46:15.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 605495 4.58 MB/sec execute 574 sec latency 456.599 ms 2024-04-05T23:46:15.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:14 smithi016 ceph-mon[38248]: pgmap v399: 97 pgs: 97 active+clean; 245 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-05T23:46:15.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:14 smithi130 ceph-mon[79417]: pgmap v399: 97 pgs: 97 active+clean; 245 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-05T23:46:16.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 605495 4.58 MB/sec execute 575 sec latency 1456.704 ms 2024-04-05T23:46:16.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:16 smithi027 ceph-mon[78597]: pgmap v400: 97 pgs: 97 active+clean; 245 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-05T23:46:17.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 605495 4.57 MB/sec execute 576 sec latency 2456.853 ms 2024-04-05T23:46:17.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:16 smithi016 ceph-mon[38248]: pgmap v400: 97 pgs: 97 active+clean; 245 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-05T23:46:17.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:16 smithi130 ceph-mon[79417]: pgmap v400: 97 pgs: 97 active+clean; 245 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-05T23:46:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.677+0000 7fdc32200700 1 -- 172.21.15.16:0/4040637518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc2c101040 msgr2=0x7fdc2c101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.677+0000 7fdc32200700 1 --2- 172.21.15.16:0/4040637518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc2c101040 0x7fdc2c101420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009990 tx=0x7fdc1c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.678+0000 7fdc32200700 1 -- 172.21.15.16:0/4040637518 shutdown_connections 2024-04-05T23:46:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.678+0000 7fdc32200700 1 --2- 172.21.15.16:0/4040637518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc2c106160 0x7fdc2c10dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.678+0000 7fdc32200700 1 --2- 172.21.15.16:0/4040637518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc2c101040 0x7fdc2c101420 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.678+0000 7fdc32200700 1 --2- 172.21.15.16:0/4040637518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c1019f0 0x7fdc2c105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.678+0000 7fdc32200700 1 -- 172.21.15.16:0/4040637518 >> 172.21.15.16:0/4040637518 conn(0x7fdc2c0fc8e0 msgr2=0x7fdc2c0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.678+0000 7fdc32200700 1 -- 172.21.15.16:0/4040637518 shutdown_connections 2024-04-05T23:46:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.678+0000 7fdc32200700 1 -- 172.21.15.16:0/4040637518 wait complete. 2024-04-05T23:46:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.679+0000 7fdc32200700 1 Processor -- start 2024-04-05T23:46:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.679+0000 7fdc32200700 1 -- start start 2024-04-05T23:46:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.679+0000 7fdc32200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c101040 0x7fdc2c1972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.679+0000 7fdc32200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc2c106160 0x7fdc2c197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.679+0000 7fdc32200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc2c19bba0 0x7fdc2c19c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.679+0000 7fdc32200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c19c620 con 0x7fdc2c101040 2024-04-05T23:46:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.679+0000 7fdc32200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c19c790 con 0x7fdc2c106160 2024-04-05T23:46:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.679+0000 7fdc32200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c19c950 con 0x7fdc2c19bba0 2024-04-05T23:46:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c101040 0x7fdc2c1972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2b400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c101040 0x7fdc2c1972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35558/0 (socket says 172.21.15.16:35558) 2024-04-05T23:46:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2b400700 1 -- 172.21.15.16:0/1228455612 learned_addr learned my addr 172.21.15.16:0/1228455612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2aa00700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc2c106160 0x7fdc2c197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2be00700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc2c19bba0 0x7fdc2c19c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2b400700 1 -- 172.21.15.16:0/1228455612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc2c19bba0 msgr2=0x7fdc2c19c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2b400700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc2c19bba0 0x7fdc2c19c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2b400700 1 -- 172.21.15.16:0/1228455612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc2c106160 msgr2=0x7fdc2c197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2b400700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc2c106160 0x7fdc2c197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.680+0000 7fdc2b400700 1 -- 172.21.15.16:0/1228455612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c0095f0 con 0x7fdc2c101040 2024-04-05T23:46:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.681+0000 7fdc2b400700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c101040 0x7fdc2c1972c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c005910 tx=0x7fdc1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.681+0000 7fdc23e00700 1 -- 172.21.15.16:0/1228455612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c01d070 con 0x7fdc2c101040 2024-04-05T23:46:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.681+0000 7fdc23e00700 1 -- 172.21.15.16:0/1228455612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc1c022470 con 0x7fdc2c101040 2024-04-05T23:46:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.681+0000 7fdc32200700 1 -- 172.21.15.16:0/1228455612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c1a3e50 con 0x7fdc2c101040 2024-04-05T23:46:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.681+0000 7fdc23e00700 1 -- 172.21.15.16:0/1228455612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c017650 con 0x7fdc2c101040 2024-04-05T23:46:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.681+0000 7fdc32200700 1 -- 172.21.15.16:0/1228455612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c1a4270 con 0x7fdc2c101040 2024-04-05T23:46:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.684+0000 7fdc23e00700 1 -- 172.21.15.16:0/1228455612 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fdc1c0225e0 con 0x7fdc2c101040 2024-04-05T23:46:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.686+0000 7fdc45e01b80 1 Processor -- start 2024-04-05T23:46:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.686+0000 7fdc45e01b80 1 -- start start 2024-04-05T23:46:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.686+0000 7fdc23e00700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc100a1290 0x7fdc100a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc23e00700 1 -- 172.21.15.16:0/1228455612 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdc1c0cdf50 con 0x7fdc2c101040 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc2be00700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc100a1290 0x7fdc100a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c090 0x561e09a8c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 0x561e09a86440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a86e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e099e8b80 con 0x561e09a8c9b0 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e099d9930 con 0x561e09a86980 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e099d72f0 con 0x561e09a8c090 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc2b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c090 0x561e09a8c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc2be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a86e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc2b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c090 0x561e09a8c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:50402/0 (socket says 172.21.15.16:50402) 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc2b400700 1 -- 172.21.15.16:0/249205628 learned_addr learned my addr 172.21.15.16:0/249205628 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc2be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a86e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:41998/0 (socket says 172.21.15.16:41998) 2024-04-05T23:46:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.687+0000 7fdc2aa00700 1 --2- 172.21.15.16:0/249205628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 0x561e09a86440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc2b400700 1 -- 172.21.15.16:0/249205628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 msgr2=0x561e09a86e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc2b400700 1 --2- 172.21.15.16:0/249205628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a86e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc2b400700 1 -- 172.21.15.16:0/249205628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 msgr2=0x561e09a86440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc2b400700 1 --2- 172.21.15.16:0/249205628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 0x561e09a86440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc2b400700 1 -- 172.21.15.16:0/249205628 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e09854350 con 0x561e09a8c090 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc2be00700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc100a1290 0x7fdc100a3750 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400f870 tx=0x7fdc2400fc30 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc2b400700 1 --2- 172.21.15.16:0/249205628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c090 0x561e09a8c470 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c09b120 tx=0x7fdc1c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=257444bed2ca0448 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc17e00700 1 -- 172.21.15.16:0/249205628 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c09b890 con 0x561e09a8c090 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc17e00700 1 -- 172.21.15.16:0/249205628 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc1c09b190 con 0x561e09a8c090 2024-04-05T23:46:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.688+0000 7fdc17e00700 1 -- 172.21.15.16:0/249205628 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c0a12d0 con 0x561e09a8c090 2024-04-05T23:46:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.689+0000 7fdc45e01b80 1 -- 172.21.15.16:0/249205628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c090 msgr2=0x561e09a8c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.689+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/249205628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c090 0x561e09a8c470 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c09b120 tx=0x7fdc1c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.690+0000 7fdc45e01b80 1 -- 172.21.15.16:0/249205628 shutdown_connections 2024-04-05T23:46:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.690+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/249205628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a86e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.690+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/249205628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c090 0x561e09a8c470 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.690+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/249205628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 0x561e09a86440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.690+0000 7fdc45e01b80 1 -- 172.21.15.16:0/249205628 >> 172.21.15.16:0/249205628 conn(0x561e099f61c0 msgr2=0x561e09a7cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.690+0000 7fdc45e01b80 1 -- 172.21.15.16:0/249205628 shutdown_connections 2024-04-05T23:46:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.690+0000 7fdc45e01b80 1 -- 172.21.15.16:0/249205628 wait complete. 2024-04-05T23:46:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.690+0000 7fdc45e01b80 1 -- start start 2024-04-05T23:46:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.691+0000 7fdc45e01b80 1 Processor -- start 2024-04-05T23:46:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.691+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a928d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.691+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.691+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a90350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.691+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e099d2be0 con 0x561e09a8c090 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.691+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e0996ed20 con 0x561e09a86980 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.691+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e0996e9c0 con 0x561e09a8c9b0 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.691+0000 7fdc45e01b80 10 client.0 _ll_register_callbacks cb 0x561e097da340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a928d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a928d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:42014/0 (socket says 172.21.15.16:42014) 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2b400700 1 -- 172.21.15.16:0/701249950 learned_addr learned my addr 172.21.15.16:0/701249950 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a90350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2aa00700 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2be00700 1 -- 172.21.15.16:0/701249950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 msgr2=0x561e09a928d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2be00700 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a928d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2be00700 1 -- 172.21.15.16:0/701249950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 msgr2=0x561e09a8fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2be00700 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2be00700 1 -- 172.21.15.16:0/701249950 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c1a4270 con 0x561e09a8c9b0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2aa00700 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2b400700 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a928d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc2be00700 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a90350 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400f650 tx=0x7fdc24004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d541ff3a6ac8f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.692+0000 7fdc14c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc14c00700 1 -- 172.21.15.16:0/701249950 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2400e040 con 0x561e09a8c9b0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc45e01b80 1 -- 172.21.15.16:0/701249950 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561e09a751e0 con 0x561e09a8c9b0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc14c00700 1 -- 172.21.15.16:0/701249950 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdc24004bb0 con 0x561e09a8c9b0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc14c00700 1 -- 172.21.15.16:0/701249950 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc24018e60 con 0x561e09a8c9b0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc45e01b80 10 client.26899 fetch_fsmap learned FSMap version 78 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc45e01b80 1 -- 172.21.15.16:0/701249950 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) v3 -- 0x561e09800b00 con 0x561e09a8c9b0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc14c00700 1 -- 172.21.15.16:0/701249950 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561e09a751e0 con 0x561e09a8c9b0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc14c00700 1 -- 172.21.15.16:0/701249950 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 78) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdc2402ca00 con 0x561e09a8c9b0 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc45e01b80 10 client.26899 fetch_fsmap finished waiting for FSMap version 78 2024-04-05T23:46:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc45e01b80 10 client.26899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc45e01b80 20 client.26899 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561e09aa0140 0x561e09aa2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc45e01b80 4 client.26899 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:46:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.693+0000 7fdc45e01b80 1 -- 172.21.15.16:0/701249950 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e099cb6a0 con 0x561e09aa0140 2024-04-05T23:46:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.694+0000 7fdc2b400700 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561e09aa0140 0x561e09aa2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.694+0000 7fdc2b400700 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561e09aa0140 0x561e09aa2560 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed819499f6838846 server_cookie=29f00ea3f6e2d5b9 in_seq=0 out_seq=0 2024-04-05T23:46:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.694+0000 7fdc14c00700 10 client.26899 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc14c00700 1 -- 172.21.15.16:0/701249950 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561e099cb6a0 con 0x561e09aa0140 2024-04-05T23:46:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc14c00700 10 client.26899 handle_command_reply: tid=0 2024-04-05T23:46:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 client.26899 shutdown 2024-04-05T23:46:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 -- 172.21.15.16:0/701249950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 msgr2=0x561e09a90350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a90350 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400f650 tx=0x7fdc24004b40 comp rx=0 tx=0).stop 2024-04-05T23:46:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 -- 172.21.15.16:0/701249950 shutdown_connections 2024-04-05T23:46:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561e09aa0140 0x561e09aa2560 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a90350 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a928d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/701249950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc14c00700 0 client.26899 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 -- 172.21.15.16:0/701249950 >> 172.21.15.16:0/701249950 conn(0x561e099f61c0 msgr2=0x561e09a7c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 -- 172.21.15.16:0/701249950 shutdown_connections 2024-04-05T23:46:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.695+0000 7fdc45e01b80 1 -- 172.21.15.16:0/701249950 wait complete. 2024-04-05T23:46:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.696+0000 7fdc45e01b80 20 client.26899 trim_cache size 0 max 16384 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.713+0000 7fdc45e01b80 1 Processor -- start 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.713+0000 7fdc45e01b80 1 -- start start 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.713+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a86980 0x561e09a90260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.713+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a8c090 0x561e09a8f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.713+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 0x561e09a7ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.713+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e09856640 con 0x561e09a8c9b0 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.713+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e09a86840 con 0x561e09a8c090 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.713+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e09a8fa60 con 0x561e09a86980 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 0x561e09a7ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a86980 0x561e09a90260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a8c090 0x561e09a8f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a86980 0x561e09a90260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:50416/0 (socket says 172.21.15.16:50416) 2024-04-05T23:46:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 0x561e09a7ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35578/0 (socket says 172.21.15.16:35578) 2024-04-05T23:46:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2b400700 1 -- 172.21.15.16:0/2356524866 learned_addr learned my addr 172.21.15.16:0/2356524866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2b400700 1 -- 172.21.15.16:0/2356524866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a8c090 msgr2=0x561e09a8f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2b400700 1 --2- 172.21.15.16:0/2356524866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a8c090 0x561e09a8f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2b400700 1 -- 172.21.15.16:0/2356524866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 msgr2=0x561e09a7ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2b400700 1 --2- 172.21.15.16:0/2356524866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 0x561e09a7ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2b400700 1 -- 172.21.15.16:0/2356524866 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e09998e40 con 0x561e09a86980 2024-04-05T23:46:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc2b400700 1 --2- 172.21.15.16:0/2356524866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a86980 0x561e09a90260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00b680 tx=0x7fdc1c01a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78a0025cbf784b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc21200700 1 -- 172.21.15.16:0/2356524866 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c09b890 con 0x561e09a86980 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.714+0000 7fdc21200700 1 -- 172.21.15.16:0/2356524866 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc1c09efc0 con 0x561e09a86980 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.715+0000 7fdc21200700 1 -- 172.21.15.16:0/2356524866 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c0b94d0 con 0x561e09a86980 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.715+0000 7fdc45e01b80 1 -- 172.21.15.16:0/2356524866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a86980 msgr2=0x561e09a90260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.715+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/2356524866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a86980 0x561e09a90260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00b680 tx=0x7fdc1c01a040 comp rx=0 tx=0).stop 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.715+0000 7fdc45e01b80 1 -- 172.21.15.16:0/2356524866 shutdown_connections 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.715+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/2356524866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c9b0 0x561e09a7ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.715+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/2356524866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a86980 0x561e09a90260 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.715+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/2356524866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a8c090 0x561e09a8f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.715+0000 7fdc45e01b80 1 -- 172.21.15.16:0/2356524866 >> 172.21.15.16:0/2356524866 conn(0x561e099f61c0 msgr2=0x561e09aa6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.715+0000 7fdc45e01b80 1 -- 172.21.15.16:0/2356524866 shutdown_connections 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.716+0000 7fdc45e01b80 1 -- 172.21.15.16:0/2356524866 wait complete. 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.716+0000 7fdc45e01b80 1 -- start start 2024-04-05T23:46:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.716+0000 7fdc45e01b80 1 Processor -- start 2024-04-05T23:46:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a97160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc45e01b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a8d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e09a8d9c0 con 0x561e09a8c090 2024-04-05T23:46:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e09a8db30 con 0x561e09a86980 2024-04-05T23:46:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc45e01b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e09af2f60 con 0x561e09a8c9b0 2024-04-05T23:46:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc45e01b80 10 client.0 _ll_register_callbacks cb 0x7fdc24009350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a8d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35582/0 (socket says 172.21.15.16:35582) 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2aa00700 1 -- 172.21.15.16:0/120394361 learned_addr learned my addr 172.21.15.16:0/120394361 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2b400700 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a97160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2be00700 1 -- 172.21.15.16:0/120394361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 msgr2=0x561e09a97160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2be00700 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a97160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2be00700 1 -- 172.21.15.16:0/120394361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 msgr2=0x561e09a8cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2be00700 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2be00700 1 -- 172.21.15.16:0/120394361 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c062040 con 0x561e09a8c9b0 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2aa00700 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.717+0000 7fdc2b400700 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a97160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.718+0000 7fdc2be00700 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a8d2d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdc2402cdb0 tx=0x7fdc240087f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbfe8b10d8c15810 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.718+0000 7fdc14c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.718+0000 7fdc14c00700 1 -- 172.21.15.16:0/120394361 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2400e070 con 0x561e09a8c9b0 2024-04-05T23:46:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.718+0000 7fdc14c00700 1 -- 172.21.15.16:0/120394361 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdc24008d10 con 0x561e09a8c9b0 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.718+0000 7fdc45e01b80 1 -- 172.21.15.16:0/120394361 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc1c09b190 con 0x561e09a8c9b0 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.718+0000 7fdc14c00700 1 -- 172.21.15.16:0/120394361 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc24008ed0 con 0x561e09a8c9b0 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.718+0000 7fdc45e01b80 10 client.26905 fetch_fsmap learned FSMap version 78 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.718+0000 7fdc45e01b80 1 -- 172.21.15.16:0/120394361 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) v3 -- 0x561e09858c70 con 0x561e09a8c9b0 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.719+0000 7fdc14c00700 1 -- 172.21.15.16:0/120394361 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc1c09b190 con 0x561e09a8c9b0 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.719+0000 7fdc14c00700 1 -- 172.21.15.16:0/120394361 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 78) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdc2402c8e0 con 0x561e09a8c9b0 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.719+0000 7fdc45e01b80 10 client.26905 fetch_fsmap finished waiting for FSMap version 78 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.719+0000 7fdc45e01b80 10 client.26905 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.719+0000 7fdc45e01b80 20 client.26905 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.719+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561e09af4660 0x561e09af6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.719+0000 7fdc45e01b80 4 client.26905 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.719+0000 7fdc45e01b80 1 -- 172.21.15.16:0/120394361 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561e09821c90 con 0x561e09af4660 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.719+0000 7fdc2b400700 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561e09af4660 0x561e09af6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.720+0000 7fdc2b400700 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561e09af4660 0x561e09af6a80 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=927b110e514fbb7d server_cookie=cdf67ea92c147b9b in_seq=0 out_seq=0 2024-04-05T23:46:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.720+0000 7fdc14c00700 10 client.26905 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc14c00700 1 -- 172.21.15.16:0/120394361 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561e09821c90 con 0x561e09af4660 2024-04-05T23:46:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc14c00700 10 client.26905 handle_command_reply: tid=0 2024-04-05T23:46:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 client.26905 shutdown 2024-04-05T23:46:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 -- 172.21.15.16:0/120394361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 msgr2=0x561e09a8d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a8d2d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdc2402cdb0 tx=0x7fdc240087f0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 -- 172.21.15.16:0/120394361 shutdown_connections 2024-04-05T23:46:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x561e09af4660 0x561e09af6a80 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e09a8c9b0 0x561e09a8d2d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561e09a86980 0x561e09a97160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 --2- 172.21.15.16:0/120394361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e09a8c090 0x561e09a8cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 -- 172.21.15.16:0/120394361 >> 172.21.15.16:0/120394361 conn(0x561e099f61c0 msgr2=0x561e09a89e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc14c00700 0 client.26905 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 -- 172.21.15.16:0/120394361 shutdown_connections 2024-04-05T23:46:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.754+0000 7fdc45e01b80 1 -- 172.21.15.16:0/120394361 wait complete. 2024-04-05T23:46:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc45e01b80 20 client.26905 trim_cache size 0 max 16384 2024-04-05T23:46:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:46:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:46:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:46:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 -- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc100a1290 msgr2=0x7fdc100a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc100a1290 0x7fdc100a3750 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400f870 tx=0x7fdc2400fc30 comp rx=0 tx=0).stop 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 -- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c101040 msgr2=0x7fdc2c1972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c101040 0x7fdc2c1972c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c005910 tx=0x7fdc1c004730 comp rx=0 tx=0).stop 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 -- 172.21.15.16:0/1228455612 shutdown_connections 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdc100a1290 0x7fdc100a3750 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c101040 0x7fdc2c1972c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc2c19bba0 0x7fdc2c19c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 --2- 172.21.15.16:0/1228455612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc2c106160 0x7fdc2c197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.755+0000 7fdc32200700 1 -- 172.21.15.16:0/1228455612 >> 172.21.15.16:0/1228455612 conn(0x7fdc2c0fc8e0 msgr2=0x7fdc2c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.756+0000 7fdc32200700 1 -- 172.21.15.16:0/1228455612 shutdown_connections 2024-04-05T23:46:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:17.756+0000 7fdc32200700 1 -- 172.21.15.16:0/1228455612 wait complete. 2024-04-05T23:46:18.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 607005 4.57 MB/sec execute 577 sec latency 2641.182 ms 2024-04-05T23:46:18.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:18 smithi027 ceph-mon[78597]: pgmap v401: 97 pgs: 97 active+clean; 249 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.8 MiB/s wr, 142 op/s 2024-04-05T23:46:19.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 608134 4.58 MB/sec execute 578 sec latency 15.464 ms 2024-04-05T23:46:19.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:18 smithi016 ceph-mon[38248]: pgmap v401: 97 pgs: 97 active+clean; 249 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.8 MiB/s wr, 142 op/s 2024-04-05T23:46:19.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:18 smithi130 ceph-mon[79417]: pgmap v401: 97 pgs: 97 active+clean; 249 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.8 MiB/s wr, 142 op/s 2024-04-05T23:46:20.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 609036 4.58 MB/sec execute 579 sec latency 417.803 ms 2024-04-05T23:46:20.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:20 smithi027 ceph-mon[78597]: pgmap v402: 97 pgs: 97 active+clean; 249 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.8 MiB/s wr, 142 op/s 2024-04-05T23:46:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 611002 4.59 MB/sec execute 580 sec latency 483.222 ms 2024-04-05T23:46:21.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:20 smithi016 ceph-mon[38248]: pgmap v402: 97 pgs: 97 active+clean; 249 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.8 MiB/s wr, 142 op/s 2024-04-05T23:46:21.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:20 smithi130 ceph-mon[79417]: pgmap v402: 97 pgs: 97 active+clean; 249 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.8 MiB/s wr, 142 op/s 2024-04-05T23:46:22.039 INFO:tasks.workunit.client.0.smithi016.stdout: 1 612081 4.59 MB/sec execute 581 sec latency 14.608 ms 2024-04-05T23:46:22.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:22 smithi027 ceph-mon[78597]: pgmap v403: 97 pgs: 97 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 2.4 MiB/s wr, 200 op/s 2024-04-05T23:46:23.040 INFO:tasks.workunit.client.0.smithi016.stdout: 1 612586 4.59 MB/sec execute 582 sec latency 697.511 ms 2024-04-05T23:46:23.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:22 smithi016 ceph-mon[38248]: pgmap v403: 97 pgs: 97 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 2.4 MiB/s wr, 200 op/s 2024-04-05T23:46:23.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:22 smithi130 ceph-mon[79417]: pgmap v403: 97 pgs: 97 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 2.4 MiB/s wr, 200 op/s 2024-04-05T23:46:24.040 INFO:tasks.workunit.client.0.smithi016.stdout: 1 612586 4.58 MB/sec execute 583 sec latency 1697.637 ms 2024-04-05T23:46:24.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:24 smithi027 ceph-mon[78597]: pgmap v404: 97 pgs: 97 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-04-05T23:46:25.040 INFO:tasks.workunit.client.0.smithi016.stdout: 1 612586 4.57 MB/sec execute 584 sec latency 2697.799 ms 2024-04-05T23:46:25.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:24 smithi016 ceph-mon[38248]: pgmap v404: 97 pgs: 97 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-04-05T23:46:25.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:24 smithi130 ceph-mon[79417]: pgmap v404: 97 pgs: 97 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-04-05T23:46:25.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:25 smithi027 ceph-mon[78597]: pgmap v405: 97 pgs: 97 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-04-05T23:46:26.040 INFO:tasks.workunit.client.0.smithi016.stdout: 1 612861 4.56 MB/sec execute 585 sec latency 3559.531 ms 2024-04-05T23:46:26.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:25 smithi016 ceph-mon[38248]: pgmap v405: 97 pgs: 97 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-04-05T23:46:26.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:25 smithi130 ceph-mon[79417]: pgmap v405: 97 pgs: 97 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-04-05T23:46:27.040 INFO:tasks.workunit.client.0.smithi016.stdout: 1 614694 4.58 MB/sec execute 586 sec latency 16.192 ms 2024-04-05T23:46:28.040 INFO:tasks.workunit.client.0.smithi016.stdout: 1 615862 4.58 MB/sec execute 587 sec latency 22.070 ms 2024-04-05T23:46:28.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:27 smithi027 ceph-mon[78597]: pgmap v406: 97 pgs: 97 active+clean; 253 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 2.0 MiB/s wr, 173 op/s 2024-04-05T23:46:28.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:27 smithi016 ceph-mon[38248]: pgmap v406: 97 pgs: 97 active+clean; 253 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 2.0 MiB/s wr, 173 op/s 2024-04-05T23:46:28.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:27 smithi130 ceph-mon[79417]: pgmap v406: 97 pgs: 97 active+clean; 253 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 2.0 MiB/s wr, 173 op/s 2024-04-05T23:46:29.040 INFO:tasks.workunit.client.0.smithi016.stdout: 1 616129 4.57 MB/sec execute 588 sec latency 803.055 ms 2024-04-05T23:46:30.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 616129 4.57 MB/sec execute 589 sec latency 1803.218 ms 2024-04-05T23:46:30.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:29 smithi027 ceph-mon[78597]: pgmap v407: 97 pgs: 97 active+clean; 253 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-05T23:46:30.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:29 smithi016 ceph-mon[38248]: pgmap v407: 97 pgs: 97 active+clean; 253 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-05T23:46:30.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:29 smithi130 ceph-mon[79417]: pgmap v407: 97 pgs: 97 active+clean; 253 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-05T23:46:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.918+0000 7fe6dd800700 1 -- 172.21.15.16:0/3229054506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 msgr2=0x7fe6d8101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.918+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3229054506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 0x7fe6d8101410 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009990 tx=0x7fe6c8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.919+0000 7fe6dd800700 1 -- 172.21.15.16:0/3229054506 shutdown_connections 2024-04-05T23:46:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.919+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3229054506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 0x7fe6d8101410 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.919+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3229054506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d8106150 0x7fe6d810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.919+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3229054506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6d81019e0 0x7fe6d8105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.919+0000 7fe6dd800700 1 -- 172.21.15.16:0/3229054506 >> 172.21.15.16:0/3229054506 conn(0x7fe6d80fc8b0 msgr2=0x7fe6d80fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.919+0000 7fe6dd800700 1 -- 172.21.15.16:0/3229054506 shutdown_connections 2024-04-05T23:46:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.919+0000 7fe6dd800700 1 -- 172.21.15.16:0/3229054506 wait complete. 2024-04-05T23:46:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.920+0000 7fe6dd800700 1 Processor -- start 2024-04-05T23:46:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.920+0000 7fe6dd800700 1 -- start start 2024-04-05T23:46:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.920+0000 7fe6dd800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 0x7fe6d8199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.920+0000 7fe6dd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d8106150 0x7fe6d8199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.920+0000 7fe6dd800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6d819de60 0x7fe6d819e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.920+0000 7fe6dd800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d819e8e0 con 0x7fe6d8101030 2024-04-05T23:46:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.920+0000 7fe6dd800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d819ea50 con 0x7fe6d819de60 2024-04-05T23:46:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.920+0000 7fe6dd800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d819ec10 con 0x7fe6d8106150 2024-04-05T23:46:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 0x7fe6d8199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 0x7fe6d8199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59322/0 (socket says 172.21.15.16:59322) 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6a00700 1 -- 172.21.15.16:0/3893343506 learned_addr learned my addr 172.21.15.16:0/3893343506 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6000700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d8106150 0x7fe6d8199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d7400700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6d819de60 0x7fe6d819e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6a00700 1 -- 172.21.15.16:0/3893343506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d8106150 msgr2=0x7fe6d8199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d8106150 0x7fe6d8199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6a00700 1 -- 172.21.15.16:0/3893343506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6d819de60 msgr2=0x7fe6d819e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6d819de60 0x7fe6d819e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6a00700 1 -- 172.21.15.16:0/3893343506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c80095f0 con 0x7fe6d8101030 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 0x7fe6d8199580 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8000c00 tx=0x7fe6c8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.921+0000 7fe6cf400700 1 -- 172.21.15.16:0/3893343506 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c801d070 con 0x7fe6d8101030 2024-04-05T23:46:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.922+0000 7fe6cf400700 1 -- 172.21.15.16:0/3893343506 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6c8022470 con 0x7fe6d8101030 2024-04-05T23:46:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.922+0000 7fe6dd800700 1 -- 172.21.15.16:0/3893343506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d81a6110 con 0x7fe6d8101030 2024-04-05T23:46:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.922+0000 7fe6cf400700 1 -- 172.21.15.16:0/3893343506 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8017650 con 0x7fe6d8101030 2024-04-05T23:46:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.922+0000 7fe6dd800700 1 -- 172.21.15.16:0/3893343506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d81a6530 con 0x7fe6d8101030 2024-04-05T23:46:30.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.925+0000 7fe6cf400700 1 -- 172.21.15.16:0/3893343506 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fe6c80225e0 con 0x7fe6d8101030 2024-04-05T23:46:30.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.926+0000 7fe6cf400700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe6bc0a1290 0x7fe6bc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.926+0000 7fe6f1280b80 1 Processor -- start 2024-04-05T23:46:30.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.926+0000 7fe6cf400700 1 -- 172.21.15.16:0/3893343506 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe6c80ce0b0 con 0x7fe6d8101030 2024-04-05T23:46:30.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.926+0000 7fe6d6000700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe6bc0a1290 0x7fe6bc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.926+0000 7fe6f1280b80 1 -- start start 2024-04-05T23:46:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 0x55bf96c94f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c8f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 0x55bf96c8fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96bf5b80 con 0x55bf96c954a0 2024-04-05T23:46:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96be6930 con 0x55bf96c8f5c0 2024-04-05T23:46:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96be42f0 con 0x55bf96c94b80 2024-04-05T23:46:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6d6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c8f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6d7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 0x55bf96c8fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6d6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c8f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59324/0 (socket says 172.21.15.16:59324) 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6d6000700 1 -- 172.21.15.16:0/2266435163 learned_addr learned my addr 172.21.15.16:0/2266435163 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/2266435163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 0x55bf96c94f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.927+0000 7fe6d6000700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe6bc0a1290 0x7fe6bc0a3750 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe6d819aad0 tx=0x7fe6c4011720 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6d6000700 1 -- 172.21.15.16:0/2266435163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 msgr2=0x55bf96c94f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6d6000700 1 --2- 172.21.15.16:0/2266435163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 0x55bf96c94f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6d6000700 1 -- 172.21.15.16:0/2266435163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 msgr2=0x55bf96c8fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6d6000700 1 --2- 172.21.15.16:0/2266435163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 0x55bf96c8fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6d6000700 1 -- 172.21.15.16:0/2266435163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf96a61350 con 0x55bf96c954a0 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/2266435163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 0x55bf96c94f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6d6000700 1 --2- 172.21.15.16:0/2266435163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c8f030 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005c50 tx=0x7fe6c401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb8a05d1d70af37 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6c3e00700 1 -- 172.21.15.16:0/2266435163 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c401ddb0 con 0x55bf96c954a0 2024-04-05T23:46:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6c3e00700 1 -- 172.21.15.16:0/2266435163 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6c4004cf0 con 0x55bf96c954a0 2024-04-05T23:46:30.921 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.928+0000 7fe6c3e00700 1 -- 172.21.15.16:0/2266435163 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c400f3c0 con 0x55bf96c954a0 2024-04-05T23:46:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.929+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2266435163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 msgr2=0x55bf96c8f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.929+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2266435163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c8f030 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005c50 tx=0x7fe6c401c000 comp rx=0 tx=0).stop 2024-04-05T23:46:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.929+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2266435163 shutdown_connections 2024-04-05T23:46:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.929+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2266435163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 0x55bf96c8fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.929+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2266435163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 0x55bf96c94f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.929+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2266435163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c8f030 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.929+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2266435163 >> 172.21.15.16:0/2266435163 conn(0x55bf96c031c0 msgr2=0x55bf96c89be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.929+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2266435163 shutdown_connections 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.929+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2266435163 wait complete. 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.930+0000 7fe6f1280b80 1 -- start start 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.930+0000 7fe6f1280b80 1 Processor -- start 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c8f5c0 0x55bf96c9b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 0x55bf96c98790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c954a0 0x55bf96c98da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96bdfbe0 con 0x55bf96c8f5c0 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96b7bd20 con 0x55bf96c954a0 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96b7b9c0 con 0x55bf96c94b80 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6f1280b80 10 client.0 _ll_register_callbacks cb 0x55bf969e7340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c8f5c0 0x55bf96c9b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c8f5c0 0x55bf96c9b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59328/0 (socket says 172.21.15.16:59328) 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d6a00700 1 -- 172.21.15.16:0/2035610802 learned_addr learned my addr 172.21.15.16:0/2035610802 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d6000700 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 0x55bf96c98790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d7400700 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c954a0 0x55bf96c98da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d6a00700 1 -- 172.21.15.16:0/2035610802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 msgr2=0x55bf96c98790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 0x55bf96c98790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d6a00700 1 -- 172.21.15.16:0/2035610802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c954a0 msgr2=0x55bf96c98da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c954a0 0x55bf96c98da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d6a00700 1 -- 172.21.15.16:0/2035610802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d81a6530 con 0x55bf96c8f5c0 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.931+0000 7fe6d7400700 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c954a0 0x55bf96c98da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c8f5c0 0x55bf96c9b410 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe6c809b160 tx=0x7fe6c8099780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45223323fc0f81eb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6c0c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2035610802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c809b630 con 0x55bf96c8f5c0 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2035610802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe6c809b1d0 con 0x55bf96c8f5c0 2024-04-05T23:46:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2035610802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bf96c821e0 con 0x55bf96c8f5c0 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2035610802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6c809f000 con 0x55bf96c8f5c0 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6f1280b80 10 client.16845 fetch_fsmap learned FSMap version 78 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2035610802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) v3 -- 0x7fe6c4011ca0 con 0x55bf96c8f5c0 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2035610802 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bf96c821e0 con 0x55bf96c8f5c0 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.932+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2035610802 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fe6c80a3130 con 0x55bf96c8f5c0 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.933+0000 7fe6f1280b80 10 client.16845 fetch_fsmap finished waiting for FSMap version 78 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.933+0000 7fe6f1280b80 10 client.16845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.933+0000 7fe6f1280b80 20 client.16845 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.933+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55bf96ca8bb0 0x55bf96caafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.933+0000 7fe6f1280b80 4 client.16845 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.933+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2035610802 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf96bd86a0 con 0x55bf96ca8bb0 2024-04-05T23:46:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.933+0000 7fe6d6000700 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55bf96ca8bb0 0x55bf96caafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.934+0000 7fe6d6000700 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55bf96ca8bb0 0x55bf96caafd0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=749053c78cb4c1 server_cookie=9d10b50673791408 in_seq=0 out_seq=0 2024-04-05T23:46:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.934+0000 7fe6c0c00700 10 client.16845 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.934+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2035610802 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bf96bd86a0 con 0x55bf96ca8bb0 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.934+0000 7fe6c0c00700 10 client.16845 handle_command_reply: tid=0 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.934+0000 7fe6f1280b80 1 client.16845 shutdown 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.934+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2035610802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c8f5c0 msgr2=0x55bf96c9b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.934+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c8f5c0 0x55bf96c9b410 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe6c809b160 tx=0x7fe6c8099780 comp rx=0 tx=0).stop 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.934+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2035610802 shutdown_connections 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.935+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55bf96ca8bb0 0x55bf96caafd0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.935+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c8f5c0 0x55bf96c9b410 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.935+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c954a0 0x55bf96c98da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.935+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2035610802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c94b80 0x55bf96c98790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.935+0000 7fe6c0c00700 0 client.16845 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.935+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2035610802 >> 172.21.15.16:0/2035610802 conn(0x55bf96c031c0 msgr2=0x55bf96c89280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.935+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2035610802 shutdown_connections 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.935+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2035610802 wait complete. 2024-04-05T23:46:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.935+0000 7fe6f1280b80 20 client.16845 trim_cache size 0 max 16384 2024-04-05T23:46:30.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.950+0000 7fe6f1280b80 1 Processor -- start 2024-04-05T23:46:30.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.950+0000 7fe6f1280b80 1 -- start start 2024-04-05T23:46:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c8f5c0 0x55bf96c8f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c94b80 0x55bf96c89a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c92940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96a63640 con 0x55bf96c954a0 2024-04-05T23:46:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96c8ac10 con 0x55bf96c94b80 2024-04-05T23:46:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96c93450 con 0x55bf96c8f5c0 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c92940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c92940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59338/0 (socket says 172.21.15.16:59338) 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d7400700 1 -- 172.21.15.16:0/1330715600 learned_addr learned my addr 172.21.15.16:0/1330715600 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d6000700 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c94b80 0x55bf96c89a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c8f5c0 0x55bf96c8f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d7400700 1 -- 172.21.15.16:0/1330715600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c8f5c0 msgr2=0x55bf96c8f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d7400700 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c8f5c0 0x55bf96c8f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d7400700 1 -- 172.21.15.16:0/1330715600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c94b80 msgr2=0x55bf96c89a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d7400700 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c94b80 0x55bf96c89a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d7400700 1 -- 172.21.15.16:0/1330715600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf96a0db00 con 0x55bf96c954a0 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.951+0000 7fe6d6000700 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c94b80 0x55bf96c89a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.952+0000 7fe6d7400700 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c92940 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0009de0 tx=0x7fe6d0009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfc922e75d0bd5c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.952+0000 7fe6ccc00700 1 -- 172.21.15.16:0/1330715600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0007e00 con 0x55bf96c954a0 2024-04-05T23:46:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.952+0000 7fe6ccc00700 1 -- 172.21.15.16:0/1330715600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6d000f040 con 0x55bf96c954a0 2024-04-05T23:46:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.952+0000 7fe6ccc00700 1 -- 172.21.15.16:0/1330715600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d001f940 con 0x55bf96c954a0 2024-04-05T23:46:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 -- 172.21.15.16:0/1330715600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 msgr2=0x55bf96c92940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c92940 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0009de0 tx=0x7fe6d0009e10 comp rx=0 tx=0).stop 2024-04-05T23:46:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 -- 172.21.15.16:0/1330715600 shutdown_connections 2024-04-05T23:46:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c954a0 0x55bf96c92940 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c8f5c0 0x55bf96c8f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/1330715600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c94b80 0x55bf96c89a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 -- 172.21.15.16:0/1330715600 >> 172.21.15.16:0/1330715600 conn(0x55bf96c031c0 msgr2=0x55bf96c985f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 -- 172.21.15.16:0/1330715600 shutdown_connections 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 -- 172.21.15.16:0/1330715600 wait complete. 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 -- start start 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.953+0000 7fe6f1280b80 1 Processor -- start 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 0x55bf96c91d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c94b80 0x55bf96c90720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6f1280b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c954a0 0x55bf96c90c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96c923e0 con 0x55bf96c94b80 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96c911a0 con 0x55bf96c8f5c0 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6f1280b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf96c91360 con 0x55bf96c954a0 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6f1280b80 10 client.0 _ll_register_callbacks cb 0x7fe6c809afa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6d6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c94b80 0x55bf96c90720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6d6000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c94b80 0x55bf96c90720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59346/0 (socket says 172.21.15.16:59346) 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.954+0000 7fe6d6000700 1 -- 172.21.15.16:0/2907428992 learned_addr learned my addr 172.21.15.16:0/2907428992 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 0x55bf96c91d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6d7400700 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c954a0 0x55bf96c90c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6d6000700 1 -- 172.21.15.16:0/2907428992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c954a0 msgr2=0x55bf96c90c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6d6000700 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c954a0 0x55bf96c90c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6d6000700 1 -- 172.21.15.16:0/2907428992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 msgr2=0x55bf96c91d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6d6000700 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 0x55bf96c91d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6d6000700 1 -- 172.21.15.16:0/2907428992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf96a61350 con 0x55bf96c94b80 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6d6000700 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c94b80 0x55bf96c90720 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005330 tx=0x7fe6c4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c9b236007fd2bf server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6c0c00700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.955+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2907428992 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c402f360 con 0x55bf96c94b80 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2907428992 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fe6c402b040 con 0x55bf96c94b80 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2907428992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6c4004cf0 con 0x55bf96c94b80 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2907428992 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6c402f4c0 con 0x55bf96c94b80 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6f1280b80 10 client.16851 fetch_fsmap learned FSMap version 78 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2907428992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) v3 -- 0x7fe6d0009a90 con 0x55bf96c94b80 2024-04-05T23:46:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2907428992 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6c4004cf0 con 0x55bf96c94b80 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2907428992 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fe6c40048b0 con 0x55bf96c94b80 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6f1280b80 10 client.16851 fetch_fsmap finished waiting for FSMap version 78 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6f1280b80 10 client.16851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6f1280b80 20 client.16851 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55bf96cfdbe0 0x55bf96d00000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6f1280b80 4 client.16851 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.956+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2907428992 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bf96a2ec90 con 0x55bf96cfdbe0 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.957+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55bf96cfdbe0 0x55bf96d00000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.957+0000 7fe6d6a00700 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55bf96cfdbe0 0x55bf96d00000 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c08d3bfb6e881d8d server_cookie=fda017363c2b6f8c in_seq=0 out_seq=0 2024-04-05T23:46:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.957+0000 7fe6c0c00700 10 client.16851 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6c0c00700 1 -- 172.21.15.16:0/2907428992 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bf96a2ec90 con 0x55bf96cfdbe0 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6c0c00700 10 client.16851 handle_command_reply: tid=0 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 client.16851 shutdown 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2907428992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c94b80 msgr2=0x55bf96c90720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c94b80 0x55bf96c90720 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005330 tx=0x7fe6c4005dc0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2907428992 shutdown_connections 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x55bf96cfdbe0 0x55bf96d00000 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf96c954a0 0x55bf96c90c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bf96c8f5c0 0x55bf96c91d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 --2- 172.21.15.16:0/2907428992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf96c94b80 0x55bf96c90720 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6c0c00700 0 client.16851 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2907428992 >> 172.21.15.16:0/2907428992 conn(0x55bf96c031c0 msgr2=0x55bf96c985a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2907428992 shutdown_connections 2024-04-05T23:46:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 1 -- 172.21.15.16:0/2907428992 wait complete. 2024-04-05T23:46:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.993+0000 7fe6f1280b80 20 client.16851 trim_cache size 0 max 16384 2024-04-05T23:46:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:46:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:46:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:46:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:46:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 -- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe6bc0a1290 msgr2=0x7fe6bc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe6bc0a1290 0x7fe6bc0a3750 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe6d819aad0 tx=0x7fe6c4011720 comp rx=0 tx=0).stop 2024-04-05T23:46:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 -- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 msgr2=0x7fe6d8199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 0x7fe6d8199580 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8000c00 tx=0x7fe6c8005740 comp rx=0 tx=0).stop 2024-04-05T23:46:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 -- 172.21.15.16:0/3893343506 shutdown_connections 2024-04-05T23:46:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fe6bc0a1290 0x7fe6bc0a3750 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d8101030 0x7fe6d8199580 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6d819de60 0x7fe6d819e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 --2- 172.21.15.16:0/3893343506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d8106150 0x7fe6d8199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 -- 172.21.15.16:0/3893343506 >> 172.21.15.16:0/3893343506 conn(0x7fe6d80fc8b0 msgr2=0x7fe6d80fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 -- 172.21.15.16:0/3893343506 shutdown_connections 2024-04-05T23:46:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:30.994+0000 7fe6dd800700 1 -- 172.21.15.16:0/3893343506 wait complete. 2024-04-05T23:46:31.040 INFO:tasks.workunit.client.0.smithi016.stdout: 1 616129 4.56 MB/sec execute 590 sec latency 2803.371 ms 2024-04-05T23:46:32.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 616129 4.55 MB/sec execute 591 sec latency 3803.505 ms 2024-04-05T23:46:32.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:31 smithi027 ceph-mon[78597]: pgmap v408: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-05T23:46:32.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:31 smithi016 ceph-mon[38248]: pgmap v408: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-05T23:46:32.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:31 smithi130 ceph-mon[79417]: pgmap v408: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-05T23:46:33.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 617741 4.56 MB/sec execute 592 sec latency 3985.672 ms 2024-04-05T23:46:34.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 618774 4.56 MB/sec execute 593 sec latency 15.319 ms 2024-04-05T23:46:34.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:33 smithi027 ceph-mon[78597]: pgmap v409: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1018 KiB/s wr, 101 op/s 2024-04-05T23:46:34.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:33 smithi016 ceph-mon[38248]: pgmap v409: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1018 KiB/s wr, 101 op/s 2024-04-05T23:46:34.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:33 smithi130 ceph-mon[79417]: pgmap v409: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1018 KiB/s wr, 101 op/s 2024-04-05T23:46:35.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 619675 4.56 MB/sec execute 594 sec latency 415.257 ms 2024-04-05T23:46:36.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 619675 4.55 MB/sec execute 595 sec latency 1415.326 ms 2024-04-05T23:46:36.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:35 smithi027 ceph-mon[78597]: pgmap v410: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1018 KiB/s wr, 101 op/s 2024-04-05T23:46:36.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:35 smithi016 ceph-mon[38248]: pgmap v410: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1018 KiB/s wr, 101 op/s 2024-04-05T23:46:36.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:35 smithi130 ceph-mon[79417]: pgmap v410: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1018 KiB/s wr, 101 op/s 2024-04-05T23:46:37.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 619675 4.54 MB/sec execute 596 sec latency 2415.444 ms 2024-04-05T23:46:38.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 621193 4.55 MB/sec execute 597 sec latency 2600.016 ms 2024-04-05T23:46:38.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:37 smithi027 ceph-mon[78597]: pgmap v411: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.8 MiB/s wr, 168 op/s 2024-04-05T23:46:38.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:37 smithi016 ceph-mon[38248]: pgmap v411: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.8 MiB/s wr, 168 op/s 2024-04-05T23:46:38.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:37 smithi130 ceph-mon[79417]: pgmap v411: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.8 MiB/s wr, 168 op/s 2024-04-05T23:46:39.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 622308 4.55 MB/sec execute 598 sec latency 17.258 ms 2024-04-05T23:46:40.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 623219 4.55 MB/sec execute 599 sec latency 376.332 ms 2024-04-05T23:46:40.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:39 smithi027 ceph-mon[78597]: pgmap v412: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-05T23:46:40.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:39 smithi016 ceph-mon[38248]: pgmap v412: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-05T23:46:40.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:39 smithi130 ceph-mon[79417]: pgmap v412: 97 pgs: 97 active+clean; 255 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-05T23:46:41.041 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 600 sec 2024-04-05T23:46:41.285 INFO:tasks.workunit.client.0.smithi016.stdout: 0 cleanup 600 sec 2024-04-05T23:46:41.285 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-04-05T23:46:41.285 INFO:tasks.workunit.client.0.smithi016.stdout: Operation Count AvgLat MaxLat 2024-04-05T23:46:41.285 INFO:tasks.workunit.client.0.smithi016.stdout: ---------------------------------------- 2024-04-05T23:46:41.285 INFO:tasks.workunit.client.0.smithi016.stdout: NTCreateX 87239 1.947 7623.021 2024-04-05T23:46:41.285 INFO:tasks.workunit.client.0.smithi016.stdout: Close 64084 0.004 0.229 2024-04-05T23:46:41.285 INFO:tasks.workunit.client.0.smithi016.stdout: Rename 3694 62.715 4931.597 2024-04-05T23:46:41.285 INFO:tasks.workunit.client.0.smithi016.stdout: Unlink 17619 2.853 13408.297 2024-04-05T23:46:41.285 INFO:tasks.workunit.client.0.smithi016.stdout: Deltree 2 84.401 121.279 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: Mkdir 1 0.016 0.016 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: Qpathinfo 79073 1.299 4895.935 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: Qfileinfo 13850 0.002 0.053 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: Qfsinfo 14510 0.243 20.657 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: Sfileinfo 7104 0.035 6.639 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: Find 30568 0.495 7003.495 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: WriteX 43452 0.106 83.980 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: ReadX 136829 0.014 26.782 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: LockX 284 0.657 2.011 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: UnlockX 284 0.622 1.975 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: Flush 6112 2.868 62.994 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-04-05T23:46:41.286 INFO:tasks.workunit.client.0.smithi016.stdout:Throughput 4.55299 MB/sec 1 clients 1 procs max_latency=13408.314 ms 2024-04-05T23:46:41.287 INFO:tasks.workunit.client.0.smithi016.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-05T23:46:41.287 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-04-05T23:46:41.287 INFO:tasks.workunit.client.0.smithi016.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-05T23:46:41.289 INFO:tasks.workunit.client.0.smithi016.stdout:9 of 10 processes prepared for launch 0 sec 2024-04-05T23:46:41.389 INFO:tasks.workunit.client.0.smithi016.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-05T23:46:41.389 INFO:tasks.workunit.client.0.smithi016.stdout:releasing clients 2024-04-05T23:46:42.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:41 smithi027 ceph-mon[78597]: pgmap v413: 97 pgs: 97 active+clean; 259 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-04-05T23:46:42.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:41 smithi016 ceph-mon[38248]: pgmap v413: 97 pgs: 97 active+clean; 259 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-04-05T23:46:42.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:41 smithi130 ceph-mon[79417]: pgmap v413: 97 pgs: 97 active+clean; 259 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.1 MiB/s wr, 188 op/s 2024-04-05T23:46:42.389 INFO:tasks.workunit.client.0.smithi016.stdout: 10 394 140.07 MB/sec warmup 1 sec latency 56.138 ms 2024-04-05T23:46:43.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:42 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:43.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:42 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:43.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:42 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:43.389 INFO:tasks.workunit.client.0.smithi016.stdout: 10 848 104.95 MB/sec warmup 2 sec latency 105.237 ms 2024-04-05T23:46:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.164+0000 7f190f800700 1 -- 172.21.15.16:0/3322627049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19081019e0 msgr2=0x7f1908105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.164+0000 7f190f800700 1 --2- 172.21.15.16:0/3322627049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19081019e0 0x7f1908105a20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009990 tx=0x7f18f4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.164+0000 7f190f800700 1 -- 172.21.15.16:0/3322627049 shutdown_connections 2024-04-05T23:46:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.164+0000 7f190f800700 1 --2- 172.21.15.16:0/3322627049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1908106150 0x7f190810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.164+0000 7f190f800700 1 --2- 172.21.15.16:0/3322627049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908101030 0x7f1908101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.164+0000 7f190f800700 1 --2- 172.21.15.16:0/3322627049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19081019e0 0x7f1908105a20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.164+0000 7f190f800700 1 -- 172.21.15.16:0/3322627049 >> 172.21.15.16:0/3322627049 conn(0x7f19080fc8b0 msgr2=0x7f19080fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.164+0000 7f190f800700 1 -- 172.21.15.16:0/3322627049 shutdown_connections 2024-04-05T23:46:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.164+0000 7f190f800700 1 -- 172.21.15.16:0/3322627049 wait complete. 2024-04-05T23:46:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190f800700 1 Processor -- start 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190f800700 1 -- start start 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190f800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908101030 0x7f1908197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190f800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908106150 0x7f1908197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190f800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190819bc10 0x7f190819c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190f800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190819c690 con 0x7f1908106150 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190f800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190819c800 con 0x7f190819bc10 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190f800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190819c9c0 con 0x7f1908101030 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f1907e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908106150 0x7f1908197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190ce00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908101030 0x7f1908197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f1907e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908106150 0x7f1908197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52270/0 (socket says 172.21.15.16:52270) 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f1907e00700 1 -- 172.21.15.16:0/3873829215 learned_addr learned my addr 172.21.15.16:0/3873829215 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.166+0000 7f190d800700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190819bc10 0x7f190819c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f1907e00700 1 -- 172.21.15.16:0/3873829215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908101030 msgr2=0x7f1908197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f1907e00700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908101030 0x7f1908197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f1907e00700 1 -- 172.21.15.16:0/3873829215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190819bc10 msgr2=0x7f190819c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f1907e00700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190819bc10 0x7f190819c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f1907e00700 1 -- 172.21.15.16:0/3873829215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f40095f0 con 0x7f1908106150 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f190ce00700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908101030 0x7f1908197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f1907e00700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908106150 0x7f1908197870 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f18f4000c00 tx=0x7f18f4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f1905600700 1 -- 172.21.15.16:0/3873829215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f401d070 con 0x7f1908106150 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f1905600700 1 -- 172.21.15.16:0/3873829215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18f4022470 con 0x7f1908106150 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f190f800700 1 -- 172.21.15.16:0/3873829215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1908073110 con 0x7f1908106150 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.167+0000 7f1905600700 1 -- 172.21.15.16:0/3873829215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4017650 con 0x7f1908106150 2024-04-05T23:46:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.168+0000 7f190f800700 1 -- 172.21.15.16:0/3873829215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19080735c0 con 0x7f1908106150 2024-04-05T23:46:44.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.171+0000 7f1905600700 1 -- 172.21.15.16:0/3873829215 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f18f400fad0 con 0x7f1908106150 2024-04-05T23:46:44.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.172+0000 7f1905600700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f18f00a1290 0x7f18f00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.172+0000 7f1905600700 1 -- 172.21.15.16:0/3873829215 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f18f40cdfe0 con 0x7f1908106150 2024-04-05T23:46:44.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.172+0000 7f190ce00700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f18f00a1290 0x7f18f00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.173+0000 7f19233c4b80 1 Processor -- start 2024-04-05T23:46:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.173+0000 7f190ce00700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f18f00a1290 0x7f18f00a3750 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009730 tx=0x7f18fc012040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:44.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.173+0000 7f19233c4b80 1 -- start start 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.173+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb187f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.173+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb182030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 0x5578cb182a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f19233c4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb0e8b80 con 0x5578cb1825c0 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f19233c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb0d9930 con 0x5578cb187b80 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f19233c4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb0d72f0 con 0x5578cb1884a0 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f190ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb187f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f190ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb187f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33574/0 (socket says 172.21.15.16:33574) 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f190ce00700 1 -- 172.21.15.16:0/3609788374 learned_addr learned my addr 172.21.15.16:0/3609788374 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f190d800700 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 0x5578cb182a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f1907e00700 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb182030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f190ce00700 1 -- 172.21.15.16:0/3609788374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 msgr2=0x5578cb182030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f190ce00700 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb182030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f190ce00700 1 -- 172.21.15.16:0/3609788374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 msgr2=0x5578cb182a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f190ce00700 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 0x5578cb182a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f190ce00700 1 -- 172.21.15.16:0/3609788374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578caf54350 con 0x5578cb187b80 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.174+0000 7f1907e00700 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb182030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.175+0000 7f190ce00700 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb187f60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f18fc004500 tx=0x7f18fc018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=844b251fb1ba2cd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.175+0000 7f18fa000700 1 -- 172.21.15.16:0/3609788374 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc023070 con 0x5578cb187b80 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.175+0000 7f18fa000700 1 -- 172.21.15.16:0/3609788374 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18fc016040 con 0x5578cb187b80 2024-04-05T23:46:44.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.175+0000 7f18fa000700 1 -- 172.21.15.16:0/3609788374 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc01c3f0 con 0x5578cb187b80 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 -- 172.21.15.16:0/3609788374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 msgr2=0x5578cb187f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb187f60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f18fc004500 tx=0x7f18fc018000 comp rx=0 tx=0).stop 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 -- 172.21.15.16:0/3609788374 shutdown_connections 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 0x5578cb182a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb187f60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 --2- 172.21.15.16:0/3609788374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb182030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 -- 172.21.15.16:0/3609788374 >> 172.21.15.16:0/3609788374 conn(0x5578cb0f61c0 msgr2=0x5578cb17cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 -- 172.21.15.16:0/3609788374 shutdown_connections 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 -- 172.21.15.16:0/3609788374 wait complete. 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.176+0000 7f19233c4b80 1 -- start start 2024-04-05T23:46:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f19233c4b80 1 Processor -- start 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb1825c0 0x5578cb18e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb187b80 0x5578cb18b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 0x5578cb18bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f19233c4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb0d2be0 con 0x5578cb1884a0 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f19233c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb06ed20 con 0x5578cb1825c0 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f19233c4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb06e9c0 con 0x5578cb187b80 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f19233c4b80 10 client.0 _ll_register_callbacks cb 0x5578caeda340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f190ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb1825c0 0x5578cb18e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f190ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb1825c0 0x5578cb18e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33588/0 (socket says 172.21.15.16:33588) 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f190ce00700 1 -- 172.21.15.16:0/2368498059 learned_addr learned my addr 172.21.15.16:0/2368498059 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f1907e00700 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb187b80 0x5578cb18b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.177+0000 7f190d800700 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 0x5578cb18bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f190ce00700 1 -- 172.21.15.16:0/2368498059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb187b80 msgr2=0x5578cb18b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f190ce00700 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb187b80 0x5578cb18b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f190ce00700 1 -- 172.21.15.16:0/2368498059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 msgr2=0x5578cb18bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f190ce00700 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 0x5578cb18bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f190ce00700 1 -- 172.21.15.16:0/2368498059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18fc004200 con 0x5578cb1825c0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f190ce00700 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb1825c0 0x5578cb18e540 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0044d0 tx=0x7f18fc005950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=681e57191f58ab45 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f18eea00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f18eea00700 1 -- 172.21.15.16:0/2368498059 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc023070 con 0x5578cb1825c0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f19233c4b80 1 -- 172.21.15.16:0/2368498059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5578cb1751e0 con 0x5578cb1825c0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f18eea00700 1 -- 172.21.15.16:0/2368498059 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f18fc027610 con 0x5578cb1825c0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.178+0000 7f18eea00700 1 -- 172.21.15.16:0/2368498059 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18fc0277d0 con 0x5578cb1825c0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f19233c4b80 10 client.26906 fetch_fsmap learned FSMap version 80 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f19233c4b80 1 -- 172.21.15.16:0/2368498059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) v3 -- 0x5578caf00b00 con 0x5578cb1825c0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f18eea00700 1 -- 172.21.15.16:0/2368498059 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5578cb1751e0 con 0x5578cb1825c0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f18eea00700 1 -- 172.21.15.16:0/2368498059 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 80) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f18fc01cb90 con 0x5578cb1825c0 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f19233c4b80 10 client.26906 fetch_fsmap finished waiting for FSMap version 80 2024-04-05T23:46:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f19233c4b80 10 client.26906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f19233c4b80 20 client.26906 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5578cb19bf30 0x5578cb19e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f19233c4b80 4 client.26906 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:46:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f19233c4b80 1 -- 172.21.15.16:0/2368498059 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578cb0cb6a0 con 0x5578cb19bf30 2024-04-05T23:46:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.179+0000 7f1907e00700 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5578cb19bf30 0x5578cb19e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.180+0000 7f1907e00700 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5578cb19bf30 0x5578cb19e350 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=949ecc90b897b8e8 server_cookie=67d804a58ed3bb1a in_seq=0 out_seq=0 2024-04-05T23:46:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.180+0000 7f18eea00700 10 client.26906 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f18eea00700 1 -- 172.21.15.16:0/2368498059 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5578cb0cb6a0 con 0x5578cb19bf30 2024-04-05T23:46:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f18eea00700 10 client.26906 handle_command_reply: tid=0 2024-04-05T23:46:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f19233c4b80 1 client.26906 shutdown 2024-04-05T23:46:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f19233c4b80 1 -- 172.21.15.16:0/2368498059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb1825c0 msgr2=0x5578cb18e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb1825c0 0x5578cb18e540 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0044d0 tx=0x7f18fc005950 comp rx=0 tx=0).stop 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f19233c4b80 1 -- 172.21.15.16:0/2368498059 shutdown_connections 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5578cb19bf30 0x5578cb19e350 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 0x5578cb18bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb1825c0 0x5578cb18e540 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f18eea00700 0 client.26906 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2368498059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb187b80 0x5578cb18b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.182+0000 7f19233c4b80 1 -- 172.21.15.16:0/2368498059 >> 172.21.15.16:0/2368498059 conn(0x5578cb0f61c0 msgr2=0x5578cb17c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.183+0000 7f19233c4b80 1 -- 172.21.15.16:0/2368498059 shutdown_connections 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.183+0000 7f19233c4b80 1 -- 172.21.15.16:0/2368498059 wait complete. 2024-04-05T23:46:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.183+0000 7f19233c4b80 20 client.26906 trim_cache size 0 max 16384 2024-04-05T23:46:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.200+0000 7f19233c4b80 1 Processor -- start 2024-04-05T23:46:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.200+0000 7f19233c4b80 1 -- start start 2024-04-05T23:46:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1825c0 0x5578cb19e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb192f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 0x5578cb1829a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f19233c4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578caf56640 con 0x5578cb1884a0 2024-04-05T23:46:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f19233c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb189e50 con 0x5578cb187b80 2024-04-05T23:46:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f19233c4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb186550 con 0x5578cb1825c0 2024-04-05T23:46:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f1907e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb192f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f1907e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb192f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33596/0 (socket says 172.21.15.16:33596) 2024-04-05T23:46:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f1907e00700 1 -- 172.21.15.16:0/3553926505 learned_addr learned my addr 172.21.15.16:0/3553926505 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f190ce00700 1 --2- 172.21.15.16:0/3553926505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1825c0 0x5578cb19e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f190d800700 1 --2- 172.21.15.16:0/3553926505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 0x5578cb1829a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f1907e00700 1 -- 172.21.15.16:0/3553926505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1825c0 msgr2=0x5578cb19e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f1907e00700 1 --2- 172.21.15.16:0/3553926505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1825c0 0x5578cb19e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f1907e00700 1 -- 172.21.15.16:0/3553926505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 msgr2=0x5578cb1829a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f1907e00700 1 --2- 172.21.15.16:0/3553926505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 0x5578cb1829a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.201+0000 7f1907e00700 1 -- 172.21.15.16:0/3553926505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578cb098e40 con 0x5578cb187b80 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.202+0000 7f1907e00700 1 --2- 172.21.15.16:0/3553926505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb192f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f18f4097b90 tx=0x7f18f4009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c04bfb36f786d6fd server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.202+0000 7f18faa00700 1 -- 172.21.15.16:0/3553926505 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f409d040 con 0x5578cb187b80 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.202+0000 7f18faa00700 1 -- 172.21.15.16:0/3553926505 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18f4063070 con 0x5578cb187b80 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.202+0000 7f18faa00700 1 -- 172.21.15.16:0/3553926505 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f400fd80 con 0x5578cb187b80 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 -- 172.21.15.16:0/3553926505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 msgr2=0x5578cb192f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 --2- 172.21.15.16:0/3553926505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb192f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f18f4097b90 tx=0x7f18f4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 -- 172.21.15.16:0/3553926505 shutdown_connections 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 --2- 172.21.15.16:0/3553926505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1884a0 0x5578cb1829a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 --2- 172.21.15.16:0/3553926505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1825c0 0x5578cb19e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 --2- 172.21.15.16:0/3553926505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb192f50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 -- 172.21.15.16:0/3553926505 >> 172.21.15.16:0/3553926505 conn(0x5578cb0f61c0 msgr2=0x5578cb1a2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 -- 172.21.15.16:0/3553926505 shutdown_connections 2024-04-05T23:46:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 -- 172.21.15.16:0/3553926505 wait complete. 2024-04-05T23:46:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.203+0000 7f19233c4b80 1 -- start start 2024-04-05T23:46:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.204+0000 7f19233c4b80 1 Processor -- start 2024-04-05T23:46:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.204+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 0x5578cb18ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.204+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb18f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.204+0000 7f19233c4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb1eed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.204+0000 7f19233c4b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb18fdb0 con 0x5578cb1825c0 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.204+0000 7f19233c4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb1ef250 con 0x5578cb187b80 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.204+0000 7f19233c4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578cb1ef390 con 0x5578cb1884a0 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f19233c4b80 10 client.0 _ll_register_callbacks cb 0x7f1900009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f1907e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb18f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f1907e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb18f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:33612/0 (socket says 172.21.15.16:33612) 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f1907e00700 1 -- 172.21.15.16:0/2240313451 learned_addr learned my addr 172.21.15.16:0/2240313451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f190d800700 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb1eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f190ce00700 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 0x5578cb18ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f1907e00700 1 -- 172.21.15.16:0/2240313451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 msgr2=0x5578cb1eed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f1907e00700 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb1eed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f1907e00700 1 -- 172.21.15.16:0/2240313451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 msgr2=0x5578cb18ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f1907e00700 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 0x5578cb18ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f1907e00700 1 -- 172.21.15.16:0/2240313451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f409fe70 con 0x5578cb187b80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f190d800700 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb1eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f1907e00700 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb18f470 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f18f4097620 tx=0x7f18f409b120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eccdcad7c25565e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f18eea00700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.205+0000 7f18eea00700 1 -- 172.21.15.16:0/2240313451 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f409d050 con 0x5578cb187b80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f18eea00700 1 -- 172.21.15.16:0/2240313451 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f18f40a3510 con 0x5578cb187b80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f19233c4b80 1 -- 172.21.15.16:0/2240313451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18fc016040 con 0x5578cb187b80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f18eea00700 1 -- 172.21.15.16:0/2240313451 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18f409ef50 con 0x5578cb187b80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f19233c4b80 10 client.26912 fetch_fsmap learned FSMap version 80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f19233c4b80 1 -- 172.21.15.16:0/2240313451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) v3 -- 0x5578caf58c70 con 0x5578cb187b80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f18eea00700 1 -- 172.21.15.16:0/2240313451 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18fc016040 con 0x5578cb187b80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f18eea00700 1 -- 172.21.15.16:0/2240313451 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 80) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f18f40b92a0 con 0x5578cb187b80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f19233c4b80 10 client.26912 fetch_fsmap finished waiting for FSMap version 80 2024-04-05T23:46:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f19233c4b80 10 client.26912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f19233c4b80 20 client.26912 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5578cb1f0720 0x5578cb1f2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f19233c4b80 4 client.26912 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:46:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.206+0000 7f19233c4b80 1 -- 172.21.15.16:0/2240313451 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5578caf21c90 con 0x5578cb1f0720 2024-04-05T23:46:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.207+0000 7f190ce00700 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5578cb1f0720 0x5578cb1f2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.207+0000 7f190ce00700 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5578cb1f0720 0x5578cb1f2b40 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7295b7cfbcbfa9 server_cookie=8bcbd8f338f36a2e in_seq=0 out_seq=0 2024-04-05T23:46:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.207+0000 7f18eea00700 10 client.26912 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:43 smithi027 ceph-mon[78597]: pgmap v414: 97 pgs: 97 active+clean; 259 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-05T23:46:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:43 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:43 smithi016 ceph-mon[38248]: pgmap v414: 97 pgs: 97 active+clean; 259 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-05T23:46:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:43 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:44.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:43 smithi130 ceph-mon[79417]: pgmap v414: 97 pgs: 97 active+clean; 259 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-05T23:46:44.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:43 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:44.389 INFO:tasks.workunit.client.0.smithi016.stdout: 10 1371 83.08 MB/sec warmup 3 sec latency 1081.302 ms 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f18eea00700 1 -- 172.21.15.16:0/2240313451 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5578caf21c90 con 0x5578cb1f0720 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f18eea00700 10 client.26912 handle_command_reply: tid=0 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f19233c4b80 1 client.26912 shutdown 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f19233c4b80 1 -- 172.21.15.16:0/2240313451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 msgr2=0x5578cb18f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb18f470 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f18f4097620 tx=0x7f18f409b120 comp rx=0 tx=0).stop 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f19233c4b80 1 -- 172.21.15.16:0/2240313451 shutdown_connections 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x5578cb1f0720 0x5578cb1f2b40 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5578cb1825c0 0x5578cb18ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578cb1884a0 0x5578cb1eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f19233c4b80 1 --2- 172.21.15.16:0/2240313451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5578cb187b80 0x5578cb18f470 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f18eea00700 0 client.26912 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.521+0000 7f19233c4b80 1 -- 172.21.15.16:0/2240313451 >> 172.21.15.16:0/2240313451 conn(0x5578cb0f61c0 msgr2=0x5578cb1829a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.522+0000 7f19233c4b80 1 -- 172.21.15.16:0/2240313451 shutdown_connections 2024-04-05T23:46:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.522+0000 7f19233c4b80 1 -- 172.21.15.16:0/2240313451 wait complete. 2024-04-05T23:46:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.522+0000 7f19233c4b80 20 client.26912 trim_cache size 0 max 16384 2024-04-05T23:46:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:46:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:46:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:46:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:46:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 -- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f18f00a1290 msgr2=0x7f18f00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f18f00a1290 0x7f18f00a3750 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009730 tx=0x7f18fc012040 comp rx=0 tx=0).stop 2024-04-05T23:46:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 -- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908106150 msgr2=0x7f1908197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908106150 0x7f1908197870 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f18f4000c00 tx=0x7f18f4005740 comp rx=0 tx=0).stop 2024-04-05T23:46:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 -- 172.21.15.16:0/3873829215 shutdown_connections 2024-04-05T23:46:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f18f00a1290 0x7f18f00a3750 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190819bc10 0x7f190819c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908101030 0x7f1908197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 --2- 172.21.15.16:0/3873829215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1908106150 0x7f1908197870 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 -- 172.21.15.16:0/3873829215 >> 172.21.15.16:0/3873829215 conn(0x7f19080fc8b0 msgr2=0x7f19080fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 -- 172.21.15.16:0/3873829215 shutdown_connections 2024-04-05T23:46:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:44.523+0000 7f190f800700 1 -- 172.21.15.16:0/3873829215 wait complete. 2024-04-05T23:46:45.389 INFO:tasks.workunit.client.0.smithi016.stdout: 10 2196 68.09 MB/sec warmup 4 sec latency 1145.729 ms 2024-04-05T23:46:46.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:45 smithi027 ceph-mon[78597]: pgmap v415: 97 pgs: 97 active+clean; 259 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-05T23:46:46.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:45 smithi016 ceph-mon[38248]: pgmap v415: 97 pgs: 97 active+clean; 259 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-05T23:46:46.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:45 smithi130 ceph-mon[79417]: pgmap v415: 97 pgs: 97 active+clean; 259 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 127 op/s 2024-04-05T23:46:46.389 INFO:tasks.workunit.client.0.smithi016.stdout: 10 2746 64.46 MB/sec warmup 5 sec latency 57.581 ms 2024-04-05T23:46:47.214 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:46 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:47.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:46 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:47.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:46 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:47.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3392 62.17 MB/sec warmup 6 sec latency 134.609 ms 2024-04-05T23:46:48.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:47 smithi027 ceph-mon[78597]: pgmap v416: 97 pgs: 97 active+clean; 263 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 420 op/s 2024-04-05T23:46:48.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:47 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:48.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:47 smithi016 ceph-mon[38248]: pgmap v416: 97 pgs: 97 active+clean; 263 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 420 op/s 2024-04-05T23:46:48.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:47 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:48.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:47 smithi130 ceph-mon[79417]: pgmap v416: 97 pgs: 97 active+clean; 263 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 420 op/s 2024-04-05T23:46:48.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:47 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:48.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3635 54.87 MB/sec warmup 7 sec latency 106.071 ms 2024-04-05T23:46:49.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3808 48.36 MB/sec warmup 8 sec latency 932.782 ms 2024-04-05T23:46:50.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:49 smithi027 ceph-mon[78597]: pgmap v417: 97 pgs: 97 active+clean; 263 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 2.2 MiB/s wr, 352 op/s 2024-04-05T23:46:50.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:49 smithi016 ceph-mon[38248]: pgmap v417: 97 pgs: 97 active+clean; 263 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 2.2 MiB/s wr, 352 op/s 2024-04-05T23:46:50.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:49 smithi130 ceph-mon[79417]: pgmap v417: 97 pgs: 97 active+clean; 263 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 2.2 MiB/s wr, 352 op/s 2024-04-05T23:46:50.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4022 43.26 MB/sec warmup 9 sec latency 1932.825 ms 2024-04-05T23:46:51.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4488 40.60 MB/sec warmup 10 sec latency 2932.876 ms 2024-04-05T23:46:52.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:51 smithi027 ceph-mon[78597]: pgmap v418: 97 pgs: 97 active+clean; 277 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 474 op/s 2024-04-05T23:46:52.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:51 smithi027 ceph-mon[78597]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:52.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:51 smithi016 ceph-mon[38248]: pgmap v418: 97 pgs: 97 active+clean; 277 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 474 op/s 2024-04-05T23:46:52.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:51 smithi016 ceph-mon[38248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:52.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:51 smithi130 ceph-mon[79417]: pgmap v418: 97 pgs: 97 active+clean; 277 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 474 op/s 2024-04-05T23:46:52.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:51 smithi130 ceph-mon[79417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-05T23:46:52.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 5317 39.96 MB/sec warmup 11 sec latency 3777.163 ms 2024-04-05T23:46:53.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 5929 40.65 MB/sec warmup 12 sec latency 91.417 ms 2024-04-05T23:46:54.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:53 smithi027 ceph-mon[78597]: pgmap v419: 97 pgs: 97 active+clean; 277 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 414 op/s 2024-04-05T23:46:54.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:53 smithi016 ceph-mon[38248]: pgmap v419: 97 pgs: 97 active+clean; 277 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 414 op/s 2024-04-05T23:46:54.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:53 smithi130 ceph-mon[79417]: pgmap v419: 97 pgs: 97 active+clean; 277 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 414 op/s 2024-04-05T23:46:54.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6524 40.57 MB/sec warmup 13 sec latency 60.339 ms 2024-04-05T23:46:55.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6919 40.24 MB/sec warmup 14 sec latency 63.068 ms 2024-04-05T23:46:56.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:55 smithi027 ceph-mon[78597]: pgmap v420: 97 pgs: 97 active+clean; 277 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 414 op/s 2024-04-05T23:46:56.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:55 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:46:56.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:55 smithi016 ceph-mon[38248]: pgmap v420: 97 pgs: 97 active+clean; 277 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 414 op/s 2024-04-05T23:46:56.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:55 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:46:56.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:55 smithi130 ceph-mon[79417]: pgmap v420: 97 pgs: 97 active+clean; 277 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 414 op/s 2024-04-05T23:46:56.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:55 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T23:46:56.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7107 37.87 MB/sec warmup 15 sec latency 66.412 ms 2024-04-05T23:46:57.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7301 35.73 MB/sec warmup 16 sec latency 179.412 ms 2024-04-05T23:46:57.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:57 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:46:57.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:57 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:46:57.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:57 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:46:57.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:57 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:46:57.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:57 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:46:57.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:57 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:46:57.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:57 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:46:57.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:57 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:46:57.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:57 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:46:57.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:57 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:46:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.699+0000 7f3730e00700 1 -- 172.21.15.16:0/3404545449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c101840 msgr2=0x7f372c105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.699+0000 7f3730e00700 1 --2- 172.21.15.16:0/3404545449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c101840 0x7f372c105880 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3714009990 tx=0x7f3714009ca0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.699+0000 7f3730e00700 1 -- 172.21.15.16:0/3404545449 shutdown_connections 2024-04-05T23:46:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.699+0000 7f3730e00700 1 --2- 172.21.15.16:0/3404545449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f372c100e90 0x7f372c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.699+0000 7f3730e00700 1 --2- 172.21.15.16:0/3404545449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c105fb0 0x7f372c10db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.699+0000 7f3730e00700 1 --2- 172.21.15.16:0/3404545449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c101840 0x7f372c105880 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.699+0000 7f3730e00700 1 -- 172.21.15.16:0/3404545449 >> 172.21.15.16:0/3404545449 conn(0x7f372c0fc770 msgr2=0x7f372c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.700+0000 7f3730e00700 1 -- 172.21.15.16:0/3404545449 shutdown_connections 2024-04-05T23:46:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.700+0000 7f3730e00700 1 -- 172.21.15.16:0/3404545449 wait complete. 2024-04-05T23:46:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.701+0000 7f3730e00700 1 Processor -- start 2024-04-05T23:46:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.702+0000 7f3730e00700 1 -- start start 2024-04-05T23:46:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.702+0000 7f3730e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c100e90 0x7f372c199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.702+0000 7f3730e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c105fb0 0x7f372c199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.702+0000 7f3730e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f372c19dc10 0x7f372c19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.702+0000 7f3730e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c19e690 con 0x7f372c19dc10 2024-04-05T23:46:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.702+0000 7f3730e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c19e800 con 0x7f372c100e90 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.702+0000 7f3730e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c19e9c0 con 0x7f372c105fb0 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.702+0000 7f372b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c100e90 0x7f372c199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c100e90 0x7f372c199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59044/0 (socket says 172.21.15.16:59044) 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372b400700 1 -- 172.21.15.16:0/1401153842 learned_addr learned my addr 172.21.15.16:0/1401153842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372aa00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c105fb0 0x7f372c199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372be00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f372c19dc10 0x7f372c19e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372b400700 1 -- 172.21.15.16:0/1401153842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c105fb0 msgr2=0x7f372c199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372b400700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c105fb0 0x7f372c199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372b400700 1 -- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f372c19dc10 msgr2=0x7f372c19e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372b400700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f372c19dc10 0x7f372c19e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372b400700 1 -- 172.21.15.16:0/1401153842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37140095f0 con 0x7f372c100e90 2024-04-05T23:46:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.703+0000 7f372b400700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c100e90 0x7f372c199330 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f371c00ef10 tx=0x7f371c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.704+0000 7f3723e00700 1 -- 172.21.15.16:0/1401153842 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c00ca70 con 0x7f372c100e90 2024-04-05T23:46:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.704+0000 7f3723e00700 1 -- 172.21.15.16:0/1401153842 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f371c00f040 con 0x7f372c100e90 2024-04-05T23:46:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.704+0000 7f3723e00700 1 -- 172.21.15.16:0/1401153842 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c0146e0 con 0x7f372c100e90 2024-04-05T23:46:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.704+0000 7f3730e00700 1 -- 172.21.15.16:0/1401153842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372c1a5ec0 con 0x7f372c100e90 2024-04-05T23:46:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.704+0000 7f3730e00700 1 -- 172.21.15.16:0/1401153842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c1a6290 con 0x7f372c100e90 2024-04-05T23:46:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.708+0000 7f3723e00700 1 -- 172.21.15.16:0/1401153842 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f371c007500 con 0x7f372c100e90 2024-04-05T23:46:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.710+0000 7f3723e00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f37100a1310 0x7f37100a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.710+0000 7f3723e00700 1 -- 172.21.15.16:0/1401153842 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f371c0cceb0 con 0x7f372c100e90 2024-04-05T23:46:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.711+0000 7f372aa00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f37100a1310 0x7f37100a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.712+0000 7f372aa00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f37100a1310 0x7f37100a37d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f371400b2e0 tx=0x7f37140058e0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.712+0000 7f3745ed8b80 1 Processor -- start 2024-04-05T23:46:57.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.713+0000 7f3745ed8b80 1 -- start start 2024-04-05T23:46:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.713+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794520 0x56404c794900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.713+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 0x56404c78e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.713+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c78f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.713+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c6ecb80 con 0x56404c794e40 2024-04-05T23:46:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.713+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c6dd930 con 0x56404c78ed80 2024-04-05T23:46:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.713+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c6db2f0 con 0x56404c794520 2024-04-05T23:46:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c78f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 0x56404c78e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794520 0x56404c794900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794520 0x56404c794900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55228/0 (socket says 172.21.15.16:55228) 2024-04-05T23:46:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372b400700 1 -- 172.21.15.16:0/868334612 learned_addr learned my addr 172.21.15.16:0/868334612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372b400700 1 -- 172.21.15.16:0/868334612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 msgr2=0x56404c78f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372b400700 1 --2- 172.21.15.16:0/868334612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c78f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372b400700 1 -- 172.21.15.16:0/868334612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 msgr2=0x56404c78e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372b400700 1 --2- 172.21.15.16:0/868334612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 0x56404c78e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.714+0000 7f372b400700 1 -- 172.21.15.16:0/868334612 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56404c558350 con 0x56404c794520 2024-04-05T23:46:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.715+0000 7f372b400700 1 --2- 172.21.15.16:0/868334612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794520 0x56404c794900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f371c0a1ef0 tx=0x7f371c099bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7d2c2acdc8d44b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.716+0000 7f3720c00700 1 -- 172.21.15.16:0/868334612 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c099fa0 con 0x56404c794520 2024-04-05T23:46:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.716+0000 7f3720c00700 1 -- 172.21.15.16:0/868334612 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f371c09ddd0 con 0x56404c794520 2024-04-05T23:46:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.716+0000 7f3720c00700 1 -- 172.21.15.16:0/868334612 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c09a200 con 0x56404c794520 2024-04-05T23:46:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.718+0000 7f3745ed8b80 1 -- 172.21.15.16:0/868334612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794520 msgr2=0x56404c794900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.718+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/868334612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794520 0x56404c794900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f371c0a1ef0 tx=0x7f371c099bf0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.719+0000 7f3745ed8b80 1 -- 172.21.15.16:0/868334612 shutdown_connections 2024-04-05T23:46:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.719+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/868334612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c78f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.719+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/868334612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794520 0x56404c794900 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.719+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/868334612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 0x56404c78e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.719+0000 7f3745ed8b80 1 -- 172.21.15.16:0/868334612 >> 172.21.15.16:0/868334612 conn(0x56404c6fa1c0 msgr2=0x56404c780f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.719+0000 7f3745ed8b80 1 -- 172.21.15.16:0/868334612 shutdown_connections 2024-04-05T23:46:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.719+0000 7f3745ed8b80 1 -- 172.21.15.16:0/868334612 wait complete. 2024-04-05T23:46:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.721+0000 7f3745ed8b80 1 -- start start 2024-04-05T23:46:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.722+0000 7f3745ed8b80 1 Processor -- start 2024-04-05T23:46:57.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:57 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:46:57.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:57 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:46:57.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:57 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T23:46:57.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:57 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T23:46:57.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:57 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c7982d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 0x56404c798810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794e40 0x56404c798d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c6d6be0 con 0x56404c794520 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c672d20 con 0x56404c794e40 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c6729c0 con 0x56404c78ed80 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f3745ed8b80 10 client.0 _ll_register_callbacks cb 0x56404c4de340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f372b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c7982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f372b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c7982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:55244/0 (socket says 172.21.15.16:55244) 2024-04-05T23:46:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f372b400700 1 -- 172.21.15.16:0/1492253438 learned_addr learned my addr 172.21.15.16:0/1492253438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.724+0000 7f372be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794e40 0x56404c798d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f372b400700 1 -- 172.21.15.16:0/1492253438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794e40 msgr2=0x56404c798d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f372b400700 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794e40 0x56404c798d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f372b400700 1 -- 172.21.15.16:0/1492253438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 msgr2=0x56404c798810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f372b400700 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 0x56404c798810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f372b400700 1 -- 172.21.15.16:0/1492253438 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c1a6290 con 0x56404c78ed80 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f372be00700 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794e40 0x56404c798d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f372b400700 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c7982d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f371c096a10 tx=0x7f371c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99547e308f01b457 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f3719600700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-05T23:46:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f3719600700 1 -- 172.21.15.16:0/1492253438 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c076de0 con 0x56404c78ed80 2024-04-05T23:46:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f3719600700 1 -- 172.21.15.16:0/1492253438 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f371c0a2420 con 0x56404c78ed80 2024-04-05T23:46:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.725+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1492253438 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56404c7791e0 con 0x56404c78ed80 2024-04-05T23:46:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.726+0000 7f3719600700 1 -- 172.21.15.16:0/1492253438 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f371c099fa0 con 0x56404c78ed80 2024-04-05T23:46:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.726+0000 7f3745ed8b80 10 client.26953 fetch_fsmap learned FSMap version 83 2024-04-05T23:46:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.726+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1492253438 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) v3 -- 0x56404c504b00 con 0x56404c78ed80 2024-04-05T23:46:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.727+0000 7f3719600700 1 -- 172.21.15.16:0/1492253438 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56404c7791e0 con 0x56404c78ed80 2024-04-05T23:46:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.727+0000 7f3719600700 1 -- 172.21.15.16:0/1492253438 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 83) v1 ==== 7522+0+0 (secure 0 0 0) 0x7f371c0a1f90 con 0x56404c78ed80 2024-04-05T23:46:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.727+0000 7f3745ed8b80 10 client.26953 fetch_fsmap finished waiting for FSMap version 83 2024-04-05T23:46:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.727+0000 7f3745ed8b80 10 client.26953 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:57.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.727+0000 7f3745ed8b80 20 client.26953 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:57.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.727+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56404c785410 0x56404c787830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.727+0000 7f3745ed8b80 4 client.26953 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-05T23:46:57.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.727+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1492253438 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56404c6cf6a0 con 0x56404c785410 2024-04-05T23:46:57.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.729+0000 7f372aa00700 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56404c785410 0x56404c787830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.730+0000 7f372aa00700 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56404c785410 0x56404c787830 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cd5338eaf04dca1 server_cookie=a76783de5c697113 in_seq=0 out_seq=0 2024-04-05T23:46:57.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.730+0000 7f3719600700 10 client.26953 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.732+0000 7f3719600700 1 -- 172.21.15.16:0/1492253438 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56404c6cf6a0 con 0x56404c785410 2024-04-05T23:46:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.732+0000 7f3719600700 10 client.26953 handle_command_reply: tid=0 2024-04-05T23:46:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.732+0000 7f3745ed8b80 1 client.26953 shutdown 2024-04-05T23:46:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.733+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1492253438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 msgr2=0x56404c7982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.733+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c7982d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f371c096a10 tx=0x7f371c009bd0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.733+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1492253438 shutdown_connections 2024-04-05T23:46:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.733+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56404c785410 0x56404c787830 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.733+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794e40 0x56404c798d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.733+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c7982d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.733+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1492253438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 0x56404c798810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.733+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1492253438 >> 172.21.15.16:0/1492253438 conn(0x56404c6fa1c0 msgr2=0x56404c780c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.734+0000 7f3719600700 0 client.26953 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.734+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1492253438 shutdown_connections 2024-04-05T23:46:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.734+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1492253438 wait complete. 2024-04-05T23:46:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.735+0000 7f3745ed8b80 20 client.26953 trim_cache size 0 max 16384 2024-04-05T23:46:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.762+0000 7f3745ed8b80 1 Processor -- start 2024-04-05T23:46:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.762+0000 7f3745ed8b80 1 -- start start 2024-04-05T23:46:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.762+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c781a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.762+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794520 0x56404c798b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 0x56404c78e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c55a640 con 0x56404c794e40 2024-04-05T23:46:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c792f90 con 0x56404c794520 2024-04-05T23:46:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c796230 con 0x56404c78ed80 2024-04-05T23:46:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794520 0x56404c798b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794520 0x56404c798b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59060/0 (socket says 172.21.15.16:59060) 2024-04-05T23:46:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372aa00700 1 -- 172.21.15.16:0/2230396771 learned_addr learned my addr 172.21.15.16:0/2230396771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372be00700 1 --2- 172.21.15.16:0/2230396771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 0x56404c78e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372b400700 1 --2- 172.21.15.16:0/2230396771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c781a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372aa00700 1 -- 172.21.15.16:0/2230396771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 msgr2=0x56404c781a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372aa00700 1 --2- 172.21.15.16:0/2230396771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c781a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372aa00700 1 -- 172.21.15.16:0/2230396771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 msgr2=0x56404c78e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372aa00700 1 --2- 172.21.15.16:0/2230396771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 0x56404c78e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.763+0000 7f372aa00700 1 -- 172.21.15.16:0/2230396771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56404c69ce40 con 0x56404c794520 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.764+0000 7f372aa00700 1 --2- 172.21.15.16:0/2230396771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794520 0x56404c798b70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3714009990 tx=0x7f3714020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f38834dd1883668 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.764+0000 7f3721600700 1 -- 172.21.15.16:0/2230396771 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371400bd70 con 0x56404c794520 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.764+0000 7f3721600700 1 -- 172.21.15.16:0/2230396771 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56404c6cf6a0 con 0x56404c794520 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.764+0000 7f3721600700 1 -- 172.21.15.16:0/2230396771 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371400f6c0 con 0x56404c794520 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.764+0000 7f3745ed8b80 1 -- 172.21.15.16:0/2230396771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794520 msgr2=0x56404c798b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.764+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/2230396771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794520 0x56404c798b70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3714009990 tx=0x7f3714020000 comp rx=0 tx=0).stop 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.765+0000 7f3745ed8b80 1 -- 172.21.15.16:0/2230396771 shutdown_connections 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.765+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/2230396771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794e40 0x56404c78e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.765+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/2230396771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c78ed80 0x56404c781a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.765+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/2230396771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c794520 0x56404c798b70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.765+0000 7f3745ed8b80 1 -- 172.21.15.16:0/2230396771 >> 172.21.15.16:0/2230396771 conn(0x56404c6fa1c0 msgr2=0x56404c78f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.765+0000 7f3745ed8b80 1 -- 172.21.15.16:0/2230396771 shutdown_connections 2024-04-05T23:46:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.765+0000 7f3745ed8b80 1 -- 172.21.15.16:0/2230396771 wait complete. 2024-04-05T23:46:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.765+0000 7f3745ed8b80 1 -- start start 2024-04-05T23:46:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.765+0000 7f3745ed8b80 1 Processor -- start 2024-04-05T23:46:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c7a9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 0x56404c7aa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f3745ed8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794e40 0x56404c7a8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c7a8890 con 0x56404c794520 2024-04-05T23:46:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c7a8a00 con 0x56404c78ed80 2024-04-05T23:46:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f3745ed8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56404c7a8bc0 con 0x56404c794e40 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f3745ed8b80 10 client.0 _ll_register_callbacks cb 0x7f3714005560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f372b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c7a9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f372b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c7a9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:59064/0 (socket says 172.21.15.16:59064) 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f372b400700 1 -- 172.21.15.16:0/1041485531 learned_addr learned my addr 172.21.15.16:0/1041485531 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f372aa00700 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 0x56404c7aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f372be00700 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794e40 0x56404c7a8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.766+0000 7f372b400700 1 -- 172.21.15.16:0/1041485531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794e40 msgr2=0x56404c7a8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f372b400700 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794e40 0x56404c7a8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f372b400700 1 -- 172.21.15.16:0/1041485531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 msgr2=0x56404c7aa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f372b400700 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 0x56404c7aa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f372b400700 1 -- 172.21.15.16:0/1041485531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c10e3a0 con 0x56404c78ed80 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f372aa00700 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 0x56404c7aa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:46:57.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f372be00700 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794e40 0x56404c7a8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f372b400700 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c7a9c50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f371c097a50 tx=0x7f371c0b84a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fcca767cad4fb1c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f3719600700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f3719600700 1 -- 172.21.15.16:0/1041485531 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c07c070 con 0x56404c78ed80 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f3719600700 1 -- 172.21.15.16:0/1041485531 <== mon.2 v2:172.21.15.27:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f371c0a2420 con 0x56404c78ed80 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1041485531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f371c09ddd0 con 0x56404c78ed80 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f3719600700 1 -- 172.21.15.16:0/1041485531 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f371c0a21a0 con 0x56404c78ed80 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f3745ed8b80 10 client.26927 fetch_fsmap learned FSMap version 83 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.767+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1041485531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) v3 -- 0x56404c55cc70 con 0x56404c78ed80 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.768+0000 7f3719600700 1 -- 172.21.15.16:0/1041485531 <== mon.2 v2:172.21.15.27:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f371c0a6020 con 0x56404c78ed80 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.768+0000 7f3719600700 1 -- 172.21.15.16:0/1041485531 <== mon.2 v2:172.21.15.27:3300/0 5 ==== fsmap(e 83) v1 ==== 7522+0+0 (secure 0 0 0) 0x7f371c07f020 con 0x56404c78ed80 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.768+0000 7f3745ed8b80 10 client.26927 fetch_fsmap finished waiting for FSMap version 83 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.768+0000 7f3745ed8b80 10 client.26927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.768+0000 7f3745ed8b80 20 client.26927 populate_metadata read hostname 'smithi016' 2024-04-05T23:46:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.768+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56404c7f4010 0x56404c7f6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:46:57.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.768+0000 7f3745ed8b80 4 client.26927 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-05T23:46:57.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.768+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1041485531 --> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56404c525c90 con 0x56404c7f4010 2024-04-05T23:46:57.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.769+0000 7f372aa00700 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56404c7f4010 0x56404c7f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:46:57.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.769+0000 7f372aa00700 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56404c7f4010 0x56404c7f6400 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f92d1f0a9df16be9 server_cookie=38667d883fdde01c in_seq=0 out_seq=0 2024-04-05T23:46:57.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:57.769+0000 7f3719600700 10 client.26927 ms_handle_connect on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.138+0000 7f3719600700 1 -- 172.21.15.16:0/1041485531 <== mds.0 v2:172.21.15.27:6838/491598025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56404c525c90 con 0x56404c7f4010 2024-04-05T23:46:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.138+0000 7f3719600700 10 client.26927 handle_command_reply: tid=0 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3745ed8b80 1 client.26927 shutdown 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1041485531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 msgr2=0x56404c7a9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c7a9c50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f371c097a50 tx=0x7f371c0b84a0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1041485531 shutdown_connections 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.27:6838/491598025,v1:172.21.15.27:6839/491598025] conn(0x56404c7f4010 0x56404c7f6400 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56404c794e40 0x56404c7a8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56404c78ed80 0x56404c7a9c50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3745ed8b80 1 --2- 172.21.15.16:0/1041485531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56404c794520 0x56404c7aa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1041485531 >> 172.21.15.16:0/1041485531 conn(0x56404c6fa1c0 msgr2=0x56404c78e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.139+0000 7f3719600700 0 client.26927 ms_handle_reset on v2:172.21.15.27:6838/491598025 2024-04-05T23:46:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.140+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1041485531 shutdown_connections 2024-04-05T23:46:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.140+0000 7f3745ed8b80 1 -- 172.21.15.16:0/1041485531 wait complete. 2024-04-05T23:46:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.140+0000 7f3745ed8b80 20 client.26927 trim_cache size 0 max 16384 2024-04-05T23:46:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-05T23:46:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-05T23:46:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-05T23:46:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-05T23:46:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 -- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f37100a1310 msgr2=0x7f37100a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f37100a1310 0x7f37100a37d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f371400b2e0 tx=0x7f37140058e0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 -- 172.21.15.16:0/1401153842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c100e90 msgr2=0x7f372c199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:46:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c100e90 0x7f372c199330 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f371c00ef10 tx=0x7f371c00c3a0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 -- 172.21.15.16:0/1401153842 shutdown_connections 2024-04-05T23:46:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f37100a1310 0x7f37100a37d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f372c19dc10 0x7f372c19e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372c100e90 0x7f372c199330 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 --2- 172.21.15.16:0/1401153842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c105fb0 0x7f372c199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:46:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 -- 172.21.15.16:0/1401153842 >> 172.21.15.16:0/1401153842 conn(0x7f372c0fc770 msgr2=0x7f372c0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:46:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 -- 172.21.15.16:0/1401153842 shutdown_connections 2024-04-05T23:46:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-05T23:46:58.142+0000 7f3730e00700 1 -- 172.21.15.16:0/1401153842 wait complete. 2024-04-05T23:46:58.390 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7593 34.12 MB/sec warmup 17 sec latency 268.719 ms 2024-04-05T23:46:58.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:46:58 smithi016 ceph-mon[38248]: pgmap v421: 97 pgs: 97 active+clean; 288 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 536 op/s 2024-04-05T23:46:58.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:46:58 smithi130 ceph-mon[79417]: pgmap v421: 97 pgs: 97 active+clean; 288 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 536 op/s 2024-04-05T23:46:58.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:46:58 smithi027 ceph-mon[78597]: pgmap v421: 97 pgs: 97 active+clean; 288 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 536 op/s 2024-04-05T23:46:59.391 INFO:tasks.workunit.client.0.smithi016.stdout: 10 9133 36.23 MB/sec warmup 18 sec latency 36.925 ms 2024-04-05T23:47:00.391 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10318 39.03 MB/sec warmup 19 sec latency 92.412 ms 2024-04-05T23:47:00.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:00 smithi016 ceph-mon[38248]: pgmap v422: 97 pgs: 97 active+clean; 288 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-04-05T23:47:00.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:00 smithi130 ceph-mon[79417]: pgmap v422: 97 pgs: 97 active+clean; 288 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-04-05T23:47:00.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:00 smithi027 ceph-mon[78597]: pgmap v422: 97 pgs: 97 active+clean; 288 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-04-05T23:47:01.391 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10343 37.10 MB/sec warmup 20 sec latency 980.748 ms 2024-04-05T23:47:02.391 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10343 35.33 MB/sec warmup 21 sec latency 1980.804 ms 2024-04-05T23:47:02.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:02 smithi016 ceph-mon[38248]: pgmap v423: 97 pgs: 97 active+clean; 285 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 626 op/s 2024-04-05T23:47:02.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:02 smithi130 ceph-mon[79417]: pgmap v423: 97 pgs: 97 active+clean; 285 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 626 op/s 2024-04-05T23:47:02.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:02 smithi027 ceph-mon[78597]: pgmap v423: 97 pgs: 97 active+clean; 285 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 626 op/s 2024-04-05T23:47:03.391 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10343 33.73 MB/sec warmup 22 sec latency 2980.854 ms 2024-04-05T23:47:04.393 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10343 32.26 MB/sec warmup 23 sec latency 3980.995 ms 2024-04-05T23:47:04.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:04 smithi016 ceph-mon[38248]: pgmap v424: 97 pgs: 97 active+clean; 285 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 503 op/s 2024-04-05T23:47:04.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:04 smithi130 ceph-mon[79417]: pgmap v424: 97 pgs: 97 active+clean; 285 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 503 op/s 2024-04-05T23:47:04.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:04 smithi027 ceph-mon[78597]: pgmap v424: 97 pgs: 97 active+clean; 285 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 503 op/s 2024-04-05T23:47:05.392 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10433 31.02 MB/sec warmup 24 sec latency 4710.884 ms 2024-04-05T23:47:06.392 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10440 29.86 MB/sec warmup 25 sec latency 1106.933 ms 2024-04-05T23:47:06.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:06 smithi027 ceph-mon[78597]: pgmap v425: 97 pgs: 97 active+clean; 285 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 503 op/s 2024-04-05T23:47:06.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:06 smithi016 ceph-mon[38248]: pgmap v425: 97 pgs: 97 active+clean; 285 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 503 op/s 2024-04-05T23:47:06.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:06 smithi130 ceph-mon[79417]: pgmap v425: 97 pgs: 97 active+clean; 285 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 503 op/s 2024-04-05T23:47:07.392 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10780 29.07 MB/sec warmup 26 sec latency 1455.781 ms 2024-04-05T23:47:08.113 INFO:tasks.workunit.client.0.smithi016.stdout:[11656] write failed on handle 12375 (Resource temporarily unavailable) 2024-04-05T23:47:08.113 INFO:tasks.workunit.client.0.smithi016.stdout:Child failed with status 1 2024-04-05T23:47:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:47:08.143 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-05T23:47:08.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-05T23:47:08.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:08 smithi016 ceph-mon[38248]: pgmap v426: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.8 MiB/s wr, 615 op/s 2024-04-05T23:47:08.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:08 smithi027 ceph-mon[78597]: pgmap v426: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.8 MiB/s wr, 615 op/s 2024-04-05T23:47:08.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:08 smithi130 ceph-mon[79417]: pgmap v426: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.8 MiB/s wr, 615 op/s 2024-04-05T23:47:09.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:09 smithi027 ceph-mon[78597]: osdmap e100: 12 total, 12 up, 12 in 2024-04-05T23:47:09.799 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:09 smithi016 ceph-mon[38248]: osdmap e100: 12 total, 12 up, 12 in 2024-04-05T23:47:09.803 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8919605b0a5dc1f1650c8aec3051c77c0e3e240e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-05T23:47:10.044 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2b9517c1385d4e6e8c8160ff5799db69 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8919605b0a5dc1f1650c8aec3051c77c0e3e240e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-05T23:47:10.047 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-05T23:47:10.057 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:47:10.061 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:09 smithi130 ceph-mon[79417]: osdmap e100: 12 total, 12 up, 12 in 2024-04-05T23:47:10.070 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-05T23:47:10.093 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-05T23:47:10.094 INFO:teuthology.task.exec:Executing custom commands... 2024-04-05T23:47:10.094 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:47:10.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-05T23:47:10.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-05T23:47:10.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-05T23:47:10.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.463+0000 7fb35cc00700 1 -- 172.21.15.16:0/2734272423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358073280 msgr2=0x7fb358073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.463+0000 7fb35cc00700 1 --2- 172.21.15.16:0/2734272423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358073280 0x7fb358073660 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb348009990 tx=0x7fb348009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.463+0000 7fb35cc00700 1 -- 172.21.15.16:0/2734272423 shutdown_connections 2024-04-05T23:47:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.463+0000 7fb35cc00700 1 --2- 172.21.15.16:0/2734272423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb358111de0 0x7fb3581141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.463+0000 7fb35cc00700 1 --2- 172.21.15.16:0/2734272423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358073280 0x7fb358073660 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.463+0000 7fb35cc00700 1 --2- 172.21.15.16:0/2734272423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb358073ba0 0x7fb3581118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.463+0000 7fb35cc00700 1 -- 172.21.15.16:0/2734272423 >> 172.21.15.16:0/2734272423 conn(0x7fb3580fc8e0 msgr2=0x7fb3580fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:10.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.463+0000 7fb35cc00700 1 -- 172.21.15.16:0/2734272423 shutdown_connections 2024-04-05T23:47:10.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.463+0000 7fb35cc00700 1 -- 172.21.15.16:0/2734272423 wait complete. 2024-04-05T23:47:10.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.464+0000 7fb35cc00700 1 Processor -- start 2024-04-05T23:47:10.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.464+0000 7fb35cc00700 1 -- start start 2024-04-05T23:47:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb35cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb358073280 0x7fb3581a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb35cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358073ba0 0x7fb3581a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb35cc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb358111de0 0x7fb35819d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb35cc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35819ddc0 con 0x7fb358073280 2024-04-05T23:47:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb35cc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35819df30 con 0x7fb358073ba0 2024-04-05T23:47:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb35cc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35819e0f0 con 0x7fb358111de0 2024-04-05T23:47:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb355e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb358073280 0x7fb3581a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb355e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb358073280 0x7fb3581a3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47964/0 (socket says 172.21.15.16:47964) 2024-04-05T23:47:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb355e00700 1 -- 172.21.15.16:0/3899463837 learned_addr learned my addr 172.21.15.16:0/3899463837 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb355400700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358073ba0 0x7fb3581a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb356800700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb358111de0 0x7fb35819d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb355e00700 1 -- 172.21.15.16:0/3899463837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb358111de0 msgr2=0x7fb35819d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb355e00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb358111de0 0x7fb35819d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb355e00700 1 -- 172.21.15.16:0/3899463837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358073ba0 msgr2=0x7fb3581a3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb355e00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358073ba0 0x7fb3581a3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.465+0000 7fb355e00700 1 -- 172.21.15.16:0/3899463837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb350009c40 con 0x7fb358073280 2024-04-05T23:47:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.466+0000 7fb355e00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb358073280 0x7fb3581a3630 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb348000c00 tx=0x7fb348005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.466+0000 7fb34ea00700 1 -- 172.21.15.16:0/3899463837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34801d070 con 0x7fb358073280 2024-04-05T23:47:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.466+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3480095f0 con 0x7fb358073280 2024-04-05T23:47:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.466+0000 7fb34ea00700 1 -- 172.21.15.16:0/3899463837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb348022470 con 0x7fb358073280 2024-04-05T23:47:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.466+0000 7fb34ea00700 1 -- 172.21.15.16:0/3899463837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348017650 con 0x7fb358073280 2024-04-05T23:47:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.466+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3581aa440 con 0x7fb358073280 2024-04-05T23:47:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.467+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3581042e0 con 0x7fb358073280 2024-04-05T23:47:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.469+0000 7fb34ea00700 1 -- 172.21.15.16:0/3899463837 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fb3480225e0 con 0x7fb358073280 2024-04-05T23:47:10.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.470+0000 7fb34ea00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb33c0a1290 0x7fb33c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:10.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.470+0000 7fb34ea00700 1 -- 172.21.15.16:0/3899463837 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fb3480ce140 con 0x7fb358073280 2024-04-05T23:47:10.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.471+0000 7fb355400700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb33c0a1290 0x7fb33c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:10.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.471+0000 7fb355400700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb33c0a1290 0x7fb33c0a3750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb35819ee70 tx=0x7fb344009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.475+0000 7fb34ea00700 1 -- 172.21.15.16:0/3899463837 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb348097690 con 0x7fb358073280 2024-04-05T23:47:10.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:10 smithi027 ceph-mon[78597]: pgmap v428: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.7 MiB/s wr, 592 op/s 2024-04-05T23:47:10.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.726+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 --> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fb35819ec20 con 0x7fb33c0a1290 2024-04-05T23:47:10.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.734+0000 7fb34ea00700 1 -- 172.21.15.16:0/3899463837 <== mgr.15240 v2:172.21.15.16:6800/3895420073 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7fb35819ec20 con 0x7fb33c0a1290 2024-04-05T23:47:10.724 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-05T00:00:00", "created": "2024-04-05T23:33:45", "first": "2024-04-05T23:34:00", "last": "2024-04-05T23:47:00", "last_pruned": "2024-04-05T23:47:05", "created_count": 14, "pruned_count": 8, "active": true} 2024-04-05T23:47:10.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb33c0a1290 msgr2=0x7fb33c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:10.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb33c0a1290 0x7fb33c0a3750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb35819ee70 tx=0x7fb344009260 comp rx=0 tx=0).stop 2024-04-05T23:47:10.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb358073280 msgr2=0x7fb3581a3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:10.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb358073280 0x7fb3581a3630 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb348000c00 tx=0x7fb348005740 comp rx=0 tx=0).stop 2024-04-05T23:47:10.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 shutdown_connections 2024-04-05T23:47:10.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fb33c0a1290 0x7fb33c0a3750 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb358073280 0x7fb3581a3630 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb358111de0 0x7fb35819d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 --2- 172.21.15.16:0/3899463837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358073ba0 0x7fb3581a3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 >> 172.21.15.16:0/3899463837 conn(0x7fb3580fc8e0 msgr2=0x7fb3580fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:10.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 shutdown_connections 2024-04-05T23:47:10.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.737+0000 7fb35cc00700 1 -- 172.21.15.16:0/3899463837 wait complete. 2024-04-05T23:47:10.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:10 smithi016 ceph-mon[38248]: pgmap v428: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.7 MiB/s wr, 592 op/s 2024-04-05T23:47:10.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-05T23:47:10.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:10 smithi130 ceph-mon[79417]: pgmap v428: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.7 MiB/s wr, 592 op/s 2024-04-05T23:47:10.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.956+0000 7ff055c00700 1 -- 172.21.15.16:0/2719458195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 msgr2=0x7ff050101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.956+0000 7ff055c00700 1 --2- 172.21.15.16:0/2719458195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 0x7ff050101410 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff040009990 tx=0x7ff040009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.956+0000 7ff055c00700 1 -- 172.21.15.16:0/2719458195 shutdown_connections 2024-04-05T23:47:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.956+0000 7ff055c00700 1 --2- 172.21.15.16:0/2719458195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff050106150 0x7ff05010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.956+0000 7ff055c00700 1 --2- 172.21.15.16:0/2719458195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 0x7ff050101410 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.956+0000 7ff055c00700 1 --2- 172.21.15.16:0/2719458195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0501019e0 0x7ff050105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.956+0000 7ff055c00700 1 -- 172.21.15.16:0/2719458195 >> 172.21.15.16:0/2719458195 conn(0x7ff0500fc8b0 msgr2=0x7ff0500fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.956+0000 7ff055c00700 1 -- 172.21.15.16:0/2719458195 shutdown_connections 2024-04-05T23:47:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.956+0000 7ff055c00700 1 -- 172.21.15.16:0/2719458195 wait complete. 2024-04-05T23:47:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.957+0000 7ff055c00700 1 Processor -- start 2024-04-05T23:47:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.957+0000 7ff055c00700 1 -- start start 2024-04-05T23:47:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.957+0000 7ff055c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 0x7ff050199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.957+0000 7ff055c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff050106150 0x7ff050199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff055c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05019dda0 0x7ff05019e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff055c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05019e820 con 0x7ff05019dda0 2024-04-05T23:47:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff055c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05019e990 con 0x7ff050101030 2024-04-05T23:47:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff055c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05019eb50 con 0x7ff050106150 2024-04-05T23:47:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 0x7ff050199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 0x7ff050199560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:36556/0 (socket says 172.21.15.16:36556) 2024-04-05T23:47:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04ea00700 1 -- 172.21.15.16:0/204583396 learned_addr learned my addr 172.21.15.16:0/204583396 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04f400700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05019dda0 0x7ff05019e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:10.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04e000700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff050106150 0x7ff050199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:10.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04ea00700 1 -- 172.21.15.16:0/204583396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff050106150 msgr2=0x7ff050199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:10.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04ea00700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff050106150 0x7ff050199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04ea00700 1 -- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05019dda0 msgr2=0x7ff05019e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:10.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04ea00700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05019dda0 0x7ff05019e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:10.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04ea00700 1 -- 172.21.15.16:0/204583396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400095f0 con 0x7ff050101030 2024-04-05T23:47:10.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.958+0000 7ff04ea00700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 0x7ff050199560 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff040000c00 tx=0x7ff040005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:10.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.959+0000 7ff047400700 1 -- 172.21.15.16:0/204583396 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04001d070 con 0x7ff050101030 2024-04-05T23:47:10.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.959+0000 7ff047400700 1 -- 172.21.15.16:0/204583396 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff040022470 con 0x7ff050101030 2024-04-05T23:47:10.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.959+0000 7ff047400700 1 -- 172.21.15.16:0/204583396 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040017650 con 0x7ff050101030 2024-04-05T23:47:10.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.959+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0501a6050 con 0x7ff050101030 2024-04-05T23:47:10.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.959+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0501a6480 con 0x7ff050101030 2024-04-05T23:47:10.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.960+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff050102500 con 0x7ff050101030 2024-04-05T23:47:10.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.962+0000 7ff047400700 1 -- 172.21.15.16:0/204583396 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7ff0400225e0 con 0x7ff050101030 2024-04-05T23:47:10.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.964+0000 7ff047400700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0340a1310 0x7ff0340a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:10.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.964+0000 7ff047400700 1 -- 172.21.15.16:0/204583396 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8600+0+0 (secure 0 0 0) 0x7ff0400ce640 con 0x7ff050101030 2024-04-05T23:47:10.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.964+0000 7ff04e000700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0340a1310 0x7ff0340a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:10.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.964+0000 7ff04e000700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0340a1310 0x7ff0340a37d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff05019aab0 tx=0x7ff03c009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:10.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:10.966+0000 7ff047400700 1 -- 172.21.15.16:0/204583396 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff040097a70 con 0x7ff050101030 2024-04-05T23:47:11.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.241+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 --> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff0501019e0 con 0x7ff0340a1310 2024-04-05T23:47:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.249+0000 7ff047400700 1 -- 172.21.15.16:0/204583396 <== mgr.15240 v2:172.21.15.16:6800/3895420073 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7ff0501019e0 con 0x7ff0340a1310 2024-04-05T23:47:11.239 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-04-05T23:47:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0340a1310 msgr2=0x7ff0340a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0340a1310 0x7ff0340a37d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff05019aab0 tx=0x7ff03c009260 comp rx=0 tx=0).stop 2024-04-05T23:47:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 msgr2=0x7ff050199560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 0x7ff050199560 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff040000c00 tx=0x7ff040005740 comp rx=0 tx=0).stop 2024-04-05T23:47:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 shutdown_connections 2024-04-05T23:47:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff0340a1310 0x7ff0340a37d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff05019dda0 0x7ff05019e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff050101030 0x7ff050199560 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 --2- 172.21.15.16:0/204583396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff050106150 0x7ff050199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.252+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 >> 172.21.15.16:0/204583396 conn(0x7ff0500fc8b0 msgr2=0x7ff0500fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:11.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.253+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 shutdown_connections 2024-04-05T23:47:11.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.253+0000 7ff055c00700 1 -- 172.21.15.16:0/204583396 wait complete. 2024-04-05T23:47:11.265 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-05T23:47:11.276 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-05T23:47:11.276 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-05T23:47:11.281 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-05T23:47:11.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:47:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.616+0000 7fcaade00700 1 -- 172.21.15.16:0/2607107780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaa8100fa0 msgr2=0x7fcaa8101380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.616+0000 7fcaade00700 1 --2- 172.21.15.16:0/2607107780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaa8100fa0 0x7fcaa8101380 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fca98009990 tx=0x7fca98009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.616+0000 7fcaade00700 1 -- 172.21.15.16:0/2607107780 shutdown_connections 2024-04-05T23:47:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.616+0000 7fcaade00700 1 --2- 172.21.15.16:0/2607107780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaa81060c0 0x7fcaa810dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.616+0000 7fcaade00700 1 --2- 172.21.15.16:0/2607107780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaa8100fa0 0x7fcaa8101380 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.616+0000 7fcaade00700 1 --2- 172.21.15.16:0/2607107780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa8101950 0x7fcaa8105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.616+0000 7fcaade00700 1 -- 172.21.15.16:0/2607107780 >> 172.21.15.16:0/2607107780 conn(0x7fcaa80fc860 msgr2=0x7fcaa80fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.616+0000 7fcaade00700 1 -- 172.21.15.16:0/2607107780 shutdown_connections 2024-04-05T23:47:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.616+0000 7fcaade00700 1 -- 172.21.15.16:0/2607107780 wait complete. 2024-04-05T23:47:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.618+0000 7fcaade00700 1 Processor -- start 2024-04-05T23:47:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.618+0000 7fcaade00700 1 -- start start 2024-04-05T23:47:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaade00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaa8100fa0 0x7fcaa81993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaade00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa81060c0 0x7fcaa8199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaade00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaa819dcd0 0x7fcaa819e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaade00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa819e750 con 0x7fcaa819dcd0 2024-04-05T23:47:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaade00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa819e8c0 con 0x7fcaa8100fa0 2024-04-05T23:47:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaade00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa819ea80 con 0x7fcaa81060c0 2024-04-05T23:47:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaa6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa81060c0 0x7fcaa8199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaa6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa81060c0 0x7fcaa8199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:33628/0 (socket says 172.21.15.16:33628) 2024-04-05T23:47:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaa6000700 1 -- 172.21.15.16:0/1225023188 learned_addr learned my addr 172.21.15.16:0/1225023188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaa6a00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaa8100fa0 0x7fcaa81993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.619+0000 7fcaa7400700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaa819dcd0 0x7fcaa819e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.620+0000 7fcaa6000700 1 -- 172.21.15.16:0/1225023188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaa8100fa0 msgr2=0x7fcaa81993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.620+0000 7fcaa6000700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaa8100fa0 0x7fcaa81993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.620+0000 7fcaa6000700 1 -- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaa819dcd0 msgr2=0x7fcaa819e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.620+0000 7fcaa6000700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaa819dcd0 0x7fcaa819e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.620+0000 7fcaa6000700 1 -- 172.21.15.16:0/1225023188 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca980095f0 con 0x7fcaa81060c0 2024-04-05T23:47:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.620+0000 7fcaa7400700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaa819dcd0 0x7fcaa819e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:47:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.620+0000 7fcaa6000700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa81060c0 0x7fcaa8199930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fca9400c8f0 tx=0x7fca9400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.620+0000 7fcaaca00700 1 -- 172.21.15.16:0/1225023188 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca94007b00 con 0x7fcaa81060c0 2024-04-05T23:47:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.621+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa81a5fe0 con 0x7fcaa81060c0 2024-04-05T23:47:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.621+0000 7fcaaca00700 1 -- 172.21.15.16:0/1225023188 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca94007c60 con 0x7fcaa81060c0 2024-04-05T23:47:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.621+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa8107eb0 con 0x7fcaa81060c0 2024-04-05T23:47:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.621+0000 7fcaaca00700 1 -- 172.21.15.16:0/1225023188 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca94018660 con 0x7fcaa81060c0 2024-04-05T23:47:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.625+0000 7fcaaca00700 1 -- 172.21.15.16:0/1225023188 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fca94004830 con 0x7fcaa81060c0 2024-04-05T23:47:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.626+0000 7fcaaca00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca8c0a1310 0x7fca8c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.626+0000 7fcaaca00700 1 -- 172.21.15.16:0/1225023188 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8600+0+0 (secure 0 0 0) 0x7fca940cc510 con 0x7fcaa81060c0 2024-04-05T23:47:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.625+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca880052f0 con 0x7fcaa81060c0 2024-04-05T23:47:11.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.627+0000 7fcaa6a00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca8c0a1310 0x7fca8c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:11.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.628+0000 7fcaa6a00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca8c0a1310 0x7fca8c0a37d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fca9800b5c0 tx=0x7fca9801a040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:11.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.634+0000 7fcaaca00700 1 -- 172.21.15.16:0/1225023188 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fca94096890 con 0x7fcaa81060c0 2024-04-05T23:47:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.901+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca88005870 con 0x7fcaa81060c0 2024-04-05T23:47:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.903+0000 7fcaaca00700 1 -- 172.21.15.16:0/1225023188 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+23120 (secure 0 0 0) 0x7fca94099eb0 con 0x7fcaa81060c0 2024-04-05T23:47:11.893 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:47:11.894 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":100,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:47:08.346826+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/985211078":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1740984099":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/4157043736":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/1565223355":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2846240830":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/4184690359":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/1565223355":"2024-04-06T23:33:34.593227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":23,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":23,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:47:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca8c0a1310 msgr2=0x7fca8c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca8c0a1310 0x7fca8c0a37d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fca9800b5c0 tx=0x7fca9801a040 comp rx=0 tx=0).stop 2024-04-05T23:47:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa81060c0 msgr2=0x7fcaa8199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa81060c0 0x7fcaa8199930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fca9400c8f0 tx=0x7fca9400ccb0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 shutdown_connections 2024-04-05T23:47:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fca8c0a1310 0x7fca8c0a37d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaa819dcd0 0x7fcaa819e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaa8100fa0 0x7fcaa81993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 --2- 172.21.15.16:0/1225023188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa81060c0 0x7fcaa8199930 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.905+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 >> 172.21.15.16:0/1225023188 conn(0x7fcaa80fc860 msgr2=0x7fcaa80fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.906+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 shutdown_connections 2024-04-05T23:47:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:11.906+0000 7fcaade00700 1 -- 172.21.15.16:0/1225023188 wait complete. 2024-04-05T23:47:11.915 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-05T23:32:02.641673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-05T23:32:03.972521+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 100, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:47:11.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:47:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.129+0000 7f1f09e00700 1 -- 172.21.15.16:0/2905821659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041010c0 msgr2=0x7f1f041014a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.129+0000 7f1f09e00700 1 --2- 172.21.15.16:0/2905821659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041010c0 0x7f1f041014a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009990 tx=0x7f1ef4009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.129+0000 7f1f09e00700 1 -- 172.21.15.16:0/2905821659 shutdown_connections 2024-04-05T23:47:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.129+0000 7f1f09e00700 1 --2- 172.21.15.16:0/2905821659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041010c0 0x7f1f041014a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.129+0000 7f1f09e00700 1 --2- 172.21.15.16:0/2905821659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f041061e0 0x7f1f0410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.129+0000 7f1f09e00700 1 --2- 172.21.15.16:0/2905821659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f04101a70 0x7f1f04105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.129+0000 7f1f09e00700 1 -- 172.21.15.16:0/2905821659 >> 172.21.15.16:0/2905821659 conn(0x7f1f040fc920 msgr2=0x7f1f040fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.129+0000 7f1f09e00700 1 -- 172.21.15.16:0/2905821659 shutdown_connections 2024-04-05T23:47:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.129+0000 7f1f09e00700 1 -- 172.21.15.16:0/2905821659 wait complete. 2024-04-05T23:47:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.131+0000 7f1f09e00700 1 Processor -- start 2024-04-05T23:47:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.131+0000 7f1f09e00700 1 -- start start 2024-04-05T23:47:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f09e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f041010c0 0x7f1f04199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f09e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041061e0 0x7f1f04199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:12.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f09e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0419dd80 0x7f1f0419e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f09e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0419e800 con 0x7f1f041061e0 2024-04-05T23:47:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f09e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0419e970 con 0x7f1f0419dd80 2024-04-05T23:47:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f09e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0419eb30 con 0x7f1f041010c0 2024-04-05T23:47:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041061e0 0x7f1f04199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f041010c0 0x7f1f04199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f041010c0 0x7f1f04199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59174/0 (socket says 172.21.15.16:59174) 2024-04-05T23:47:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02a00700 1 -- 172.21.15.16:0/1766814494 learned_addr learned my addr 172.21.15.16:0/1766814494 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02000700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041061e0 0x7f1f04199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51068/0 (socket says 172.21.15.16:51068) 2024-04-05T23:47:12.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f03400700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0419dd80 0x7f1f0419e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02000700 1 -- 172.21.15.16:0/1766814494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f041010c0 msgr2=0x7f1f04199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02000700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f041010c0 0x7f1f04199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02000700 1 -- 172.21.15.16:0/1766814494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0419dd80 msgr2=0x7f1f0419e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02000700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0419dd80 0x7f1f0419e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02000700 1 -- 172.21.15.16:0/1766814494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef40095f0 con 0x7f1f041061e0 2024-04-05T23:47:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.132+0000 7f1f02a00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f041010c0 0x7f1f04199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:47:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.133+0000 7f1f02000700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041061e0 0x7f1f04199a50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000e9c0 tx=0x7f1ef000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:12.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.133+0000 7f1f08a00700 1 -- 172.21.15.16:0/1766814494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef000c9c0 con 0x7f1f041061e0 2024-04-05T23:47:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.133+0000 7f1f08a00700 1 -- 172.21.15.16:0/1766814494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ef0010430 con 0x7f1f041061e0 2024-04-05T23:47:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.133+0000 7f1f08a00700 1 -- 172.21.15.16:0/1766814494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0018680 con 0x7f1f041061e0 2024-04-05T23:47:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.133+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f041a6090 con 0x7f1f041061e0 2024-04-05T23:47:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.133+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f041a6520 con 0x7f1f041061e0 2024-04-05T23:47:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.134+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f04102590 con 0x7f1f041061e0 2024-04-05T23:47:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.136+0000 7f1f08a00700 1 -- 172.21.15.16:0/1766814494 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f1ef000cb20 con 0x7f1f041061e0 2024-04-05T23:47:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.138+0000 7f1f08a00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1ee80a1310 0x7f1ee80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.138+0000 7f1f08a00700 1 -- 172.21.15.16:0/1766814494 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8600+0+0 (secure 0 0 0) 0x7f1ef0014070 con 0x7f1f041061e0 2024-04-05T23:47:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.138+0000 7f1f02a00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1ee80a1310 0x7f1ee80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:12.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.139+0000 7f1f02a00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1ee80a1310 0x7f1ee80a37d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4006010 tx=0x7f1ef400b260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.142+0000 7f1f08a00700 1 -- 172.21.15.16:0/1766814494 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1ef0096440 con 0x7f1f041061e0 2024-04-05T23:47:12.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.410+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f04067c00 con 0x7f1f041061e0 2024-04-05T23:47:12.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.410+0000 7f1f08a00700 1 -- 172.21.15.16:0/1766814494 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ef0099a60 con 0x7f1f041061e0 2024-04-05T23:47:12.401 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:47:12.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1ee80a1310 msgr2=0x7f1ee80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1ee80a1310 0x7f1ee80a37d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4006010 tx=0x7f1ef400b260 comp rx=0 tx=0).stop 2024-04-05T23:47:12.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041061e0 msgr2=0x7f1f04199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041061e0 0x7f1f04199a50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000e9c0 tx=0x7f1ef000ecd0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 shutdown_connections 2024-04-05T23:47:12.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f1ee80a1310 0x7f1ee80a37d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0419dd80 0x7f1f0419e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f041010c0 0x7f1f04199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 --2- 172.21.15.16:0/1766814494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f041061e0 0x7f1f04199a50 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 >> 172.21.15.16:0/1766814494 conn(0x7f1f040fc920 msgr2=0x7f1f040fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:12.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 shutdown_connections 2024-04-05T23:47:12.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.413+0000 7f1f09e00700 1 -- 172.21.15.16:0/1766814494 wait complete. 2024-04-05T23:47:12.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-05T23:47:12.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.638+0000 7f34ec600700 1 -- 172.21.15.16:0/1441384360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34e4067f90 msgr2=0x7f34e4068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.638+0000 7f34ec600700 1 --2- 172.21.15.16:0/1441384360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34e4067f90 0x7f34e4068370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009990 tx=0x7f34d8009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.638+0000 7f34ec600700 1 -- 172.21.15.16:0/1441384360 shutdown_connections 2024-04-05T23:47:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.638+0000 7f34ec600700 1 --2- 172.21.15.16:0/1441384360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e410d880 0x7f34e410fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.638+0000 7f34ec600700 1 --2- 172.21.15.16:0/1441384360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34e4067f90 0x7f34e4068370 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.638+0000 7f34ec600700 1 --2- 172.21.15.16:0/1441384360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e40688b0 0x7f34e410d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.638+0000 7f34ec600700 1 -- 172.21.15.16:0/1441384360 >> 172.21.15.16:0/1441384360 conn(0x7f34e4076e80 msgr2=0x7f34e4077290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.638+0000 7f34ec600700 1 -- 172.21.15.16:0/1441384360 shutdown_connections 2024-04-05T23:47:12.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.638+0000 7f34ec600700 1 -- 172.21.15.16:0/1441384360 wait complete. 2024-04-05T23:47:12.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.639+0000 7f34ec600700 1 Processor -- start 2024-04-05T23:47:12.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.639+0000 7f34ec600700 1 -- start start 2024-04-05T23:47:12.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.639+0000 7f34ec600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e4067f90 0x7f34e4199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:12.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.639+0000 7f34ec600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34e40688b0 0x7f34e4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:12.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34ec600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e410d880 0x7f34e419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34ec600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e419e460 con 0x7f34e410d880 2024-04-05T23:47:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34ec600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e419e5d0 con 0x7f34e4067f90 2024-04-05T23:47:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34ec600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e419e790 con 0x7f34e40688b0 2024-04-05T23:47:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e4067f90 0x7f34e4199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:12.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e4067f90 0x7f34e4199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35158/0 (socket says 172.21.15.16:35158) 2024-04-05T23:47:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9c00700 1 -- 172.21.15.16:0/2504411011 learned_addr learned my addr 172.21.15.16:0/2504411011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34e40688b0 0x7f34e4199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34ea600700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e410d880 0x7f34e419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9c00700 1 -- 172.21.15.16:0/2504411011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34e40688b0 msgr2=0x7f34e4199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9c00700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34e40688b0 0x7f34e4199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9c00700 1 -- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e410d880 msgr2=0x7f34e419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9c00700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e410d880 0x7f34e419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9c00700 1 -- 172.21.15.16:0/2504411011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d80095f0 con 0x7f34e4067f90 2024-04-05T23:47:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34e9200700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34e40688b0 0x7f34e4199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:47:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.640+0000 7f34ea600700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e410d880 0x7f34e419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T23:47:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.641+0000 7f34e9c00700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e4067f90 0x7f34e4199540 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f34d8000c00 tx=0x7f34d8005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.641+0000 7f34e2000700 1 -- 172.21.15.16:0/2504411011 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d801d070 con 0x7f34e4067f90 2024-04-05T23:47:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.641+0000 7f34e2000700 1 -- 172.21.15.16:0/2504411011 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34d8022470 con 0x7f34e4067f90 2024-04-05T23:47:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.641+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e419ea10 con 0x7f34e4067f90 2024-04-05T23:47:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.641+0000 7f34e2000700 1 -- 172.21.15.16:0/2504411011 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8017650 con 0x7f34e4067f90 2024-04-05T23:47:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.641+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e41a62b0 con 0x7f34e4067f90 2024-04-05T23:47:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.642+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e4108aa0 con 0x7f34e4067f90 2024-04-05T23:47:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.644+0000 7f34e2000700 1 -- 172.21.15.16:0/2504411011 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f34d80225e0 con 0x7f34e4067f90 2024-04-05T23:47:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.646+0000 7f34e2000700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34cc0a1310 0x7f34cc0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.646+0000 7f34e2000700 1 -- 172.21.15.16:0/2504411011 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8600+0+0 (secure 0 0 0) 0x7f34d80ce3a0 con 0x7f34e4067f90 2024-04-05T23:47:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.646+0000 7f34e9200700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34cc0a1310 0x7f34cc0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.647+0000 7f34e9200700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34cc0a1310 0x7f34cc0a37d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f34e419aa90 tx=0x7f34d4009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:12.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.648+0000 7f34e2000700 1 -- 172.21.15.16:0/2504411011 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f34d8097830 con 0x7f34e4067f90 2024-04-05T23:47:12.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:12 smithi027 ceph-mon[78597]: from='client.16890 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:47:12.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:12 smithi027 ceph-mon[78597]: pgmap v429: 97 pgs: 97 active+clean; 565 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 30 MiB/s wr, 540 op/s 2024-04-05T23:47:12.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:12 smithi027 ceph-mon[78597]: from='client.26939 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:47:12.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:12 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1225023188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:47:12.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:12 smithi016 ceph-mon[38248]: from='client.16890 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:47:12.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:12 smithi016 ceph-mon[38248]: pgmap v429: 97 pgs: 97 active+clean; 565 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 30 MiB/s wr, 540 op/s 2024-04-05T23:47:12.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:12 smithi016 ceph-mon[38248]: from='client.26939 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:47:12.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:12 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1225023188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:47:12.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:12 smithi130 ceph-mon[79417]: from='client.16890 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:47:12.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:12 smithi130 ceph-mon[79417]: pgmap v429: 97 pgs: 97 active+clean; 565 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 30 MiB/s wr, 540 op/s 2024-04-05T23:47:12.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:12 smithi130 ceph-mon[79417]: from='client.26939 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T23:47:12.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:12 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1225023188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:47:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.878+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34e4067c30 con 0x7f34e4067f90 2024-04-05T23:47:12.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.878+0000 7f34e2000700 1 -- 172.21.15.16:0/2504411011 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f34d809ae50 con 0x7f34e4067f90 2024-04-05T23:47:12.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-05T23:47:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34cc0a1310 msgr2=0x7f34cc0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34cc0a1310 0x7f34cc0a37d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f34e419aa90 tx=0x7f34d4009260 comp rx=0 tx=0).stop 2024-04-05T23:47:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e4067f90 msgr2=0x7f34e4199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e4067f90 0x7f34e4199540 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f34d8000c00 tx=0x7f34d8005740 comp rx=0 tx=0).stop 2024-04-05T23:47:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 shutdown_connections 2024-04-05T23:47:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34cc0a1310 0x7f34cc0a37d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e410d880 0x7f34e419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e4067f90 0x7f34e4199540 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 --2- 172.21.15.16:0/2504411011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34e40688b0 0x7f34e4199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 >> 172.21.15.16:0/2504411011 conn(0x7f34e4076e80 msgr2=0x7f34e410eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 shutdown_connections 2024-04-05T23:47:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:12.881+0000 7f34ec600700 1 -- 172.21.15.16:0/2504411011 wait complete. 2024-04-05T23:47:12.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-05T23:47:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.104+0000 7ff4e0e00700 1 -- 172.21.15.16:0/885258664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc1010a0 msgr2=0x7ff4dc101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.104+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/885258664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc1010a0 0x7ff4dc101480 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc009990 tx=0x7ff4cc009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.104+0000 7ff4e0e00700 1 -- 172.21.15.16:0/885258664 shutdown_connections 2024-04-05T23:47:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.104+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/885258664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc1010a0 0x7ff4dc101480 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.104+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/885258664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4dc1061c0 0x7ff4dc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.104+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/885258664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4dc101a50 0x7ff4dc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.104+0000 7ff4e0e00700 1 -- 172.21.15.16:0/885258664 >> 172.21.15.16:0/885258664 conn(0x7ff4dc0fc900 msgr2=0x7ff4dc0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.104+0000 7ff4e0e00700 1 -- 172.21.15.16:0/885258664 shutdown_connections 2024-04-05T23:47:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.104+0000 7ff4e0e00700 1 -- 172.21.15.16:0/885258664 wait complete. 2024-04-05T23:47:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.105+0000 7ff4e0e00700 1 Processor -- start 2024-04-05T23:47:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.105+0000 7ff4e0e00700 1 -- start start 2024-04-05T23:47:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4e0e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc1010a0 0x7ff4dc197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4e0e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4dc1061c0 0x7ff4dc1978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4e0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4dc19bc70 0x7ff4dc19c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4e0e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc19c6f0 con 0x7ff4dc1010a0 2024-04-05T23:47:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4e0e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc19c860 con 0x7ff4dc1061c0 2024-04-05T23:47:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4e0e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc19ca20 con 0x7ff4dc19bc70 2024-04-05T23:47:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4d9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc1010a0 0x7ff4dc197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4d9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4dc1061c0 0x7ff4dc1978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4d9400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4dc1061c0 0x7ff4dc1978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35178/0 (socket says 172.21.15.16:35178) 2024-04-05T23:47:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4d9400700 1 -- 172.21.15.16:0/1082402049 learned_addr learned my addr 172.21.15.16:0/1082402049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4da800700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4dc19bc70 0x7ff4dc19c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4d9e00700 1 -- 172.21.15.16:0/1082402049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4dc19bc70 msgr2=0x7ff4dc19c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4d9e00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4dc19bc70 0x7ff4dc19c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4d9e00700 1 -- 172.21.15.16:0/1082402049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4dc1061c0 msgr2=0x7ff4dc1978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4d9e00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4dc1061c0 0x7ff4dc1978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.106+0000 7ff4d9e00700 1 -- 172.21.15.16:0/1082402049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d4009c40 con 0x7ff4dc1010a0 2024-04-05T23:47:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.107+0000 7ff4d9e00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc1010a0 0x7ff4dc197390 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0098e0 tx=0x7ff4cc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.107+0000 7ff4d2a00700 1 -- 172.21.15.16:0/1082402049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc01d070 con 0x7ff4dc1010a0 2024-04-05T23:47:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.107+0000 7ff4d2a00700 1 -- 172.21.15.16:0/1082402049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4cc022470 con 0x7ff4dc1010a0 2024-04-05T23:47:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.107+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc0095f0 con 0x7ff4dc1010a0 2024-04-05T23:47:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.107+0000 7ff4d2a00700 1 -- 172.21.15.16:0/1082402049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc0175f0 con 0x7ff4dc1010a0 2024-04-05T23:47:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.107+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc0733a0 con 0x7ff4dc1010a0 2024-04-05T23:47:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.108+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc1093e0 con 0x7ff4dc1010a0 2024-04-05T23:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.110+0000 7ff4d2a00700 1 -- 172.21.15.16:0/1082402049 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7ff4cc022850 con 0x7ff4dc1010a0 2024-04-05T23:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.111+0000 7ff4d2a00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff4c00a1290 0x7ff4c00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.111+0000 7ff4d2a00700 1 -- 172.21.15.16:0/1082402049 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8600+0+0 (secure 0 0 0) 0x7ff4cc0cdfa0 con 0x7ff4dc1010a0 2024-04-05T23:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.112+0000 7ff4d9400700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff4c00a1290 0x7ff4c00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.112+0000 7ff4d9400700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff4c00a1290 0x7ff4c00a3750 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc1988e0 tx=0x7ff4c800b480 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.115+0000 7ff4d2a00700 1 -- 172.21.15.16:0/1082402049 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff4cc0973d0 con 0x7ff4dc1010a0 2024-04-05T23:47:13.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.331+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff4dc067c60 con 0x7ff4dc1010a0 2024-04-05T23:47:13.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.332+0000 7ff4d2a00700 1 -- 172.21.15.16:0/1082402049 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff4cc09a9f0 con 0x7ff4dc1010a0 2024-04-05T23:47:13.322 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-05T23:47:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.334+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff4c00a1290 msgr2=0x7ff4c00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.334+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff4c00a1290 0x7ff4c00a3750 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc1988e0 tx=0x7ff4c800b480 comp rx=0 tx=0).stop 2024-04-05T23:47:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.334+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc1010a0 msgr2=0x7ff4dc197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.334+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc1010a0 0x7ff4dc197390 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0098e0 tx=0x7ff4cc005740 comp rx=0 tx=0).stop 2024-04-05T23:47:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.335+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 shutdown_connections 2024-04-05T23:47:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.335+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff4c00a1290 0x7ff4c00a3750 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.335+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc1010a0 0x7ff4dc197390 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.335+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4dc19bc70 0x7ff4dc19c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.335+0000 7ff4e0e00700 1 --2- 172.21.15.16:0/1082402049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4dc1061c0 0x7ff4dc1978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.335+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 >> 172.21.15.16:0/1082402049 conn(0x7ff4dc0fc900 msgr2=0x7ff4dc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.335+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 shutdown_connections 2024-04-05T23:47:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.335+0000 7ff4e0e00700 1 -- 172.21.15.16:0/1082402049 wait complete. 2024-04-05T23:47:13.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-05T23:47:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.555+0000 7ff96c400700 1 -- 172.21.15.16:0/2130622127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9641060b0 msgr2=0x7ff96410dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.555+0000 7ff96c400700 1 --2- 172.21.15.16:0/2130622127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9641060b0 0x7ff96410dc30 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00b0a0 tx=0x7ff95c00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.555+0000 7ff96c400700 1 -- 172.21.15.16:0/2130622127 shutdown_connections 2024-04-05T23:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.555+0000 7ff96c400700 1 --2- 172.21.15.16:0/2130622127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9641060b0 0x7ff96410dc30 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.555+0000 7ff96c400700 1 --2- 172.21.15.16:0/2130622127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff964100f90 0x7ff964101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.555+0000 7ff96c400700 1 --2- 172.21.15.16:0/2130622127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff964101940 0x7ff964105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.555+0000 7ff96c400700 1 -- 172.21.15.16:0/2130622127 >> 172.21.15.16:0/2130622127 conn(0x7ff9640fc830 msgr2=0x7ff9640fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.555+0000 7ff96c400700 1 -- 172.21.15.16:0/2130622127 shutdown_connections 2024-04-05T23:47:13.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.555+0000 7ff96c400700 1 -- 172.21.15.16:0/2130622127 wait complete. 2024-04-05T23:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.556+0000 7ff96c400700 1 Processor -- start 2024-04-05T23:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.557+0000 7ff96c400700 1 -- start start 2024-04-05T23:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.557+0000 7ff96c400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff964100f90 0x7ff9640731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.557+0000 7ff96c400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9641060b0 0x7ff964073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.557+0000 7ff96c400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964073c60 0x7ff9641a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.557+0000 7ff96c400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9641a5b20 con 0x7ff964073c60 2024-04-05T23:47:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.557+0000 7ff96c400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9641a5c90 con 0x7ff9641060b0 2024-04-05T23:47:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.557+0000 7ff96c400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9641a5e50 con 0x7ff964100f90 2024-04-05T23:47:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff964100f90 0x7ff9640731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9641060b0 0x7ff964073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff964100f90 0x7ff9640731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:59244/0 (socket says 172.21.15.16:59244) 2024-04-05T23:47:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9641060b0 0x7ff964073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:35196/0 (socket says 172.21.15.16:35196) 2024-04-05T23:47:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969a00700 1 -- 172.21.15.16:0/1210546376 learned_addr learned my addr 172.21.15.16:0/1210546376 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff96a400700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964073c60 0x7ff9641a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969a00700 1 -- 172.21.15.16:0/1210546376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9641060b0 msgr2=0x7ff964073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969a00700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9641060b0 0x7ff964073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969a00700 1 -- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964073c60 msgr2=0x7ff9641a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969a00700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964073c60 0x7ff9641a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969a00700 1 -- 172.21.15.16:0/1210546376 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c009d00 con 0x7ff964100f90 2024-04-05T23:47:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.558+0000 7ff969a00700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff964100f90 0x7ff9640731e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff95800e9a0 tx=0x7ff95800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.559+0000 7ff962000700 1 -- 172.21.15.16:0/1210546376 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958009790 con 0x7ff964100f90 2024-04-05T23:47:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.559+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9641a6130 con 0x7ff964100f90 2024-04-05T23:47:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.559+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9641a6690 con 0x7ff964100f90 2024-04-05T23:47:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.559+0000 7ff962000700 1 -- 172.21.15.16:0/1210546376 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff958010430 con 0x7ff964100f90 2024-04-05T23:47:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.559+0000 7ff962000700 1 -- 172.21.15.16:0/1210546376 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958018680 con 0x7ff964100f90 2024-04-05T23:47:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.560+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9440052f0 con 0x7ff964100f90 2024-04-05T23:47:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.562+0000 7ff962000700 1 -- 172.21.15.16:0/1210546376 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7ff95800ca10 con 0x7ff964100f90 2024-04-05T23:47:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.563+0000 7ff962000700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff94c0a1310 0x7ff94c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.564+0000 7ff962000700 1 -- 172.21.15.16:0/1210546376 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8600+0+0 (secure 0 0 0) 0x7ff958014070 con 0x7ff964100f90 2024-04-05T23:47:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.564+0000 7ff969000700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff94c0a1310 0x7ff94c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.564+0000 7ff969000700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff94c0a1310 0x7ff94c0a37d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff964074910 tx=0x7ff950009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.565+0000 7ff962000700 1 -- 172.21.15.16:0/1210546376 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff958052ab0 con 0x7ff964100f90 2024-04-05T23:47:13.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:13 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1766814494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:47:13.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:13 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2504411011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:47:13.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:13 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1082402049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:47:13.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:13 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1766814494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:47:13.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:13 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2504411011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:47:13.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:13 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1082402049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:47:13.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:13 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1766814494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:47:13.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:13 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2504411011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:47:13.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:13 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1082402049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:47:13.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.843+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7ff944005160 con 0x7ff964100f90 2024-04-05T23:47:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.852+0000 7ff962000700 1 -- 172.21.15.16:0/1210546376 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) v1 ==== 81+0+733940 (secure 0 0 0) 0x7ff958099a70 con 0x7ff964100f90 2024-04-05T23:47:13.854 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:47:13.854 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:47:13.854 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 26, 2024-04-05T23:47:13.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.855 INFO:teuthology.orchestra.run.smithi016.stdout: "active_gid": 15240, 2024-04-05T23:47:13.855 INFO:teuthology.orchestra.run.smithi016.stdout: "active_name": "x", 2024-04-05T23:47:13.855 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addrs": { 2024-04-05T23:47:13.855 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-05T23:47:13.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:13.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-05T23:47:13.855 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-04-05T23:47:13.855 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3895420073 2024-04-05T23:47:13.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:13.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-05T23:47:13.856 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-04-05T23:47:13.856 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3895420073 2024-04-05T23:47:13.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:13.856 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:13.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addr": "172.21.15.16:6801/3895420073", 2024-04-05T23:47:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: "active_change": "2024-04-05T23:33:34.593448+0000", 2024-04-05T23:47:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: "active_mgr_features": 4540138322906710015, 2024-04-05T23:47:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: "available": true, 2024-04-05T23:47:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: "standbys": [ 2024-04-05T23:47:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 25259, 2024-04-05T23:47:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "y", 2024-04-05T23:47:13.857 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540138322906710015, 2024-04-05T23:47:13.858 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-04-05T23:47:13.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:13.858 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-04-05T23:47:13.858 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:13.858 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:13.858 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:13.858 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:13.858 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:13.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:13.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.859 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:13.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.859 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.859 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-04-05T23:47:13.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.860 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:13.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:13.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:13.861 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:13.861 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:13.861 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:13.861 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:13.861 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:13.862 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:13.862 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:13.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.862 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.862 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.862 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:13.863 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:13.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.863 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.863 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.863 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.864 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.864 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:13.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:13.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:13.865 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:13.866 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:13.866 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:13.866 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:13.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.867 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:13.867 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:13.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.868 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-04-05T23:47:13.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-04-05T23:47:13.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.868 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-04-05T23:47:13.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.870 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-04-05T23:47:13.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-04-05T23:47:13.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.870 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-04-05T23:47:13.870 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-04-05T23:47:13.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.871 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-04-05T23:47:13.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-04-05T23:47:13.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.872 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.872 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.872 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.872 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.872 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-04-05T23:47:13.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.873 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.873 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-04-05T23:47:13.873 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-04-05T23:47:13.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.873 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.873 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.874 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-04-05T23:47:13.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.874 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.875 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-04-05T23:47:13.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-04-05T23:47:13.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:13.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.875 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-04-05T23:47:13.875 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.875 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.876 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-04-05T23:47:13.876 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.876 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.876 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-04-05T23:47:13.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-04-05T23:47:13.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-04-05T23:47:13.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.878 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-04-05T23:47:13.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-04-05T23:47:13.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:13.879 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.879 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-05T23:47:13.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-04-05T23:47:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-04-05T23:47:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-04-05T23:47:13.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:13.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:13.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:13.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-04-05T23:47:13.882 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:13.882 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:13.882 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:13.882 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-04-05T23:47:13.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-04-05T23:47:13.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.883 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.883 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:13.883 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-04-05T23:47:13.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.884 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.884 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-04-05T23:47:13.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-04-05T23:47:13.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-04-05T23:47:13.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.885 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-04-05T23:47:13.885 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-05T23:47:13.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.885 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.885 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-04-05T23:47:13.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-04-05T23:47:13.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:13.886 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.886 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:13.886 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-05T23:47:13.886 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-05T23:47:13.886 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-05T23:47:13.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-05T23:47:13.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.887 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.887 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-04-05T23:47:13.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-04-05T23:47:13.887 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:13.887 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-04-05T23:47:13.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:13.888 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-04-05T23:47:13.888 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.888 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-05T23:47:13.888 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.888 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.888 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.889 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-04-05T23:47:13.889 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-04-05T23:47:13.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.889 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.889 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.889 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-04-05T23:47:13.889 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.890 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-05T23:47:13.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-05T23:47:13.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.890 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-04-05T23:47:13.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-04-05T23:47:13.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:13.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-04-05T23:47:13.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-04-05T23:47:13.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-04-05T23:47:13.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.891 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-04-05T23:47:13.891 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-05T23:47:13.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.892 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-04-05T23:47:13.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-04-05T23:47:13.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.892 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-04-05T23:47:13.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-04-05T23:47:13.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-05T23:47:13.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.893 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.894 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-04-05T23:47:13.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-04-05T23:47:13.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:13.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:13.894 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-05T23:47:13.894 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-05T23:47:13.894 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-05T23:47:13.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-05T23:47:13.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.895 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:13.895 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:13.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:13.896 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:13.896 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:13.896 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.898 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:13.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:13.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.898 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.898 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:13.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:13.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.900 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.900 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:13.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:13.901 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:13.901 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:13.901 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:13.901 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:13.901 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:13.901 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:13.901 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:13.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.902 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.902 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:13.902 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:13.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.903 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.903 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-04-05T23:47:13.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-04-05T23:47:13.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:13.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:13.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-05T23:47:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-05T23:47:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-05T23:47:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-04-05T23:47:13.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.906 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:13.906 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-04-05T23:47:13.906 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-04-05T23:47:13.906 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap" 2024-04-05T23:47:13.907 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:13.907 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-04-05T23:47:13.907 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.907 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.907 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-04-05T23:47:13.907 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-04-05T23:47:13.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.908 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.908 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.908 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.908 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.908 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.908 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-05T23:47:13.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.909 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.909 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:13.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:13.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:13.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.909 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:13.910 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.910 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.910 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.910 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-05T23:47:13.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.910 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.910 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.911 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-04-05T23:47:13.911 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-04-05T23:47:13.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:13.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:13.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:13.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-05T23:47:13.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-05T23:47:13.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.912 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-04-05T23:47:13.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-04-05T23:47:13.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:13.912 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:13.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.913 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-05T23:47:13.913 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:13.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:13.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:13.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-04-05T23:47:13.914 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:13.914 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:13.914 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:13.915 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-04-05T23:47:13.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-04-05T23:47:13.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:13.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.915 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-05T23:47:13.915 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.916 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-05T23:47:13.916 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.916 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.916 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-04-05T23:47:13.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-04-05T23:47:13.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-04-05T23:47:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-05T23:47:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.918 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-04-05T23:47:13.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-04-05T23:47:13.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:13.918 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.918 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-04-05T23:47:13.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.919 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.919 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.919 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-05T23:47:13.919 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.920 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-04-05T23:47:13.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-04-05T23:47:13.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.920 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.921 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.921 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-05T23:47:13.921 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-05T23:47:13.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.921 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.921 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-04-05T23:47:13.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-04-05T23:47:13.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:13.922 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.922 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.922 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:13.922 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-05T23:47:13.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.923 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.923 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-04-05T23:47:13.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-04-05T23:47:13.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:13.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-04-05T23:47:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-05T23:47:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.925 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-04-05T23:47:13.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-04-05T23:47:13.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.925 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.925 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.925 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:13.925 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-05T23:47:13.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.926 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.926 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-04-05T23:47:13.926 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-04-05T23:47:13.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-05T23:47:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.928 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.928 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-04-05T23:47:13.928 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-04-05T23:47:13.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.928 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.928 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:13.928 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.929 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.929 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.929 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-05T23:47:13.929 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.929 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.929 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.929 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-04-05T23:47:13.930 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-04-05T23:47:13.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.930 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.930 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-05T23:47:13.930 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.930 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.930 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:13.931 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.931 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-04-05T23:47:13.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-04-05T23:47:13.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.932 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-05T23:47:13.932 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-04-05T23:47:13.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.932 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.933 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-04-05T23:47:13.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-04-05T23:47:13.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.933 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-05T23:47:13.933 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.934 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-04-05T23:47:13.934 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.934 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.934 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-04-05T23:47:13.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-04-05T23:47:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-04-05T23:47:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:13.936 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.936 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.936 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-04-05T23:47:13.936 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-04-05T23:47:13.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.937 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-05T23:47:13.937 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.937 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.937 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.937 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-04-05T23:47:13.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.938 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-04-05T23:47:13.938 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-04-05T23:47:13.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-05T23:47:13.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.938 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.939 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.939 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-04-05T23:47:13.939 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.939 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.939 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-04-05T23:47:13.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-04-05T23:47:13.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-05T23:47:13.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-04-05T23:47:13.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.941 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.941 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-04-05T23:47:13.941 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-04-05T23:47:13.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.941 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.942 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-05T23:47:13.942 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.942 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.942 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.942 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-04-05T23:47:13.942 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.942 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.943 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-04-05T23:47:13.943 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-04-05T23:47:13.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.943 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-05T23:47:13.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.944 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-04-05T23:47:13.944 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.944 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.944 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.944 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-04-05T23:47:13.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-04-05T23:47:13.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.945 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.945 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-05T23:47:13.945 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.945 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.945 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.945 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-04-05T23:47:13.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.946 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.946 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-04-05T23:47:13.946 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-04-05T23:47:13.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.946 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-05T23:47:13.947 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.947 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.947 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.947 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:13.947 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.947 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-04-05T23:47:13.948 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-04-05T23:47:13.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.948 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.948 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.948 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-04-05T23:47:13.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-04-05T23:47:13.949 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.949 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.949 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-04-05T23:47:13.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-04-05T23:47:13.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-05T23:47:13.950 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.950 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:13.950 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.950 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.950 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.951 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-04-05T23:47:13.951 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-04-05T23:47:13.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.951 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.951 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-05T23:47:13.951 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.951 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.952 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.952 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-04-05T23:47:13.952 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.952 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.952 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-04-05T23:47:13.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-04-05T23:47:13.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.953 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-05T23:47:13.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.953 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.953 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.953 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-04-05T23:47:13.953 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.954 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-04-05T23:47:13.954 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-04-05T23:47:13.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.955 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:13.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.955 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.955 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-04-05T23:47:13.955 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.955 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.955 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.956 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-04-05T23:47:13.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-04-05T23:47:13.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:13.956 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.956 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.956 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:13.956 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.956 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.957 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-05T23:47:13.957 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.957 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.957 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-04-05T23:47:13.957 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-04-05T23:47:13.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:13.958 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.958 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-04-05T23:47:13.958 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.958 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.958 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.958 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-05T23:47:13.958 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.959 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-04-05T23:47:13.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-04-05T23:47:13.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.959 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-04-05T23:47:13.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.960 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.960 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-05T23:47:13.960 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.960 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.960 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.961 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-04-05T23:47:13.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-04-05T23:47:13.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:13.961 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-04-05T23:47:13.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.961 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.962 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.962 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-04-05T23:47:13.962 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.962 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.962 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-04-05T23:47:13.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-04-05T23:47:13.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.963 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.963 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.963 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.963 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.963 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.963 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.963 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-05T23:47:13.963 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.964 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.964 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-04-05T23:47:13.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-04-05T23:47:13.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:13.964 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.965 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:13.965 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.965 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.965 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.965 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-04-05T23:47:13.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.966 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-04-05T23:47:13.966 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-04-05T23:47:13.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:13.966 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:13.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.967 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.967 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-04-05T23:47:13.967 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.967 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.967 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-04-05T23:47:13.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-04-05T23:47:13.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.968 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.968 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.968 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-05T23:47:13.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.969 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-04-05T23:47:13.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-04-05T23:47:13.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-05T23:47:13.970 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.970 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.971 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:13.972 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:13.972 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:13.972 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:13.972 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:13.972 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:13.972 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:13.972 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:13.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.973 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-04-05T23:47:13.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-04-05T23:47:13.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.973 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-05T23:47:13.974 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.974 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.975 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:13.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:13.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:13.975 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-05T23:47:13.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.976 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:13.976 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:13.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.977 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:13.977 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.977 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.977 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:13.977 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:13.977 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:13.977 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:13.978 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:13.978 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:13.978 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:13.978 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:13.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.978 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.979 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:13.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:13.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:13.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.980 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.980 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:13.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.980 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-05T23:47:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-05T23:47:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-05T23:47:13.982 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.982 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-05T23:47:13.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-05T23:47:13.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-04-05T23:47:13.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.983 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-05T23:47:13.983 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.983 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.984 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-04-05T23:47:13.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-04-05T23:47:13.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:13.984 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.984 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:13.984 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.984 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.985 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-04-05T23:47:13.985 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.985 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-04-05T23:47:13.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-04-05T23:47:13.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:13.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:13.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-05T23:47:13.986 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.987 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-04-05T23:47:13.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-04-05T23:47:13.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:13.987 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.987 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.988 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.988 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.988 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-04-05T23:47:13.988 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.988 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.988 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.989 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-05T23:47:13.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-05T23:47:13.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.989 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.989 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-04-05T23:47:13.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.989 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.990 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:13.990 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-04-05T23:47:13.990 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-04-05T23:47:13.990 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:13.990 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-04-05T23:47:13.990 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.990 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.991 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-04-05T23:47:13.991 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-04-05T23:47:13.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.991 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-04-05T23:47:13.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.992 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.992 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-05T23:47:13.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.992 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-04-05T23:47:13.992 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-04-05T23:47:13.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.993 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.993 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-05T23:47:13.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-05T23:47:13.993 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.994 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.994 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.994 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-04-05T23:47:13.994 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-04-05T23:47:13.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:13.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-05T23:47:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.996 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-04-05T23:47:13.996 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-04-05T23:47:13.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.996 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.996 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.997 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-05T23:47:13.997 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.997 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.997 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.997 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-04-05T23:47:13.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-04-05T23:47:13.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.998 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.998 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:13.998 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:13.998 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:13.998 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:13.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-05T23:47:13.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:13.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:13.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:13.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:13.999 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-04-05T23:47:13.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-04-05T23:47:13.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:13.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:13.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:13.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.000 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.000 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-05T23:47:14.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.001 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-04-05T23:47:14.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-04-05T23:47:14.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.002 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.002 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-05T23:47:14.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.002 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-04-05T23:47:14.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-04-05T23:47:14.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.003 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.003 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-04-05T23:47:14.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-04-05T23:47:14.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.004 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-04-05T23:47:14.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-04-05T23:47:14.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.004 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.005 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.005 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-05T23:47:14.005 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.005 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.006 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-04-05T23:47:14.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-04-05T23:47:14.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.006 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.006 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.007 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-05T23:47:14.007 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.007 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.007 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-04-05T23:47:14.007 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-04-05T23:47:14.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-05T23:47:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.009 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.009 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-04-05T23:47:14.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-04-05T23:47:14.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.010 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.010 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.010 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.010 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-04-05T23:47:14.010 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.011 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-04-05T23:47:14.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-04-05T23:47:14.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.011 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-05T23:47:14.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.012 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-04-05T23:47:14.012 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-04-05T23:47:14.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.012 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.013 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.013 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-05T23:47:14.013 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.014 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-04-05T23:47:14.014 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.014 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.014 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.015 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.016 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.016 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.016 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.016 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.016 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.016 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.016 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.016 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.016 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.017 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.017 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.017 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.017 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.018 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.018 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.018 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.018 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.018 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.018 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.019 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.019 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.019 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.019 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.020 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.020 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.020 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.020 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.020 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.020 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.020 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.020 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.021 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.021 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.022 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.022 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.022 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.022 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.022 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.022 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.023 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-04-05T23:47:14.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-04-05T23:47:14.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.023 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.023 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-04-05T23:47:14.023 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.024 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.024 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.024 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-05T23:47:14.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.024 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-04-05T23:47:14.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-04-05T23:47:14.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.025 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-04-05T23:47:14.025 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.025 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.025 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-05T23:47:14.026 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.026 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.026 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-04-05T23:47:14.027 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.027 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.027 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.027 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-05T23:47:14.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-05T23:47:14.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.027 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.028 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.028 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.028 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.028 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.028 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.029 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-04-05T23:47:14.029 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-05T23:47:14.029 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.029 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.029 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.029 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.029 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.029 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.030 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.030 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.030 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.030 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.030 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-05T23:47:14.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-05T23:47:14.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.031 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.031 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.031 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.031 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.031 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.032 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.032 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-04-05T23:47:14.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-04-05T23:47:14.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.033 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.033 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-05T23:47:14.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-05T23:47:14.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.034 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.035 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.035 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.035 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-04-05T23:47:14.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-05T23:47:14.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.035 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.036 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.036 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.036 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.036 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.036 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-05T23:47:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-05T23:47:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.038 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.038 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.038 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-05T23:47:14.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-05T23:47:14.039 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.039 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.039 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.039 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.039 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.042 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.042 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.042 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.042 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.042 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.042 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-05T23:47:14.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-05T23:47:14.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.043 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.043 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.043 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.043 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.043 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.044 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.044 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.044 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-05T23:47:14.044 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-05T23:47:14.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.044 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.045 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.045 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.045 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.045 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.045 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.045 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.046 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.046 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-05T23:47:14.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-05T23:47:14.046 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.046 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.047 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.047 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.047 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.047 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.047 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.047 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.048 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-05T23:47:14.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-05T23:47:14.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.048 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.048 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.048 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.048 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.048 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-05T23:47:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-05T23:47:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.050 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.050 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.050 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.050 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.050 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.050 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.051 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.051 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-05T23:47:14.051 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-05T23:47:14.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.052 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.052 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.052 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-04-05T23:47:14.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-05T23:47:14.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.053 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-05T23:47:14.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.053 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-05T23:47:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-05T23:47:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.055 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.055 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.055 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.055 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.056 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-04-05T23:47:14.056 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-04-05T23:47:14.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.056 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.056 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.056 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.056 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.056 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.056 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.057 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.057 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.057 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-04-05T23:47:14.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-05T23:47:14.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.058 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-05T23:47:14.058 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.058 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.058 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.059 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-05T23:47:14.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-05T23:47:14.059 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.059 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.059 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.059 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.059 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.060 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-05T23:47:14.061 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-05T23:47:14.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.061 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.061 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.061 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.061 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.062 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.062 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.062 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.062 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-05T23:47:14.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-05T23:47:14.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.062 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.063 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.063 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.063 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.063 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.063 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.063 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.064 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-05T23:47:14.064 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-05T23:47:14.064 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.064 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.065 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.065 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.065 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.065 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.065 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-04-05T23:47:14.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-05T23:47:14.066 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.066 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.066 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.066 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.066 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.067 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-05T23:47:14.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-05T23:47:14.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.067 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-05T23:47:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-05T23:47:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.070 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.070 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.070 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.070 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.070 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.071 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-05T23:47:14.071 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-05T23:47:14.071 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.071 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.071 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.071 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.071 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.071 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.071 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.072 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.072 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.072 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.072 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.072 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-05T23:47:14.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-05T23:47:14.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.073 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.073 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.073 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.073 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.073 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.073 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.074 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.074 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-05T23:47:14.074 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-05T23:47:14.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.074 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.075 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.075 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.075 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.075 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.075 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-05T23:47:14.076 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-05T23:47:14.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.076 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.077 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.077 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-05T23:47:14.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-05T23:47:14.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.077 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.078 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.078 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.078 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.078 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.078 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.078 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.078 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.078 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.078 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.079 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-05T23:47:14.079 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-05T23:47:14.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.079 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.079 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.079 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.080 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.080 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.080 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.080 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.080 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-04-05T23:47:14.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-05T23:47:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.082 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.082 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-05T23:47:14.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-05T23:47:14.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.083 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-05T23:47:14.083 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.083 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.084 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-05T23:47:14.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-05T23:47:14.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.084 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.085 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.085 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.085 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.085 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.085 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-05T23:47:14.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-05T23:47:14.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.086 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-04-05T23:47:14.086 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.086 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.086 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.087 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-05T23:47:14.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-05T23:47:14.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-04-05T23:47:14.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.088 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.088 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.089 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-04-05T23:47:14.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-05T23:47:14.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.089 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.090 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.090 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-05T23:47:14.090 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-05T23:47:14.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-05T23:47:14.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.091 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.091 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.091 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.092 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-04-05T23:47:14.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-05T23:47:14.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.092 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.093 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-04-05T23:47:14.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-05T23:47:14.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.094 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.094 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.095 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.095 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.095 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.095 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-04-05T23:47:14.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-05T23:47:14.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.095 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.096 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.097 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-05T23:47:14.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-05T23:47:14.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.097 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:14.097 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.098 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.098 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-05T23:47:14.098 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-05T23:47:14.099 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.099 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.099 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.099 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.100 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.100 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-05T23:47:14.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-05T23:47:14.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.100 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.102 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-04-05T23:47:14.102 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-04-05T23:47:14.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.102 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.103 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.103 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-04-05T23:47:14.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-04-05T23:47:14.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.104 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.104 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.104 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.105 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-04-05T23:47:14.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-04-05T23:47:14.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.105 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.105 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.106 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-04-05T23:47:14.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.106 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.106 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-04-05T23:47:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-04-05T23:47:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-04-05T23:47:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.107 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-05T23:47:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-05T23:47:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.110 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.111 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.115 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.117 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.117 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.117 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.117 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.117 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.118 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.118 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.118 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-04-05T23:47:14.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-04-05T23:47:14.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.119 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.119 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-04-05T23:47:14.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.120 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.120 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-04-05T23:47:14.120 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-04-05T23:47:14.120 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.120 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.120 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.121 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-05T23:47:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-05T23:47:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-05T23:47:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.122 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-05T23:47:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-05T23:47:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-04-05T23:47:14.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.124 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.124 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.124 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.125 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-05T23:47:14.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-05T23:47:14.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.125 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.125 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.126 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.126 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.126 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-04-05T23:47:14.126 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-04-05T23:47:14.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.127 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.127 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-04-05T23:47:14.127 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.127 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.127 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.127 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.128 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-05T23:47:14.128 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-05T23:47:14.128 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.128 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.128 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.128 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-04-05T23:47:14.129 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.129 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.129 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.129 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-04-05T23:47:14.129 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.129 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.129 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.130 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.130 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-05T23:47:14.130 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-05T23:47:14.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.130 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.130 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.130 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-05T23:47:14.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-05T23:47:14.131 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-05T23:47:14.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.131 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-04-05T23:47:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-04-05T23:47:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-05T23:47:14.134 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.134 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.134 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-04-05T23:47:14.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-04-05T23:47:14.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.135 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.135 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-04-05T23:47:14.135 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.135 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.135 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.136 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.136 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.136 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.136 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.136 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.137 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.137 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.137 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.137 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.137 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.137 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.137 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.137 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.138 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.139 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.139 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.139 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.139 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.139 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.139 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.139 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.140 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.140 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.140 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.140 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.141 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.141 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.141 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.141 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.141 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.141 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.141 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.141 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.142 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.143 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.143 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.143 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.143 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.143 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.143 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.144 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-04-05T23:47:14.144 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-04-05T23:47:14.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.144 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.144 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-04-05T23:47:14.144 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-05T23:47:14.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.145 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.145 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.145 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-04-05T23:47:14.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-04-05T23:47:14.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.146 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.146 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-04-05T23:47:14.146 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.146 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.146 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.146 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-05T23:47:14.147 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.147 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.147 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.147 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-04-05T23:47:14.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-04-05T23:47:14.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.148 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-04-05T23:47:14.148 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.148 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.148 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.148 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-04-05T23:47:14.148 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.148 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.149 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-04-05T23:47:14.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-04-05T23:47:14.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.149 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-05T23:47:14.149 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-05T23:47:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-04-05T23:47:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-04-05T23:47:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-05T23:47:14.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.152 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:14.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:14.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.153 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:14.153 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.153 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.153 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-04-05T23:47:14.153 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.153 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.154 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-04-05T23:47:14.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-04-05T23:47:14.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.154 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.154 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.154 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-04-05T23:47:14.154 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-05T23:47:14.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.155 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.155 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.155 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-04-05T23:47:14.156 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.156 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.156 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.156 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.157 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.157 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.157 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.158 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.158 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.158 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.158 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.158 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.158 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.159 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.160 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.160 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.160 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.161 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.161 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.161 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.161 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.161 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.162 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.162 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.162 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.162 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.162 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.163 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.165 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-04-05T23:47:14.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-04-05T23:47:14.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.165 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-05T23:47:14.165 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.165 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.166 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.166 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.166 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.166 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-04-05T23:47:14.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-04-05T23:47:14.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.167 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-04-05T23:47:14.167 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.167 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.168 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:14.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:14.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:14.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-04-05T23:47:14.170 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-04-05T23:47:14.170 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-04-05T23:47:14.170 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.170 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-04-05T23:47:14.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-04-05T23:47:14.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-04-05T23:47:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-05T23:47:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-04-05T23:47:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-04-05T23:47:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-04-05T23:47:14.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-05T23:47:14.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.173 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.173 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-04-05T23:47:14.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-04-05T23:47:14.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.174 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.174 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.174 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.174 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-04-05T23:47:14.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.175 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:14.175 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:14.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.175 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.176 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-04-05T23:47:14.176 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-04-05T23:47:14.176 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.176 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.176 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-05T23:47:14.176 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.177 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.177 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.177 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.178 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.178 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.178 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.178 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.178 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.179 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.179 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.179 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.179 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.180 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.180 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.181 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.181 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.182 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.182 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.182 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.182 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.182 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.182 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.182 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.182 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.183 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.183 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.183 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.183 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.184 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.184 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.184 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.184 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.184 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.184 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.185 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.185 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-04-05T23:47:14.185 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-04-05T23:47:14.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.186 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.186 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.186 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.186 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-04-05T23:47:14.186 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.186 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-04-05T23:47:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-04-05T23:47:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-04-05T23:47:14.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-04-05T23:47:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-05T23:47:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-05T23:47:14.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-04-05T23:47:14.189 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-05T23:47:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-04-05T23:47:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-04-05T23:47:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.191 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:14.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:14.191 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-04-05T23:47:14.191 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.191 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-05T23:47:14.191 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.192 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-04-05T23:47:14.192 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-04-05T23:47:14.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.192 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.192 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.192 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.193 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.193 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-04-05T23:47:14.193 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.193 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.193 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-04-05T23:47:14.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-04-05T23:47:14.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.194 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.194 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-04-05T23:47:14.194 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.194 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.194 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-05T23:47:14.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.195 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-04-05T23:47:14.196 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.196 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.196 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.196 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.196 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.197 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.197 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.197 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.197 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.197 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.198 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.198 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.198 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.198 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.198 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.198 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.198 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.199 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.199 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.199 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.199 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.199 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.199 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.200 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.200 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.200 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.200 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.201 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.201 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.201 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.201 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.201 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.201 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.201 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.204 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.204 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.204 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.204 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-04-05T23:47:14.205 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.205 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.205 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.205 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.205 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.205 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.205 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.208 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.208 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.208 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.208 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.208 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.209 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.209 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.209 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.209 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.209 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.210 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.210 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.210 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.210 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.211 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.211 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.211 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.211 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.211 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.211 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.211 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.212 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.212 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.212 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.213 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.213 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-04-05T23:47:14.214 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.214 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.214 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.214 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-04-05T23:47:14.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-04-05T23:47:14.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.215 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-04-05T23:47:14.215 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.215 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.215 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.215 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-05T23:47:14.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.216 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-04-05T23:47:14.216 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-04-05T23:47:14.216 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.216 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.216 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-04-05T23:47:14.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.217 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.217 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-05T23:47:14.217 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.217 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.217 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.217 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.218 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.218 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.218 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.218 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.218 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.218 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.219 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.219 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.219 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.219 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.219 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.220 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.221 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.221 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.221 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.221 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.222 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.222 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.222 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.222 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.222 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.222 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.224 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.224 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.225 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.225 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.225 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.225 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.226 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-04-05T23:47:14.226 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.226 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.226 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.226 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-04-05T23:47:14.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-04-05T23:47:14.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.227 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.227 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-04-05T23:47:14.227 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.227 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-04-05T23:47:14.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-05T23:47:14.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.228 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.228 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.229 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.229 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.229 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.230 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.230 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.230 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.230 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.230 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.230 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.230 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.231 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.231 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.231 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.231 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.231 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.231 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.232 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.232 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.232 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.232 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.232 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.233 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.233 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.233 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.234 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.234 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.234 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.234 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.234 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.234 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.234 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.234 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.235 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.235 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.235 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.235 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.236 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-04-05T23:47:14.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-04-05T23:47:14.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.237 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.237 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-04-05T23:47:14.237 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-05T23:47:14.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.238 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-04-05T23:47:14.238 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-04-05T23:47:14.238 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.238 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.238 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.238 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-04-05T23:47:14.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-04-05T23:47:14.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.239 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.240 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-04-05T23:47:14.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-04-05T23:47:14.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.240 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.240 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-04-05T23:47:14.240 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.240 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.240 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.241 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-04-05T23:47:14.241 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.241 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.241 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.241 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-04-05T23:47:14.241 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-04-05T23:47:14.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.242 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.242 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.242 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-04-05T23:47:14.242 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.243 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.243 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-04-05T23:47:14.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-04-05T23:47:14.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-04-05T23:47:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-05T23:47:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-05T23:47:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-04-05T23:47:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.246 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.247 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.249 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.250 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.250 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.250 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.250 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.250 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.251 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.251 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.251 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.251 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.251 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.251 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.251 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.251 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.252 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.252 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.252 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.253 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.253 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.253 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.254 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-04-05T23:47:14.254 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.254 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.254 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.254 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.254 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.255 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.255 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.255 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.255 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.255 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.255 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.256 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.256 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.256 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.256 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.256 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.257 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.258 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.258 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.259 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.260 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.260 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.260 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.260 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.260 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.260 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.261 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.261 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.261 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.261 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.262 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.262 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.262 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.263 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-04-05T23:47:14.263 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.263 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.263 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.263 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.264 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.264 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.264 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.264 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.264 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.265 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.265 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.265 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.265 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.265 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.265 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.265 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.265 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.266 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.266 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.266 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.266 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.266 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.267 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.267 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.267 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.267 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.267 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.267 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.267 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.269 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.269 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.269 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.269 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.269 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.269 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.269 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.269 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.269 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.270 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.270 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.270 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.271 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.271 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.271 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.271 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.271 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-05T23:47:14.272 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.272 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.272 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.272 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-04-05T23:47:14.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-04-05T23:47:14.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.273 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-05T23:47:14.273 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.274 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.275 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.275 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.275 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.275 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.275 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.275 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.276 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.276 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.276 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.276 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.276 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.276 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.276 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.277 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.277 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.277 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.277 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.277 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.277 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.278 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.278 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.278 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.279 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.279 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.279 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.279 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.279 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.279 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.279 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.279 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.279 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.280 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.280 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.280 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.280 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.280 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.280 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.280 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.280 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.281 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.281 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.281 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.281 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.281 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.281 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.281 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.282 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-04-05T23:47:14.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-05T23:47:14.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.282 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.283 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.283 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-05T23:47:14.283 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-04-05T23:47:14.283 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-04-05T23:47:14.284 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.284 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-04-05T23:47:14.284 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.284 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-04-05T23:47:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.286 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.287 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.287 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.287 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.287 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.288 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.288 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.288 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.288 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.288 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.289 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.290 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.290 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.291 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.291 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.291 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.291 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.291 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.291 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.291 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.292 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.292 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.292 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.292 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.293 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.293 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.293 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-04-05T23:47:14.294 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.294 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.294 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.294 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.295 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.295 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.295 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.296 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.297 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.297 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.298 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.299 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.299 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.300 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.300 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.300 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.300 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.300 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.300 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.302 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.302 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.302 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-04-05T23:47:14.303 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.303 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.303 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.303 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.304 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.304 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.304 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.304 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.304 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.304 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.304 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.305 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.305 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.305 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.305 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.305 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.305 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.305 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.305 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.306 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.306 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.306 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.306 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.307 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.307 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.308 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.308 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.308 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.309 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.310 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:14.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.312 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.312 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:14.312 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.312 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.312 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.312 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.313 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.313 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.313 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-04-05T23:47:14.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-04-05T23:47:14.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:14.313 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.314 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.314 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-05T23:47:14.314 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-04-05T23:47:14.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-04-05T23:47:14.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-05T23:47:14.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.314 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-04-05T23:47:14.315 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.315 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.315 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.315 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-04-05T23:47:14.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-04-05T23:47:14.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.316 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.316 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.316 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.316 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-05T23:47:14.317 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-05T23:47:14.317 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-05T23:47:14.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.317 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.318 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.318 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.318 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.318 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.318 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.319 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.319 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.319 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.319 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.319 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.319 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.320 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.320 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.320 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.320 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.320 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.320 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.320 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.320 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.321 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.321 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.321 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.322 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.322 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.322 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.322 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.322 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.322 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.323 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.323 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.323 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.323 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.324 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.324 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.324 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.324 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.324 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.324 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.324 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.325 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.325 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.325 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.326 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.326 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.326 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.326 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.326 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.326 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.326 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.327 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-04-05T23:47:14.327 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-04-05T23:47:14.327 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.327 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.327 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.327 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-04-05T23:47:14.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.328 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.328 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-04-05T23:47:14.328 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.328 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.328 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.328 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:14.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:14.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.329 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.329 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:14.329 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-04-05T23:47:14.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.330 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.330 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-04-05T23:47:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-04-05T23:47:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-04-05T23:47:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.332 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.332 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.332 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.332 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.332 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.332 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-04-05T23:47:14.333 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-04-05T23:47:14.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.333 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.333 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.333 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.334 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-05T23:47:14.334 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-05T23:47:14.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.334 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.334 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.334 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.335 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.335 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.335 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.335 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.335 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.336 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.336 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.336 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.336 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.336 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.336 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.337 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.338 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.338 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.338 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.338 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.339 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.340 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.340 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.340 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.340 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.340 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.340 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.340 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.340 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.340 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.341 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.341 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.341 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.342 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.342 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.342 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.342 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.342 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.342 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-04-05T23:47:14.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-04-05T23:47:14.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.343 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.343 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.343 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.343 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.344 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.344 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.344 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.344 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.344 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-05T23:47:14.344 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-05T23:47:14.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-04-05T23:47:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.346 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-04-05T23:47:14.346 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-04-05T23:47:14.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:14.346 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.346 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-04-05T23:47:14.347 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.347 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.347 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.347 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.347 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.347 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-05T23:47:14.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-05T23:47:14.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-05T23:47:14.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.348 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.348 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-05T23:47:14.349 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.349 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.349 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.349 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-05T23:47:14.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-05T23:47:14.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.349 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.350 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-04-05T23:47:14.350 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.350 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.350 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.350 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-05T23:47:14.350 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.350 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.351 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-04-05T23:47:14.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-04-05T23:47:14.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-04-05T23:47:14.351 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.352 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.352 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.352 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.352 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.352 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-05T23:47:14.353 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-05T23:47:14.353 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.353 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.353 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-04-05T23:47:14.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.353 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-04-05T23:47:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-04-05T23:47:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-05T23:47:14.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-05T23:47:14.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.355 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.355 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-05T23:47:14.355 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-05T23:47:14.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-05T23:47:14.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.355 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.356 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.356 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.356 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.356 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.356 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-05T23:47:14.357 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.357 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.357 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.357 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.357 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.357 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.357 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.357 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.358 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.358 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.358 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.358 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.358 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.358 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.358 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.359 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.359 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.359 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.359 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.359 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.359 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.360 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.360 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.360 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.360 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.360 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.360 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.360 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.361 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.361 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.361 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.361 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.361 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.361 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.361 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.362 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.362 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.362 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.362 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.362 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.362 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.362 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.362 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.363 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.363 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.363 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.363 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.363 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.363 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.363 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.364 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.364 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.364 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.364 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.364 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.364 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.365 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.365 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.365 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-04-05T23:47:14.365 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-04-05T23:47:14.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-04-05T23:47:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-04-05T23:47:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-04-05T23:47:14.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-04-05T23:47:14.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.369 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.369 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.369 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.369 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.369 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.369 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.369 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.370 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.370 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.370 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.370 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-04-05T23:47:14.371 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.371 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.371 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-04-05T23:47:14.371 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-04-05T23:47:14.371 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.371 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.372 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.372 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.372 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.372 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.372 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.372 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.373 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-05T23:47:14.373 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-05T23:47:14.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.373 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.373 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.373 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.373 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.373 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.374 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.374 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.374 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.374 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.374 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.374 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.374 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.375 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.375 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.375 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.375 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.375 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.375 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.375 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.376 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.376 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.376 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.376 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.376 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.376 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.376 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.376 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.377 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.377 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.377 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.377 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.377 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.378 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.378 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.378 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.378 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.378 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.378 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.379 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.379 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.379 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.379 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.380 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.380 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.380 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.380 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.380 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.380 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.380 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.380 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.382 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.382 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.382 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.382 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.382 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.383 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.383 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-05T23:47:14.383 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-05T23:47:14.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.383 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.383 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.383 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.384 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.384 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.384 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.384 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.384 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-05T23:47:14.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-05T23:47:14.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.385 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.386 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.386 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.386 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.386 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-04-05T23:47:14.386 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.387 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.387 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.387 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.387 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.387 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.387 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.387 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.389 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.389 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.389 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.389 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.389 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.390 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.390 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.390 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.390 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.390 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.390 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.390 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.390 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.391 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.391 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.391 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.392 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.392 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.392 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.392 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.392 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.392 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.392 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.392 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.393 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.393 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.393 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.393 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.393 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.393 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.394 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.394 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.394 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.394 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.394 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.394 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.395 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.395 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-04-05T23:47:14.395 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.395 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.396 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.396 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.396 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.396 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.396 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.396 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.397 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.397 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.397 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.397 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.397 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.397 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.397 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.397 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.398 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.398 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.398 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.398 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.398 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.399 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.399 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.399 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.400 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.400 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.401 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.401 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.401 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.401 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.401 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.401 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.401 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.401 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.402 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.402 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.402 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.404 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-04-05T23:47:14.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-04-05T23:47:14.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.404 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.404 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-04-05T23:47:14.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.405 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.405 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-05T23:47:14.405 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.405 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.406 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-04-05T23:47:14.406 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.406 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.406 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.406 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.407 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.407 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.407 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.407 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.407 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.407 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.407 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.407 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.409 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.409 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.409 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.409 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.410 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.410 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.410 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.411 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.411 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.411 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.411 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.411 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.412 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.412 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.412 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.412 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.412 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.412 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.412 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.412 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.412 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.413 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.413 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.413 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.413 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.414 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.414 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.414 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.414 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.414 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-04-05T23:47:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-04-05T23:47:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.415 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-04-05T23:47:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-04-05T23:47:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.417 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.417 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.417 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-04-05T23:47:14.417 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.418 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.418 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-04-05T23:47:14.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-04-05T23:47:14.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.418 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.419 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.419 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.419 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-04-05T23:47:14.420 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-04-05T23:47:14.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.420 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.420 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.420 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.420 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.421 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.421 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.421 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.421 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.421 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-04-05T23:47:14.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-04-05T23:47:14.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:14.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.422 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.422 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.422 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.422 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.422 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.423 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-04-05T23:47:14.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-04-05T23:47:14.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.423 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.423 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.423 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-04-05T23:47:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-04-05T23:47:14.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.425 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.425 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.425 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.425 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.426 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.426 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.426 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-04-05T23:47:14.426 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-04-05T23:47:14.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.426 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-04-05T23:47:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-04-05T23:47:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-04-05T23:47:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-04-05T23:47:14.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-04-05T23:47:14.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.430 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.430 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.430 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.430 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.430 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.430 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.431 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.431 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.431 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-04-05T23:47:14.431 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-04-05T23:47:14.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.431 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.432 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.432 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.432 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.432 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.432 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.432 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.432 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.433 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-04-05T23:47:14.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-04-05T23:47:14.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.433 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.433 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.434 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.434 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.434 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.434 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.434 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.435 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-05T23:47:14.435 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.435 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.435 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.435 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-04-05T23:47:14.435 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-04-05T23:47:14.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.436 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.436 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.436 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.436 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.436 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.436 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-04-05T23:47:14.436 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.437 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.437 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-04-05T23:47:14.437 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-04-05T23:47:14.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.437 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.438 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.438 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.438 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.438 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-04-05T23:47:14.438 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.438 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.438 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.439 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.439 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.439 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.439 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.439 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.439 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.439 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.440 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.440 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.440 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.440 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.440 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.440 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.440 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.440 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.440 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.441 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.441 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.441 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.441 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.441 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.441 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.443 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.443 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.443 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.443 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.443 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.443 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.444 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.444 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.444 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.444 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.444 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.444 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.444 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.444 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.445 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.445 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.445 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.445 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.445 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.445 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.445 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.445 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.446 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.446 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.446 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.446 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.446 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.446 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-04-05T23:47:14.447 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.450 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.450 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.450 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.450 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.450 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.450 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.450 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.451 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.451 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.451 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.451 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.451 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.451 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.451 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.451 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.452 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.452 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.452 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.452 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.452 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.452 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.452 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.453 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.453 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.453 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.453 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.453 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.453 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.454 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.454 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.454 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.454 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.454 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.454 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.454 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.455 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.455 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.455 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.455 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.455 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.455 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.455 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.455 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.456 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.456 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.456 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.456 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.457 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.457 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.457 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.457 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-04-05T23:47:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.460 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.460 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.460 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.460 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.460 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.460 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.461 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.461 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.461 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.461 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.461 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.462 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.462 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.462 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.462 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.462 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.464 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.464 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.464 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.464 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.464 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.464 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.464 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.464 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.464 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.465 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.465 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.465 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.465 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.465 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.465 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.465 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.466 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.466 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.466 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.466 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.466 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.466 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.466 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.466 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.467 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.467 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.467 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-04-05T23:47:14.468 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.468 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.468 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.468 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-04-05T23:47:14.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-04-05T23:47:14.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.468 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.469 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-05T23:47:14.469 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.469 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.469 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.469 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.469 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.469 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.470 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:14.470 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:14.470 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.470 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.470 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.470 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-04-05T23:47:14.470 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.470 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.470 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.471 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.471 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.471 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.471 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.472 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.472 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.472 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.472 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.472 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.472 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.472 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.473 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.473 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.473 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.473 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.473 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.473 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.474 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.474 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.474 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.474 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.474 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.474 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.475 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.475 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.475 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.475 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.475 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.475 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.476 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.476 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.478 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.478 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.478 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.478 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.479 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.479 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.479 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.479 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-04-05T23:47:14.480 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.480 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.480 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.480 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-04-05T23:47:14.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-04-05T23:47:14.481 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.481 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.481 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.481 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.481 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.481 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-05T23:47:14.481 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.482 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-04-05T23:47:14.482 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-04-05T23:47:14.482 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.482 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.482 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-05T23:47:14.483 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.483 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.483 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.484 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-04-05T23:47:14.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-04-05T23:47:14.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.484 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.484 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.484 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.484 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-05T23:47:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-04-05T23:47:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-04-05T23:47:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-05T23:47:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-04-05T23:47:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-04-05T23:47:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-05T23:47:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-04-05T23:47:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-04-05T23:47:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-04-05T23:47:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-04-05T23:47:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-04-05T23:47:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-04-05T23:47:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-05T23:47:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-05T23:47:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-05T23:47:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-04-05T23:47:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-04-05T23:47:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-04-05T23:47:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-04-05T23:47:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-04-05T23:47:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-04-05T23:47:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-04-05T23:47:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-04-05T23:47:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-04-05T23:47:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-04-05T23:47:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-04-05T23:47:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-04-05T23:47:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-04-05T23:47:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-04-05T23:47:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-04-05T23:47:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-05T23:47:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-04-05T23:47:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-05T23:47:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-04-05T23:47:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-04-05T23:47:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-04-05T23:47:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-05T23:47:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-04-05T23:47:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-04-05T23:47:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-05T23:47:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-04-05T23:47:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-04-05T23:47:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-05T23:47:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-04-05T23:47:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-04-05T23:47:14.539 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-04-05T23:47:14.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:14.539 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-04-05T23:47:14.539 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.539 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-04-05T23:47:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-04-05T23:47:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-04-05T23:47:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-04-05T23:47:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-04-05T23:47:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-04-05T23:47:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-04-05T23:47:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-04-05T23:47:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-04-05T23:47:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-05T23:47:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 25276, 2024-04-05T23:47:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-04-05T23:47:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540138322906710015, 2024-04-05T23:47:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-04-05T23:47:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-04-05T23:47:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-04-05T23:47:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-04-05T23:47:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-04-05T23:47:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-04-05T23:47:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-04-05T23:47:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-04-05T23:47:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-04-05T23:47:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-04-05T23:47:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-04-05T23:47:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-04-05T23:47:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-04-05T23:47:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-04-05T23:47:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-04-05T23:47:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-04-05T23:47:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-04-05T23:47:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-04-05T23:47:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-04-05T23:47:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-04-05T23:47:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-04-05T23:47:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-04-05T23:47:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-04-05T23:47:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-04-05T23:47:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-04-05T23:47:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-05T23:47:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-04-05T23:47:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-04-05T23:47:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-04-05T23:47:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-04-05T23:47:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-04-05T23:47:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-04-05T23:47:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-04-05T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-04-05T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-04-05T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-04-05T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-04-05T23:47:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-05T23:47:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-04-05T23:47:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-04-05T23:47:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-05T23:47:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-05T23:47:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-05T23:47:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-05T23:47:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-04-05T23:47:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-04-05T23:47:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-04-05T23:47:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-04-05T23:47:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-05T23:47:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-04-05T23:47:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-04-05T23:47:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-04-05T23:47:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-05T23:47:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-05T23:47:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-04-05T23:47:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-04-05T23:47:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-04-05T23:47:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-04-05T23:47:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-04-05T23:47:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-04-05T23:47:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-05T23:47:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-04-05T23:47:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-04-05T23:47:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-04-05T23:47:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-04-05T23:47:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-05T23:47:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-04-05T23:47:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-04-05T23:47:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-05T23:47:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-05T23:47:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-05T23:47:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-05T23:47:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-04-05T23:47:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-04-05T23:47:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-05T23:47:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-05T23:47:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-05T23:47:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-04-05T23:47:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-04-05T23:47:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-04-05T23:47:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap" 2024-04-05T23:47:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-04-05T23:47:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-04-05T23:47:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-04-05T23:47:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-05T23:47:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-05T23:47:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-04-05T23:47:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-04-05T23:47:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-05T23:47:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-05T23:47:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-04-05T23:47:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-04-05T23:47:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-05T23:47:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-04-05T23:47:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-04-05T23:47:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-04-05T23:47:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-05T23:47:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-05T23:47:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-04-05T23:47:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-04-05T23:47:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-04-05T23:47:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-05T23:47:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-04-05T23:47:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-04-05T23:47:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-04-05T23:47:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-05T23:47:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-04-05T23:47:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-04-05T23:47:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-05T23:47:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-05T23:47:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-04-05T23:47:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-04-05T23:47:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-05T23:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-04-05T23:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-04-05T23:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-04-05T23:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-05T23:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-04-05T23:47:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-04-05T23:47:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-05T23:47:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-04-05T23:47:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-04-05T23:47:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-05T23:47:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-04-05T23:47:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-04-05T23:47:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-05T23:47:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-04-05T23:47:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-04-05T23:47:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-05T23:47:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-04-05T23:47:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-04-05T23:47:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-05T23:47:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-04-05T23:47:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-04-05T23:47:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-04-05T23:47:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-05T23:47:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-04-05T23:47:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-04-05T23:47:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-04-05T23:47:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-04-05T23:47:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-04-05T23:47:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-04-05T23:47:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.655 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-05T23:47:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-04-05T23:47:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-04-05T23:47:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-04-05T23:47:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-05T23:47:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-04-05T23:47:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-04-05T23:47:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-04-05T23:47:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-05T23:47:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-04-05T23:47:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-04-05T23:47:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-04-05T23:47:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-05T23:47:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-04-05T23:47:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-04-05T23:47:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-04-05T23:47:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-05T23:47:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-04-05T23:47:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-04-05T23:47:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-04-05T23:47:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-05T23:47:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-04-05T23:47:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-04-05T23:47:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-04-05T23:47:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-05T23:47:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-04-05T23:47:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-04-05T23:47:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-04-05T23:47:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-04-05T23:47:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-04-05T23:47:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-04-05T23:47:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-05T23:47:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-04-05T23:47:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-04-05T23:47:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-05T23:47:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-04-05T23:47:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-04-05T23:47:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-04-05T23:47:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-05T23:47:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-04-05T23:47:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-04-05T23:47:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-04-05T23:47:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-04-05T23:47:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-04-05T23:47:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-04-05T23:47:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-05T23:47:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-04-05T23:47:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-04-05T23:47:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-04-05T23:47:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-05T23:47:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-04-05T23:47:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-04-05T23:47:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-04-05T23:47:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-05T23:47:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-04-05T23:47:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-04-05T23:47:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-04-05T23:47:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-04-05T23:47:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-04-05T23:47:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-04-05T23:47:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-05T23:47:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-04-05T23:47:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-04-05T23:47:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-04-05T23:47:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-04-05T23:47:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-04-05T23:47:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-04-05T23:47:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-04-05T23:47:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-04-05T23:47:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-05T23:47:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-04-05T23:47:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-04-05T23:47:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-05T23:47:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-04-05T23:47:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-04-05T23:47:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-05T23:47:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-05T23:47:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.714 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.714 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.714 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.714 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.714 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-05T23:47:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-05T23:47:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.715 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.716 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-05T23:47:14.716 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.716 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-05T23:47:14.716 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-05T23:47:14.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.716 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.717 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-04-05T23:47:14.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.717 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-05T23:47:14.717 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.718 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-04-05T23:47:14.718 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-04-05T23:47:14.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.718 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.719 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.719 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.719 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.719 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-04-05T23:47:14.719 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.720 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-04-05T23:47:14.720 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-04-05T23:47:14.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.720 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.720 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.721 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.721 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.721 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.721 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-05T23:47:14.721 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.721 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.721 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.722 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-04-05T23:47:14.722 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-04-05T23:47:14.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.722 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.723 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.723 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.723 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-04-05T23:47:14.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.723 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.723 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-05T23:47:14.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-05T23:47:14.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.724 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.724 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-04-05T23:47:14.724 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.725 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-04-05T23:47:14.725 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-04-05T23:47:14.725 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-04-05T23:47:14.725 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.726 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.726 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-04-05T23:47:14.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-04-05T23:47:14.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.727 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-04-05T23:47:14.727 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.727 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.727 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-05T23:47:14.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.728 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.728 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-04-05T23:47:14.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-04-05T23:47:14.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.729 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-05T23:47:14.729 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.729 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.729 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-05T23:47:14.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.730 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-04-05T23:47:14.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-04-05T23:47:14.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.730 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.730 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.731 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-05T23:47:14.731 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.732 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-04-05T23:47:14.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-04-05T23:47:14.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.732 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.732 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.732 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.732 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.733 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.733 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-05T23:47:14.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.734 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-04-05T23:47:14.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-04-05T23:47:14.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.735 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-05T23:47:14.735 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.736 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.736 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-04-05T23:47:14.736 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-04-05T23:47:14.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.736 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.736 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.737 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.737 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.737 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-05T23:47:14.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.738 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-04-05T23:47:14.738 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-04-05T23:47:14.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.738 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.739 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.739 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-05T23:47:14.739 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.739 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-04-05T23:47:14.739 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-04-05T23:47:14.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.740 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-04-05T23:47:14.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.740 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-04-05T23:47:14.740 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.741 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-04-05T23:47:14.741 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-04-05T23:47:14.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.741 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.741 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.742 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.742 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-05T23:47:14.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.742 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.743 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-04-05T23:47:14.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-04-05T23:47:14.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.743 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.743 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.744 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.744 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.744 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-05T23:47:14.744 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.744 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.745 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-04-05T23:47:14.745 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-04-05T23:47:14.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.745 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.745 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.745 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.745 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.746 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-05T23:47:14.746 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.746 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.747 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-04-05T23:47:14.747 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-04-05T23:47:14.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.747 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.747 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.747 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.748 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-04-05T23:47:14.748 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.748 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-04-05T23:47:14.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-04-05T23:47:14.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.749 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.749 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.750 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-05T23:47:14.750 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.750 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-04-05T23:47:14.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-04-05T23:47:14.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.751 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.751 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.751 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-05T23:47:14.752 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.752 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.752 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.752 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.752 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-04-05T23:47:14.753 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.753 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.753 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.753 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.754 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.754 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.754 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.755 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.756 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.756 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.756 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.757 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.757 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.757 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.758 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.758 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.759 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.760 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.760 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.760 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.761 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.761 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.761 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.761 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.762 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.763 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.763 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.763 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.763 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.763 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.763 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.763 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.764 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.764 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.764 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.764 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.765 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.765 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.765 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.766 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.766 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-04-05T23:47:14.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-04-05T23:47:14.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.767 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-04-05T23:47:14.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.767 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-05T23:47:14.768 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.768 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-04-05T23:47:14.768 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-04-05T23:47:14.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.768 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.769 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-04-05T23:47:14.769 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-05T23:47:14.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.769 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-04-05T23:47:14.770 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.770 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.771 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.771 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-05T23:47:14.771 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-05T23:47:14.771 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.771 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.772 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.772 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.772 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.772 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.772 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.772 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-04-05T23:47:14.773 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-05T23:47:14.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.773 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.773 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.774 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.774 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.774 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.774 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-05T23:47:14.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-05T23:47:14.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.775 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.775 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.775 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.775 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.775 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.776 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.776 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-04-05T23:47:14.776 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-04-05T23:47:14.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.776 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.777 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.777 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.777 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.777 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.777 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.777 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.778 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.778 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-05T23:47:14.778 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-05T23:47:14.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.779 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.779 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.779 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.779 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-04-05T23:47:14.780 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-05T23:47:14.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.780 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.780 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.781 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.781 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.781 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.781 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.781 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.781 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-05T23:47:14.782 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-05T23:47:14.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.782 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.782 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.782 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.782 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.782 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.783 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.783 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.783 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.783 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.783 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-05T23:47:14.783 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-05T23:47:14.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.784 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-05T23:47:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-05T23:47:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.785 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.786 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.786 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.786 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.786 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.786 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.786 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.787 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-05T23:47:14.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-05T23:47:14.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.787 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.788 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.788 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-05T23:47:14.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-05T23:47:14.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.789 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.789 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.789 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.789 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.790 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.790 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.790 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.790 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.790 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-05T23:47:14.790 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-05T23:47:14.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.791 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.791 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.791 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.791 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.792 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.792 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.792 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-05T23:47:14.792 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-05T23:47:14.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.792 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.793 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.793 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.793 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.793 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.793 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.793 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.794 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-05T23:47:14.794 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-05T23:47:14.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.794 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.795 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.795 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.795 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.795 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.795 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.795 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.796 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-04-05T23:47:14.796 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-05T23:47:14.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.796 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-05T23:47:14.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.797 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.797 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.797 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.797 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.797 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.797 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.798 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-05T23:47:14.798 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-05T23:47:14.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.798 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.798 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.799 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.799 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.799 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.799 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-04-05T23:47:14.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-04-05T23:47:14.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.800 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.800 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.800 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.800 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.800 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.801 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.801 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.801 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.801 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-04-05T23:47:14.801 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-05T23:47:14.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.802 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-05T23:47:14.802 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.802 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.802 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.803 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.803 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.803 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-05T23:47:14.803 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-05T23:47:14.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.804 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.804 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.804 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.804 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.804 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.804 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.804 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.805 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.805 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-05T23:47:14.805 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-05T23:47:14.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.806 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.806 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.806 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.806 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.806 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.807 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-05T23:47:14.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-05T23:47:14.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.808 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.808 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.808 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.808 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-05T23:47:14.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-05T23:47:14.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-04-05T23:47:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-05T23:47:14.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.811 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-05T23:47:14.812 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-05T23:47:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-05T23:47:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-05T23:47:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.815 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-05T23:47:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-05T23:47:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-05T23:47:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-05T23:47:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-05T23:47:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-05T23:47:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-05T23:47:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-05T23:47:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-05T23:47:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-05T23:47:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-05T23:47:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-05T23:47:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-04-05T23:47:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-05T23:47:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-05T23:47:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-05T23:47:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-05T23:47:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-05T23:47:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-05T23:47:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-05T23:47:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-05T23:47:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-04-05T23:47:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-05T23:47:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-05T23:47:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-04-05T23:47:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-04-05T23:47:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-05T23:47:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-05T23:47:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-05T23:47:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-05T23:47:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-04-05T23:47:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-05T23:47:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-04-05T23:47:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-05T23:47:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-04-05T23:47:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-05T23:47:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-05T23:47:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-05T23:47:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-05T23:47:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-05T23:47:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-05T23:47:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-05T23:47:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-04-05T23:47:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-04-05T23:47:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-04-05T23:47:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-04-05T23:47:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-04-05T23:47:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-04-05T23:47:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-04-05T23:47:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-04-05T23:47:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-04-05T23:47:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-04-05T23:47:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-05T23:47:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-05T23:47:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-04-05T23:47:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-04-05T23:47:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-04-05T23:47:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-04-05T23:47:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-04-05T23:47:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-05T23:47:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-05T23:47:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-05T23:47:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-05T23:47:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-05T23:47:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-04-05T23:47:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-05T23:47:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-05T23:47:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-04-05T23:47:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-04-05T23:47:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-04-05T23:47:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-05T23:47:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-05T23:47:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-04-05T23:47:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-04-05T23:47:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-05T23:47:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-05T23:47:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-05T23:47:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-05T23:47:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-05T23:47:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-04-05T23:47:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-04-05T23:47:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-05T23:47:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-04-05T23:47:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-04-05T23:47:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-04-05T23:47:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-04-05T23:47:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-04-05T23:47:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-04-05T23:47:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-05T23:47:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-04-05T23:47:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-04-05T23:47:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-04-05T23:47:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-05T23:47:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-04-05T23:47:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-04-05T23:47:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-04-05T23:47:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-04-05T23:47:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-04-05T23:47:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-04-05T23:47:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-05T23:47:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-05T23:47:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-04-05T23:47:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-04-05T23:47:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-05T23:47:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-04-05T23:47:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-04-05T23:47:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-04-05T23:47:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-04-05T23:47:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-05T23:47:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-04-05T23:47:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-04-05T23:47:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-04-05T23:47:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-05T23:47:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-04-05T23:47:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-04-05T23:47:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-04-05T23:47:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-04-05T23:47:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-04-05T23:47:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-04-05T23:47:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-04-05T23:47:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-04-05T23:47:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-04-05T23:47:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-05T23:47:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-04-05T23:47:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-04-05T23:47:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-04-05T23:47:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-05T23:47:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-04-05T23:47:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-04-05T23:47:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-04-05T23:47:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-04-05T23:47:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-04-05T23:47:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-05T23:47:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-04-05T23:47:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-04-05T23:47:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-04-05T23:47:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-04-05T23:47:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-04-05T23:47:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-04-05T23:47:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-04-05T23:47:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-05T23:47:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-05T23:47:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-04-05T23:47:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-05T23:47:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-04-05T23:47:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-04-05T23:47:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-04-05T23:47:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-05T23:47:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-04-05T23:47:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-04-05T23:47:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-04-05T23:47:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-04-05T23:47:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-04-05T23:47:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-04-05T23:47:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-05T23:47:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-04-05T23:47:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-04-05T23:47:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-04-05T23:47:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-04-05T23:47:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-04-05T23:47:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-04-05T23:47:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-05T23:47:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-04-05T23:47:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-04-05T23:47:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-04-05T23:47:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-05T23:47:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-04-05T23:47:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-04-05T23:47:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-04-05T23:47:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-04-05T23:47:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-04-05T23:47:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-05T23:47:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-04-05T23:47:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-04-05T23:47:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-04-05T23:47:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-05T23:47:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-04-05T23:47:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-04-05T23:47:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-04-05T23:47:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-04-05T23:47:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-04-05T23:47:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-04-05T23:47:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-04-05T23:47:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-04-05T23:47:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-04-05T23:47:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-04-05T23:47:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-04-05T23:47:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-04-05T23:47:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-04-05T23:47:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-04-05T23:47:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-05T23:47:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-05T23:47:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-04-05T23:47:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-04-05T23:47:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.013 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.013 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.014 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.014 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.014 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.014 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.014 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.014 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.015 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.015 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.015 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.015 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.015 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.016 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.016 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.016 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.017 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.017 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.017 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-04-05T23:47:15.018 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.018 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.018 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.018 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.018 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.019 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.019 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.019 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.019 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.020 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.020 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.020 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.020 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.020 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.020 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.020 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.020 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.021 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.021 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.021 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.021 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.022 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.022 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.022 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.022 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.022 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.022 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.023 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.023 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.023 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.023 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.023 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.023 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.024 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.024 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.024 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.024 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.024 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.024 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.025 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.025 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.025 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-05T23:47:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-04-05T23:47:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-04-05T23:47:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.028 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.028 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.028 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.028 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.028 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.028 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-05T23:47:15.028 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.029 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.029 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.029 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.029 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.029 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.029 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.030 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.030 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.030 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.030 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.030 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.030 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.030 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.031 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.031 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.031 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.031 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.032 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.033 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.033 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.035 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.035 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.035 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.035 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.035 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.035 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.035 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.036 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.036 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.036 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.037 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.037 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.037 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.037 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.037 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-04-05T23:47:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-05T23:47:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-05T23:47:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-04-05T23:47:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-04-05T23:47:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-04-05T23:47:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.040 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.040 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.040 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.040 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.040 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-04-05T23:47:15.040 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.041 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.041 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.041 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.041 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.041 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.041 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.041 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.042 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.042 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.042 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.042 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.042 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.042 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.042 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.042 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.042 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.043 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.043 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.043 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.046 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.046 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.046 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.046 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.046 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.046 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.046 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.047 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.047 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.047 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.047 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.047 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.047 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.048 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.048 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.048 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.048 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.048 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.048 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.049 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.049 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-04-05T23:47:15.050 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.050 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.050 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.050 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.050 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.051 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.051 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.051 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.051 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.051 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.053 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.054 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.055 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.055 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.055 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.056 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.056 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.056 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.056 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.056 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.056 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.056 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.057 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.057 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.057 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.057 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.057 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.057 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.058 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-04-05T23:47:15.062 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.062 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.062 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.063 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.063 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.063 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.063 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.063 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.063 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.064 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.064 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.064 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.064 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.064 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.064 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.064 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.065 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.065 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.065 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.065 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.066 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.066 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.066 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.066 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.066 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.067 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.067 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.068 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.068 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.068 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.068 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.068 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.068 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.069 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.069 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.069 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.069 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.069 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.069 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.069 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.070 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.070 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.070 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.070 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.070 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.070 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.070 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.070 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.070 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.071 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.071 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.071 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.071 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.071 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:15.071 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:15.071 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.072 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.072 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:15.072 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.072 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.072 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.072 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.072 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.073 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.073 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-04-05T23:47:15.073 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-04-05T23:47:15.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.073 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.074 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-05T23:47:15.074 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-04-05T23:47:15.074 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.074 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.074 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-04-05T23:47:15.074 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-05T23:47:15.074 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.074 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.075 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.075 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.075 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-04-05T23:47:15.075 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.075 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.075 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.076 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-04-05T23:47:15.076 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-04-05T23:47:15.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-05T23:47:15.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.077 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.077 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-05T23:47:15.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-05T23:47:15.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.078 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.078 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.078 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.078 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.078 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.078 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.078 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.078 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.079 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.079 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.079 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.079 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.080 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.082 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.082 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.082 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.082 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.083 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.083 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.084 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.084 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.084 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.085 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.085 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.085 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.085 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.085 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.085 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.085 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.086 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.086 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.086 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-04-05T23:47:15.088 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-04-05T23:47:15.088 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.088 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.088 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.088 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-04-05T23:47:15.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.089 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-04-05T23:47:15.089 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.089 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.089 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.089 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:15.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:15.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.090 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.090 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:15.090 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.090 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.090 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-04-05T23:47:15.091 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.091 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.091 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-04-05T23:47:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-04-05T23:47:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-04-05T23:47:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.094 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-04-05T23:47:15.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-04-05T23:47:15.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.094 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.095 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-05T23:47:15.095 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-05T23:47:15.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.095 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.095 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.096 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.096 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.096 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.097 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.097 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.097 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.097 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.097 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.097 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.097 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.098 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.098 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.098 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.098 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.099 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.099 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.099 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.100 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.100 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.100 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.100 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.100 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.100 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.101 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.101 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.101 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.101 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.101 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.101 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.101 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.101 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.102 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.102 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.102 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.102 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.103 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.103 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.103 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.103 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.104 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-04-05T23:47:15.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-04-05T23:47:15.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.105 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.105 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.105 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.105 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.105 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.105 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-05T23:47:15.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-05T23:47:15.106 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.106 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-04-05T23:47:15.106 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.107 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.107 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.107 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.107 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-04-05T23:47:15.107 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-04-05T23:47:15.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.107 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-04-05T23:47:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-05T23:47:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-05T23:47:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-05T23:47:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-05T23:47:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-05T23:47:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-05T23:47:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-04-05T23:47:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-05T23:47:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-04-05T23:47:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-04-05T23:47:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-04-05T23:47:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-05T23:47:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-05T23:47:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-04-05T23:47:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-04-05T23:47:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-04-05T23:47:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-05T23:47:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-05T23:47:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-05T23:47:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-05T23:47:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-05T23:47:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-05T23:47:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.121 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.122 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.122 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.122 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.122 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.122 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.122 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.122 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.123 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.123 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.123 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.123 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.123 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.124 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.124 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.124 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.124 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.124 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.124 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.124 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.124 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.125 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.125 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.125 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.126 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-04-05T23:47:15.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-04-05T23:47:15.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.127 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.127 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:15.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.128 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.128 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.128 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.128 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-04-05T23:47:15.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-04-05T23:47:15.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.129 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.129 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.129 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.130 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.130 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.130 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.130 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.130 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-04-05T23:47:15.130 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-04-05T23:47:15.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.131 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.131 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.131 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-04-05T23:47:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.132 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-04-05T23:47:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-04-05T23:47:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.133 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-05T23:47:15.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-05T23:47:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.135 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.136 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.136 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.136 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.137 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.137 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.137 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.137 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.137 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.137 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.138 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.138 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.138 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.139 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.139 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.139 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.139 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.139 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.140 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.140 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.141 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.141 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.141 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.141 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.141 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.141 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.142 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.142 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.142 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.142 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.143 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.143 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.143 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.143 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.144 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.144 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.144 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-05T23:47:15.144 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-05T23:47:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-05T23:47:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-05T23:47:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.147 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-04-05T23:47:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.148 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.149 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.151 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.153 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.154 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.155 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-04-05T23:47:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.158 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.158 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.158 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.158 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.158 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.159 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.159 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.159 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.159 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.159 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.159 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.160 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.160 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.160 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.160 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.161 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.161 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.161 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.162 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.162 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.162 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.162 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.163 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.163 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.163 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.163 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.163 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.163 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.165 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.165 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.165 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.166 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.166 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.166 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.166 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-04-05T23:47:15.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-04-05T23:47:15.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.167 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-04-05T23:47:15.167 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.167 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-05T23:47:15.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.168 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.168 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-04-05T23:47:15.169 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.169 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.169 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.169 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.170 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.170 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.170 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.170 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.170 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.170 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.170 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.170 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.171 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.171 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.171 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.171 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.171 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.173 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.173 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.173 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.174 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.174 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.174 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.174 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.174 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.174 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.175 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.175 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.175 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.175 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.176 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.176 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.176 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.176 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.177 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.177 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.177 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.177 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-04-05T23:47:15.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-04-05T23:47:15.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.178 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.178 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.178 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.178 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-04-05T23:47:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-04-05T23:47:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-04-05T23:47:15.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.180 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.181 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-04-05T23:47:15.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-04-05T23:47:15.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.181 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.182 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.182 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-04-05T23:47:15.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-04-05T23:47:15.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.183 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.183 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.184 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.184 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-04-05T23:47:15.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-04-05T23:47:15.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.184 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.184 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.185 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.186 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-04-05T23:47:15.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-04-05T23:47:15.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.186 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.186 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.186 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.186 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.187 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-04-05T23:47:15.187 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-04-05T23:47:15.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.188 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.188 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.188 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.189 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-04-05T23:47:15.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-04-05T23:47:15.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.190 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.190 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.190 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-04-05T23:47:15.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-04-05T23:47:15.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.191 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:15.191 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-04-05T23:47:15.191 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.192 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.192 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.192 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-04-05T23:47:15.192 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-04-05T23:47:15.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.193 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.193 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.193 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.193 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.193 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.194 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.194 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-04-05T23:47:15.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-04-05T23:47:15.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.194 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.194 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.195 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.195 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.195 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-04-05T23:47:15.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-04-05T23:47:15.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.196 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.196 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.196 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-05T23:47:15.198 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.198 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.198 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.198 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-04-05T23:47:15.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-04-05T23:47:15.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.199 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.199 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.199 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.199 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-04-05T23:47:15.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-04-05T23:47:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-04-05T23:47:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.201 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.201 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.201 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-04-05T23:47:15.201 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.201 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.201 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.202 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.202 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.202 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.202 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.202 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.204 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.205 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.205 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.205 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.205 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.205 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.206 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.207 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.207 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.207 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.207 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.207 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.207 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.207 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.207 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.208 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.208 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.208 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.208 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.209 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.209 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.209 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.209 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.209 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.210 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.210 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.210 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-04-05T23:47:15.210 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.210 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.211 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.211 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.211 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.211 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.211 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.212 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.212 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.212 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.212 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.212 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.212 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.213 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.213 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.213 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.213 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.214 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.214 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.214 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.214 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.215 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.215 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.216 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.216 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.216 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.216 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.216 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.217 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.217 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.217 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.217 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.217 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.217 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.217 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.218 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.218 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.218 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.218 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.218 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.218 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-04-05T23:47:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.221 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.221 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.221 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.221 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.221 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.221 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.223 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.224 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.225 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.225 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.225 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.225 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.226 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.226 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.226 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.226 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.226 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.226 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.227 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.227 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.227 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.227 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.228 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.228 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-04-05T23:47:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-04-05T23:47:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-04-05T23:47:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-05T23:47:15.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.230 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.230 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.230 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.231 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:15.231 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:15.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.231 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.231 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-04-05T23:47:15.232 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.232 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.232 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.232 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.232 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.232 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.232 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.233 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.233 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.234 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.234 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.234 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.234 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.234 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.234 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.234 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.234 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.235 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.235 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.235 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.235 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.236 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.236 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.236 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.237 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.237 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.237 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.239 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.239 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.239 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.240 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.240 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.240 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.240 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.240 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.240 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.241 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-04-05T23:47:15.241 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.241 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.241 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-04-05T23:47:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-04-05T23:47:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.243 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-05T23:47:15.243 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.243 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.243 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-04-05T23:47:15.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-04-05T23:47:15.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.244 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.244 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-05T23:47:15.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.245 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.245 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-04-05T23:47:15.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-04-05T23:47:15.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.245 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.245 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.246 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-05T23:47:15.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.246 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.246 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.247 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-04-05T23:47:15.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-04-05T23:47:15.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.247 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.247 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.248 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.248 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-05T23:47:15.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.248 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-04-05T23:47:15.248 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-04-05T23:47:15.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.249 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.249 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.249 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-05T23:47:15.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.250 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.250 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-04-05T23:47:15.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-04-05T23:47:15.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.250 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.251 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.251 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.251 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.251 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.251 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.251 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.252 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-04-05T23:47:15.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-04-05T23:47:15.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.252 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.252 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.252 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.252 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-04-05T23:47:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-04-05T23:47:15.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.254 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.254 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-05T23:47:15.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.254 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.255 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-05T23:47:15.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-05T23:47:15.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.256 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.256 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.257 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:15.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:15.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-04-05T23:47:15.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-04-05T23:47:15.257 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.258 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.258 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-04-05T23:47:15.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-04-05T23:47:15.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-04-05T23:47:15.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.259 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.260 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.260 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-04-05T23:47:15.260 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-04-05T23:47:15.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.260 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.261 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.261 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.261 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.261 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.261 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.262 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-04-05T23:47:15.262 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-04-05T23:47:15.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.262 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.262 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.262 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.264 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.266 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.266 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.266 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.266 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.266 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.266 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.266 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.267 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.267 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.267 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.267 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.267 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.267 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.267 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.267 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.268 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.268 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.268 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.268 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.269 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.269 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.269 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.269 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.269 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.269 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.271 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.271 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.271 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.271 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.271 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.271 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.271 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.272 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.272 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.272 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.272 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.273 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.273 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.273 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.273 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-04-05T23:47:15.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-04-05T23:47:15.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.274 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.274 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.274 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.274 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.274 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.275 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.275 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-04-05T23:47:15.275 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-04-05T23:47:15.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.275 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.276 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.276 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.276 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.276 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.276 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.276 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.276 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.276 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.277 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-04-05T23:47:15.277 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-04-05T23:47:15.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.277 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.277 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-05T23:47:15.277 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.278 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.278 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.278 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.278 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.278 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-04-05T23:47:15.279 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.279 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.279 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.279 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.279 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.280 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.280 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.280 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.280 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.280 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.281 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.281 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.281 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.281 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.281 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.281 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.281 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.282 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.282 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.282 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.282 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.282 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.283 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.283 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.283 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.283 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.283 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.284 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.284 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.284 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.284 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.284 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.284 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.284 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.285 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.285 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.285 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.285 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.285 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.285 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.285 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.286 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.286 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.286 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.286 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.286 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.287 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.287 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.287 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.287 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-05T23:47:15.288 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.288 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.288 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.288 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.289 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.289 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.289 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.289 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.289 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.289 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.289 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.291 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.291 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.291 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.292 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.292 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.292 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.292 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.293 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.293 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.293 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.294 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.294 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.294 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.294 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.294 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.294 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.294 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.294 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.295 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.295 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.295 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.296 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.296 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.297 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-04-05T23:47:15.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-04-05T23:47:15.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-04-05T23:47:15.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-05T23:47:15.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.298 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-04-05T23:47:15.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-04-05T23:47:15.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.299 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:15.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.299 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-05T23:47:15.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.300 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.300 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-04-05T23:47:15.300 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-04-05T23:47:15.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.300 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.301 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.301 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.301 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.301 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-05T23:47:15.301 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.301 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.301 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.302 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-04-05T23:47:15.302 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.302 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.302 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.303 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-04-05T23:47:15.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-04-05T23:47:15.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:15.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-04-05T23:47:15.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.304 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.304 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-04-05T23:47:15.304 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-04-05T23:47:15.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.305 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.305 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.305 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.305 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.308 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.308 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.310 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.310 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.311 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.311 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.311 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.312 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.312 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.312 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.312 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.312 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.314 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.314 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.314 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.314 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.315 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.315 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.315 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.315 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.315 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.315 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-04-05T23:47:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-04-05T23:47:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.317 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.317 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.318 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-04-05T23:47:15.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-04-05T23:47:15.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.318 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.318 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-04-05T23:47:15.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.319 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.319 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.319 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.319 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-04-05T23:47:15.319 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-04-05T23:47:15.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.320 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.320 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-05T23:47:15.320 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.320 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.320 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.320 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.320 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.321 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.321 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.321 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.321 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.321 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:15.321 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.321 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.322 INFO:teuthology.orchestra.run.smithi016.stdout: "modules": [ 2024-04-05T23:47:15.322 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-05T23:47:15.322 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard", 2024-04-05T23:47:15.322 INFO:teuthology.orchestra.run.smithi016.stdout: "iostat", 2024-04-05T23:47:15.322 INFO:teuthology.orchestra.run.smithi016.stdout: "nfs", 2024-04-05T23:47:15.322 INFO:teuthology.orchestra.run.smithi016.stdout: "restful", 2024-04-05T23:47:15.322 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_schedule" 2024-04-05T23:47:15.322 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.322 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-04-05T23:47:15.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-04-05T23:47:15.323 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.323 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.323 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.323 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:15.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:15.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.324 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:15.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.324 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-04-05T23:47:15.324 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.325 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.325 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.326 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.326 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.326 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.326 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.326 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.326 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.326 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.327 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.327 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.327 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.328 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.328 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.330 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.330 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.330 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.330 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.330 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.330 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.330 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.332 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.332 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.332 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.332 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.332 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-04-05T23:47:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-04-05T23:47:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.334 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.334 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.334 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.334 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.334 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.334 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-04-05T23:47:15.334 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.335 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-04-05T23:47:15.335 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-04-05T23:47:15.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-04-05T23:47:15.336 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-04-05T23:47:15.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.336 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.337 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-04-05T23:47:15.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-04-05T23:47:15.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.337 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.337 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.337 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.337 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.337 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-04-05T23:47:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-04-05T23:47:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-04-05T23:47:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.339 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-04-05T23:47:15.339 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.340 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.340 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-04-05T23:47:15.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-04-05T23:47:15.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.340 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-04-05T23:47:15.341 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.341 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.341 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.341 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-04-05T23:47:15.341 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.341 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.342 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-04-05T23:47:15.342 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-04-05T23:47:15.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.342 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.342 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.342 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.342 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.343 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.343 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.343 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-04-05T23:47:15.343 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.343 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.343 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.343 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-04-05T23:47:15.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-04-05T23:47:15.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.344 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.344 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.344 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.344 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.344 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.344 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.344 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-05T23:47:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-04-05T23:47:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-04-05T23:47:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.346 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.346 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.346 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.346 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.346 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-04-05T23:47:15.346 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.347 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.347 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.347 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-04-05T23:47:15.347 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.347 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.347 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.348 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-04-05T23:47:15.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-04-05T23:47:15.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.348 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.349 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-04-05T23:47:15.349 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.349 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.349 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.349 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-04-05T23:47:15.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-04-05T23:47:15.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.350 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.350 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-04-05T23:47:15.350 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.350 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.350 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.350 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-04-05T23:47:15.350 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-05T23:47:15.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.351 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.351 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-04-05T23:47:15.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-04-05T23:47:15.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:15.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:15.352 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-05T23:47:15.352 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-05T23:47:15.352 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.352 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-05T23:47:15.352 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-05T23:47:15.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.352 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-04-05T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-04-05T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-04-05T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-04-05T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-05T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-04-05T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-04-05T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.355 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.355 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-04-05T23:47:15.355 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.355 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-05T23:47:15.355 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-05T23:47:15.356 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.356 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.356 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-04-05T23:47:15.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-04-05T23:47:15.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.356 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.357 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-04-05T23:47:15.357 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-04-05T23:47:15.357 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-04-05T23:47:15.357 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-04-05T23:47:15.357 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-05T23:47:15.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.357 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.358 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-04-05T23:47:15.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-04-05T23:47:15.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.358 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.358 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-04-05T23:47:15.358 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.359 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.359 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-04-05T23:47:15.359 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-05T23:47:15.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.359 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.359 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-04-05T23:47:15.359 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-04-05T23:47:15.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:15.360 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.360 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:15.360 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-05T23:47:15.360 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-05T23:47:15.360 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.360 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-05T23:47:15.360 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-05T23:47:15.361 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.361 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.361 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.361 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.361 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.361 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.361 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.362 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.362 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.362 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.362 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.362 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.362 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.362 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.362 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.363 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.364 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.364 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.364 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.364 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.364 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.365 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.365 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.365 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.365 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.365 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.366 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.366 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.366 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.366 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.366 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.366 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.366 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.366 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.367 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.367 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.367 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.367 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.367 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.367 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.367 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.368 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.368 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.368 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-04-05T23:47:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-04-05T23:47:15.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.370 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.370 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:15.370 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.370 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.370 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.370 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-05T23:47:15.371 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.371 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.371 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-05T23:47:15.371 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-05T23:47:15.371 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-04-05T23:47:15.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.372 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.372 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.372 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-04-05T23:47:15.372 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-04-05T23:47:15.372 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap" 2024-04-05T23:47:15.372 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.373 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-04-05T23:47:15.373 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.373 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.373 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-04-05T23:47:15.373 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-04-05T23:47:15.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-05T23:47:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.375 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.375 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:15.375 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:15.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.375 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.375 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:15.376 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.376 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.376 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.376 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-05T23:47:15.376 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.376 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.376 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-04-05T23:47:15.377 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-04-05T23:47:15.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.377 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.377 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:15.377 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:15.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.377 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.378 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-05T23:47:15.378 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-05T23:47:15.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.378 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.378 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-04-05T23:47:15.378 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-04-05T23:47:15.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:15.379 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.379 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:15.379 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.379 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.379 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.379 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-05T23:47:15.379 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.380 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.380 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-04-05T23:47:15.380 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.380 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.381 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.381 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-04-05T23:47:15.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-04-05T23:47:15.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.381 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.381 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-05T23:47:15.381 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.381 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.382 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.382 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-05T23:47:15.382 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.382 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.382 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-04-05T23:47:15.382 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-04-05T23:47:15.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.383 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.383 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-04-05T23:47:15.383 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.383 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.383 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.383 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-05T23:47:15.383 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.384 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-04-05T23:47:15.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-04-05T23:47:15.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.384 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.384 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-04-05T23:47:15.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-05T23:47:15.385 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.385 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.385 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.386 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-04-05T23:47:15.386 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-04-05T23:47:15.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.386 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.386 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.386 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.386 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.387 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-05T23:47:15.387 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-05T23:47:15.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.387 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.387 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-04-05T23:47:15.387 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-04-05T23:47:15.388 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.388 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.388 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:15.388 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.388 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.388 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.388 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-05T23:47:15.388 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-04-05T23:47:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-04-05T23:47:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.390 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-04-05T23:47:15.390 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.390 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.390 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.390 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-05T23:47:15.390 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.390 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.391 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-04-05T23:47:15.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-04-05T23:47:15.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.391 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-05T23:47:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-04-05T23:47:15.392 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-04-05T23:47:15.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.393 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.393 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.393 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.393 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-05T23:47:15.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.394 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.394 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-04-05T23:47:15.394 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-04-05T23:47:15.394 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.394 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.395 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.395 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.395 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.395 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.395 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-05T23:47:15.395 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.396 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-04-05T23:47:15.396 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-04-05T23:47:15.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.396 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-05T23:47:15.396 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.396 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.397 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.397 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:15.397 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.397 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.397 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.397 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-04-05T23:47:15.397 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-04-05T23:47:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-05T23:47:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.398 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-04-05T23:47:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-04-05T23:47:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-04-05T23:47:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-05T23:47:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-04-05T23:47:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.400 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-04-05T23:47:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-04-05T23:47:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-04-05T23:47:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.402 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.402 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:15.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.402 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.402 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-04-05T23:47:15.403 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-04-05T23:47:15.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.403 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-05T23:47:15.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.404 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-04-05T23:47:15.404 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.404 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-04-05T23:47:15.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-04-05T23:47:15.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.404 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.405 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-05T23:47:15.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.405 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.405 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-04-05T23:47:15.405 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.406 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.406 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-04-05T23:47:15.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-04-05T23:47:15.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.406 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.406 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-05T23:47:15.406 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.407 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.407 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.407 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-04-05T23:47:15.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.407 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-04-05T23:47:15.408 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-04-05T23:47:15.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.408 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-05T23:47:15.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.408 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-04-05T23:47:15.409 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.409 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.409 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-04-05T23:47:15.409 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-04-05T23:47:15.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.409 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.410 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-05T23:47:15.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-04-05T23:47:15.410 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.410 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.411 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-04-05T23:47:15.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-04-05T23:47:15.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.411 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.411 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-05T23:47:15.411 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.412 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.412 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.412 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-04-05T23:47:15.412 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.412 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.412 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-04-05T23:47:15.412 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-04-05T23:47:15.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.413 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.413 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-05T23:47:15.413 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.413 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.413 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.413 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:15.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.414 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-04-05T23:47:15.414 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-04-05T23:47:15.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.414 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-04-05T23:47:15.415 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.415 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.415 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.415 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-04-05T23:47:15.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.416 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-04-05T23:47:15.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-04-05T23:47:15.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.416 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-05T23:47:15.416 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-05T23:47:15.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.417 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.417 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-04-05T23:47:15.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-04-05T23:47:15.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.418 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-05T23:47:15.418 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.418 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.419 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-04-05T23:47:15.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.419 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-04-05T23:47:15.419 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-04-05T23:47:15.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.420 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-05T23:47:15.420 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.420 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.420 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-04-05T23:47:15.420 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.421 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.421 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-04-05T23:47:15.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-04-05T23:47:15.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.422 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.422 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.422 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-04-05T23:47:15.422 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.422 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-04-05T23:47:15.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-04-05T23:47:15.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.423 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.423 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:15.423 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.423 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.423 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.424 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-05T23:47:15.424 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.424 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.424 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-04-05T23:47:15.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-04-05T23:47:15.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.424 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.425 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-04-05T23:47:15.425 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.425 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-05T23:47:15.425 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.425 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.426 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.426 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-04-05T23:47:15.426 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-04-05T23:47:15.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.426 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.426 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.426 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-04-05T23:47:15.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.427 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.427 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-05T23:47:15.427 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.427 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.427 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.427 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-04-05T23:47:15.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-04-05T23:47:15.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-04-05T23:47:15.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.428 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.428 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.429 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-04-05T23:47:15.429 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.429 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.429 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-04-05T23:47:15.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-04-05T23:47:15.429 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.430 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.430 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.430 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.430 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.430 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.430 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-05T23:47:15.430 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.431 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.431 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-04-05T23:47:15.431 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-04-05T23:47:15.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.431 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.431 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:15.432 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.432 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.432 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.432 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-04-05T23:47:15.432 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.432 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.433 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-04-05T23:47:15.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-04-05T23:47:15.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.433 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:15.433 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.433 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.433 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.434 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-04-05T23:47:15.434 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.434 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.434 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-04-05T23:47:15.434 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-04-05T23:47:15.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.435 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.435 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.435 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.435 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.435 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.435 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-05T23:47:15.435 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.436 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.436 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-04-05T23:47:15.436 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-04-05T23:47:15.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.436 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.436 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-05T23:47:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.437 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.438 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.438 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.438 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.438 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.438 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.438 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.438 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.439 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.439 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.439 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.439 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.439 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.439 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.439 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.439 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.440 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.440 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.440 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-04-05T23:47:15.440 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-04-05T23:47:15.440 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.440 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.441 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.441 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.441 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.441 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.441 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.441 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-05T23:47:15.441 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.442 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.442 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.442 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.442 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.442 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.443 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.443 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.443 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-05T23:47:15.443 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.443 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.443 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.444 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.444 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.444 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.444 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.444 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.444 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.444 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.446 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.446 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.446 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.446 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.447 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.447 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.447 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.447 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.447 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.447 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.447 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-05T23:47:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-05T23:47:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.449 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.449 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-05T23:47:15.449 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.449 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.449 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-05T23:47:15.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-05T23:47:15.450 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.450 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.450 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.450 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-04-05T23:47:15.450 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.450 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.450 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.450 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-05T23:47:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-04-05T23:47:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-04-05T23:47:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.452 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:15.452 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.452 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.452 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.452 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-04-05T23:47:15.452 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.452 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-04-05T23:47:15.453 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-04-05T23:47:15.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.453 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.453 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.453 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:15.453 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.454 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.454 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-05T23:47:15.454 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.454 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.454 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-04-05T23:47:15.454 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-04-05T23:47:15.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.455 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.455 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.455 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.455 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.455 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.455 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-04-05T23:47:15.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.456 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.456 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-05T23:47:15.456 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-05T23:47:15.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.456 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.457 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-04-05T23:47:15.457 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.457 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.457 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-04-05T23:47:15.457 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-04-05T23:47:15.457 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.457 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-04-05T23:47:15.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.458 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-04-05T23:47:15.458 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-04-05T23:47:15.458 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.458 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.459 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-04-05T23:47:15.459 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.459 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.459 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-05T23:47:15.459 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.459 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-04-05T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-04-05T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-05T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.461 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.461 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.461 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-05T23:47:15.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.461 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-04-05T23:47:15.462 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-04-05T23:47:15.462 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.462 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.462 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.462 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.463 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-05T23:47:15.463 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.463 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.463 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.463 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-04-05T23:47:15.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-04-05T23:47:15.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.464 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.464 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.464 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.464 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.464 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.464 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-05T23:47:15.464 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.465 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.465 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-04-05T23:47:15.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-04-05T23:47:15.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.465 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.465 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.466 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.466 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.466 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.466 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-05T23:47:15.466 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.467 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-04-05T23:47:15.467 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-04-05T23:47:15.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.467 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.467 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.469 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.469 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.470 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-05T23:47:15.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.470 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.470 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-04-05T23:47:15.470 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-04-05T23:47:15.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.471 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.471 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.471 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.471 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.471 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.471 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-05T23:47:15.471 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.472 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.472 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-04-05T23:47:15.472 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-04-05T23:47:15.472 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.472 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.472 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.473 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-04-05T23:47:15.473 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.473 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.473 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.473 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-04-05T23:47:15.473 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.473 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.474 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-04-05T23:47:15.474 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-04-05T23:47:15.474 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.474 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.474 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.474 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.475 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.475 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-05T23:47:15.475 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.475 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.475 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-04-05T23:47:15.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-04-05T23:47:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-05T23:47:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.477 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.477 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.477 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-04-05T23:47:15.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-04-05T23:47:15.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.477 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.477 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.478 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.478 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.478 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.478 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.478 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-05T23:47:15.478 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.479 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-04-05T23:47:15.479 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-04-05T23:47:15.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.479 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.480 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.480 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-04-05T23:47:15.480 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.480 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.480 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-04-05T23:47:15.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-04-05T23:47:15.481 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.481 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.481 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.481 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.481 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.481 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-05T23:47:15.481 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.482 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-04-05T23:47:15.482 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-04-05T23:47:15.482 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.482 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.482 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-05T23:47:15.483 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.483 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.483 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-04-05T23:47:15.484 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.484 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.484 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.484 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.485 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.485 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.485 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.485 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.485 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.486 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.486 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.486 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.486 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.486 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.486 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.486 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.486 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.487 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.487 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.487 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.488 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.488 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-04-05T23:47:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-04-05T23:47:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-04-05T23:47:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-05T23:47:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-04-05T23:47:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-04-05T23:47:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-04-05T23:47:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-05T23:47:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-04-05T23:47:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-05T23:47:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-05T23:47:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-04-05T23:47:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-05T23:47:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-05T23:47:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-05T23:47:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-04-05T23:47:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-04-05T23:47:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-05T23:47:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-05T23:47:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-04-05T23:47:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-05T23:47:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-05T23:47:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-05T23:47:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-05T23:47:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-05T23:47:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-05T23:47:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-05T23:47:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-05T23:47:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-05T23:47:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-05T23:47:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-05T23:47:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-05T23:47:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-05T23:47:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-05T23:47:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-05T23:47:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-05T23:47:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-05T23:47:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-04-05T23:47:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-05T23:47:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-05T23:47:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-05T23:47:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-05T23:47:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-04-05T23:47:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-04-05T23:47:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-04-05T23:47:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-05T23:47:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-05T23:47:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-05T23:47:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-05T23:47:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-05T23:47:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-05T23:47:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-05T23:47:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-05T23:47:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-05T23:47:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-05T23:47:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-04-05T23:47:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-05T23:47:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-05T23:47:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-05T23:47:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-05T23:47:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-05T23:47:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-05T23:47:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-05T23:47:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-05T23:47:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-05T23:47:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-05T23:47:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-05T23:47:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-05T23:47:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-05T23:47:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-05T23:47:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-05T23:47:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-05T23:47:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-05T23:47:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-04-05T23:47:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-05T23:47:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-05T23:47:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-05T23:47:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-05T23:47:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-05T23:47:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-05T23:47:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-05T23:47:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-05T23:47:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-04-05T23:47:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-05T23:47:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-05T23:47:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-04-05T23:47:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-04-05T23:47:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-05T23:47:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-05T23:47:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-05T23:47:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-05T23:47:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-04-05T23:47:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-05T23:47:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-04-05T23:47:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-05T23:47:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-04-05T23:47:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-05T23:47:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-05T23:47:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-05T23:47:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-05T23:47:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-05T23:47:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-05T23:47:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-05T23:47:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-04-05T23:47:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-04-05T23:47:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-04-05T23:47:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-04-05T23:47:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-04-05T23:47:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-04-05T23:47:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-04-05T23:47:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-04-05T23:47:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-04-05T23:47:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-04-05T23:47:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-05T23:47:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-05T23:47:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-04-05T23:47:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-04-05T23:47:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-04-05T23:47:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-04-05T23:47:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-04-05T23:47:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-05T23:47:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-05T23:47:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-05T23:47:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-05T23:47:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-05T23:47:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-04-05T23:47:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-05T23:47:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-05T23:47:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-04-05T23:47:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-04-05T23:47:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-04-05T23:47:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-05T23:47:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-05T23:47:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-04-05T23:47:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-04-05T23:47:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-05T23:47:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-05T23:47:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-05T23:47:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-05T23:47:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-05T23:47:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-04-05T23:47:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-04-05T23:47:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-05T23:47:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-04-05T23:47:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-04-05T23:47:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-04-05T23:47:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-04-05T23:47:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-04-05T23:47:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-04-05T23:47:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-05T23:47:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-04-05T23:47:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-04-05T23:47:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-04-05T23:47:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-05T23:47:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-04-05T23:47:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-04-05T23:47:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-04-05T23:47:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-04-05T23:47:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-04-05T23:47:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-04-05T23:47:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-05T23:47:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-05T23:47:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-04-05T23:47:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-04-05T23:47:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-05T23:47:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-04-05T23:47:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-04-05T23:47:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-04-05T23:47:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-04-05T23:47:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-05T23:47:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-04-05T23:47:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-04-05T23:47:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-04-05T23:47:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-05T23:47:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-04-05T23:47:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-04-05T23:47:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-04-05T23:47:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-05T23:47:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-04-05T23:47:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-04-05T23:47:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-04-05T23:47:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-04-05T23:47:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-04-05T23:47:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-04-05T23:47:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-05T23:47:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-04-05T23:47:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-04-05T23:47:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-04-05T23:47:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-05T23:47:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-04-05T23:47:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-04-05T23:47:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-04-05T23:47:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-04-05T23:47:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-04-05T23:47:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-05T23:47:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-04-05T23:47:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-04-05T23:47:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-04-05T23:47:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-04-05T23:47:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-04-05T23:47:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-04-05T23:47:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-04-05T23:47:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-05T23:47:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-05T23:47:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-04-05T23:47:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-05T23:47:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-04-05T23:47:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-04-05T23:47:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-04-05T23:47:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-05T23:47:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-04-05T23:47:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-04-05T23:47:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-04-05T23:47:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-04-05T23:47:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-04-05T23:47:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-04-05T23:47:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-05T23:47:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-04-05T23:47:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-04-05T23:47:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-04-05T23:47:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-04-05T23:47:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-04-05T23:47:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-04-05T23:47:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-05T23:47:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-04-05T23:47:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-04-05T23:47:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-04-05T23:47:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-05T23:47:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-04-05T23:47:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-04-05T23:47:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-04-05T23:47:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-04-05T23:47:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-04-05T23:47:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-05T23:47:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-04-05T23:47:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-04-05T23:47:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-04-05T23:47:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-05T23:47:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-04-05T23:47:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-04-05T23:47:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-04-05T23:47:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-04-05T23:47:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-04-05T23:47:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-04-05T23:47:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-04-05T23:47:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-04-05T23:47:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-04-05T23:47:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-04-05T23:47:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-04-05T23:47:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-04-05T23:47:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-04-05T23:47:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-04-05T23:47:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-05T23:47:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-05T23:47:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-04-05T23:47:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-04-05T23:47:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-04-05T23:47:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-05T23:47:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-04-05T23:47:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-04-05T23:47:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-05T23:47:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-04-05T23:47:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-05T23:47:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-05T23:47:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-04-05T23:47:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-04-05T23:47:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-04-05T23:47:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-04-05T23:47:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-04-05T23:47:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-04-05T23:47:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-04-05T23:47:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-04-05T23:47:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-05T23:47:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-04-05T23:47:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-04-05T23:47:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-05T23:47:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-04-05T23:47:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-04-05T23:47:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-04-05T23:47:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-05T23:47:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-05T23:47:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-05T23:47:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-04-05T23:47:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-04-05T23:47:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-04-05T23:47:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-04-05T23:47:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-05T23:47:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-05T23:47:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-05T23:47:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-04-05T23:47:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-04-05T23:47:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-04-05T23:47:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-04-05T23:47:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-04-05T23:47:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-04-05T23:47:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-05T23:47:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-05T23:47:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-04-05T23:47:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-04-05T23:47:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-05T23:47:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-05T23:47:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-04-05T23:47:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-04-05T23:47:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-04-05T23:47:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-04-05T23:47:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-05T23:47:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-05T23:47:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-05T23:47:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-05T23:47:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-05T23:47:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-05T23:47:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-04-05T23:47:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-05T23:47:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-04-05T23:47:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-04-05T23:47:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-04-05T23:47:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-05T23:47:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-05T23:47:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-04-05T23:47:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-04-05T23:47:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-04-05T23:47:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-05T23:47:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-05T23:47:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-05T23:47:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-05T23:47:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-05T23:47:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-05T23:47:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-04-05T23:47:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-04-05T23:47:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-05T23:47:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-04-05T23:47:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-04-05T23:47:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-04-05T23:47:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-04-05T23:47:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-04-05T23:47:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-04-05T23:47:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-04-05T23:47:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-05T23:47:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-05T23:47:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-05T23:47:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-05T23:47:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-05T23:47:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-05T23:47:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-04-05T23:47:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-04-05T23:47:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-04-05T23:47:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-04-05T23:47:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-04-05T23:47:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-05T23:47:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-04-05T23:47:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-04-05T23:47:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-04-05T23:47:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-04-05T23:47:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-04-05T23:47:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-04-05T23:47:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-04-05T23:47:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-04-05T23:47:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-04-05T23:47:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-04-05T23:47:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-04-05T23:47:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-04-05T23:47:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-05T23:47:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-04-05T23:47:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-04-05T23:47:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-04-05T23:47:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-04-05T23:47:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-04-05T23:47:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-04-05T23:47:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-04-05T23:47:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-04-05T23:47:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-05T23:47:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-04-05T23:47:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-04-05T23:47:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-04-05T23:47:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-04-05T23:47:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-04-05T23:47:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-04-05T23:47:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-04-05T23:47:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-05T23:47:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-04-05T23:47:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-04-05T23:47:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-04-05T23:47:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-04-05T23:47:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-04-05T23:47:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-04-05T23:47:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-04-05T23:47:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-04-05T23:47:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-04-05T23:47:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-04-05T23:47:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-04-05T23:47:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-05T23:47:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-04-05T23:47:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-04-05T23:47:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-04-05T23:47:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-04-05T23:47:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-05T23:47:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-04-05T23:47:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-04-05T23:47:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-05T23:47:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-04-05T23:47:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-04-05T23:47:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-05T23:47:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-04-05T23:47:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-04-05T23:47:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-05T23:47:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-04-05T23:47:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-04-05T23:47:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-05T23:47:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-04-05T23:47:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-04-05T23:47:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-04-05T23:47:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-04-05T23:47:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-04-05T23:47:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-04-05T23:47:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-05T23:47:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-05T23:47:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-05T23:47:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-04-05T23:47:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-04-05T23:47:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-04-05T23:47:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-04-05T23:47:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-04-05T23:47:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-04-05T23:47:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-04-05T23:47:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-04-05T23:47:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-04-05T23:47:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-04-05T23:47:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-04-05T23:47:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-04-05T23:47:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-04-05T23:47:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-04-05T23:47:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-04-05T23:47:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-05T23:47:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-04-05T23:47:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:16.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.004 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:16.004 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.004 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.004 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-05T23:47:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-04-05T23:47:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-04-05T23:47:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-04-05T23:47:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-05T23:47:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-04-05T23:47:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-04-05T23:47:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-05T23:47:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-05T23:47:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-04-05T23:47:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-04-05T23:47:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-05T23:47:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-05T23:47:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-04-05T23:47:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-05T23:47:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-05T23:47:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-05T23:47:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-04-05T23:47:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-04-05T23:47:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-05T23:47:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-04-05T23:47:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-04-05T23:47:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-04-05T23:47:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-05T23:47:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-05T23:47:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-05T23:47:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-05T23:47:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-05T23:47:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-05T23:47:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-05T23:47:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-05T23:47:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-05T23:47:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-05T23:47:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-05T23:47:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-05T23:47:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-05T23:47:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-05T23:47:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-05T23:47:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-05T23:47:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-05T23:47:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-05T23:47:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-05T23:47:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-05T23:47:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-05T23:47:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-05T23:47:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-05T23:47:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-04-05T23:47:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-04-05T23:47:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-05T23:47:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-04-05T23:47:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-04-05T23:47:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-05T23:47:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-04-05T23:47:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-04-05T23:47:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-04-05T23:47:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-05T23:47:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-05T23:47:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-05T23:47:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-05T23:47:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-05T23:47:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-05T23:47:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-05T23:47:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-05T23:47:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-05T23:47:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-05T23:47:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-05T23:47:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "services": { 2024-04-05T23:47:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard": "https://172.21.15.16:8443/" 2024-04-05T23:47:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "always_on_modules": { 2024-04-05T23:47:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "octopus": [ 2024-04-05T23:47:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-05T23:47:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-05T23:47:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-05T23:47:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-05T23:47:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-05T23:47:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-05T23:47:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-05T23:47:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-05T23:47:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-05T23:47:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-05T23:47:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "pacific": [ 2024-04-05T23:47:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-05T23:47:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-05T23:47:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-05T23:47:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-05T23:47:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-05T23:47:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-05T23:47:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-05T23:47:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-05T23:47:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-05T23:47:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-05T23:47:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "quincy": [ 2024-04-05T23:47:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-05T23:47:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-05T23:47:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-05T23:47:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-05T23:47:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-05T23:47:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-05T23:47:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-05T23:47:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-05T23:47:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-05T23:47:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-05T23:47:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-05T23:47:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "reef": [ 2024-04-05T23:47:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-05T23:47:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-05T23:47:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-05T23:47:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-05T23:47:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-05T23:47:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-05T23:47:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-05T23:47:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-05T23:47:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-05T23:47:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-05T23:47:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "last_failure_osd_epoch": 82, 2024-04-05T23:47:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "active_clients": [ 2024-04-05T23:47:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-05T23:47:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-05T23:47:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-05T23:47:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-05T23:47:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2763394769 2024-04-05T23:47:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "libcephsqlite", 2024-04-05T23:47:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-05T23:47:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-05T23:47:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-05T23:47:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2270912430 2024-04-05T23:47:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-05T23:47:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-05T23:47:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-05T23:47:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-05T23:47:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 778317070 2024-04-05T23:47:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-05T23:47:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-05T23:47:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-05T23:47:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-05T23:47:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2230608924 2024-04-05T23:47:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-05T23:47:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-05T23:47:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-05T23:47:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-05T23:47:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 392563456 2024-04-05T23:47:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-05T23:47:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-05T23:47:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-05T23:47:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-05T23:47:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-05T23:47:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1506378199 2024-04-05T23:47:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-05T23:47:16.053 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:47:16.054 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:14 smithi130 ceph-mon[79417]: pgmap v430: 97 pgs: 97 active+clean; 565 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 30 MiB/s wr, 540 op/s 2024-04-05T23:47:16.054 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:14 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1210546376' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-05T23:47:16.055 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:14 smithi027 ceph-mon[78597]: pgmap v430: 97 pgs: 97 active+clean; 565 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 30 MiB/s wr, 540 op/s 2024-04-05T23:47:16.055 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:14 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1210546376' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-05T23:47:16.055 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:15 smithi027 ceph-mon[78597]: osdmap e101: 12 total, 12 up, 12 in 2024-04-05T23:47:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff94c0a1310 msgr2=0x7ff94c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff94c0a1310 0x7ff94c0a37d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff964074910 tx=0x7ff950009260 comp rx=0 tx=0).stop 2024-04-05T23:47:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff964100f90 msgr2=0x7ff9640731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff964100f90 0x7ff9640731e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff95800e9a0 tx=0x7ff95800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:47:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 shutdown_connections 2024-04-05T23:47:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff94c0a1310 0x7ff94c0a37d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964073c60 0x7ff9641a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff964100f90 0x7ff9640731e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 --2- 172.21.15.16:0/1210546376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9641060b0 0x7ff964073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 >> 172.21.15.16:0/1210546376 conn(0x7ff9640fc830 msgr2=0x7ff9640fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 shutdown_connections 2024-04-05T23:47:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:13.860+0000 7ff96c400700 1 -- 172.21.15.16:0/1210546376 wait complete. 2024-04-05T23:47:16.063 DEBUG:tasks.check_counter:Getting stats from a 2024-04-05T23:47:16.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.a perf dump 2024-04-05T23:47:16.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:14 smithi016 ceph-mon[38248]: pgmap v430: 97 pgs: 97 active+clean; 565 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 30 MiB/s wr, 540 op/s 2024-04-05T23:47:16.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:14 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1210546376' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-05T23:47:16.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:15 smithi016 ceph-mon[38248]: osdmap e101: 12 total, 12 up, 12 in 2024-04-05T23:47:16.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:15 smithi130 ceph-mon[79417]: osdmap e101: 12 total, 12 up, 12 in 2024-04-05T23:47:16.443 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:47:16.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:16 smithi016 ceph-mon[38248]: pgmap v432: 97 pgs: 97 active+clean; 565 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 31 MiB/s wr, 507 op/s 2024-04-05T23:47:16.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:16 smithi027 ceph-mon[78597]: pgmap v432: 97 pgs: 97 active+clean; 565 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 31 MiB/s wr, 507 op/s 2024-04-05T23:47:17.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:16 smithi130 ceph-mon[79417]: pgmap v432: 97 pgs: 97 active+clean; 565 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 31 MiB/s wr, 507 op/s 2024-04-05T23:47:18.014 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:47:18.015 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:18.015 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1435, 2024-04-05T23:47:18.015 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1435, 2024-04-05T23:47:18.015 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 15755865, 2024-04-05T23:47:18.015 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 445490, 2024-04-05T23:47:18.015 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 9, 2024-04-05T23:47:18.016 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 4, 2024-04-05T23:47:18.016 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.182455521, 2024-04-05T23:47:18.016 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.069546304, 2024-04-05T23:47:18.016 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.053745327, 2024-04-05T23:47:18.016 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.039380680, 2024-04-05T23:47:18.016 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:18.016 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1434, 2024-04-05T23:47:18.016 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.069018589, 2024-04-05T23:47:18.017 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000048130 2024-04-05T23:47:18.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.017 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:18.017 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.017 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.017 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.017 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-05T23:47:18.018 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-04-05T23:47:18.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.018 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:18.018 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1274, 2024-04-05T23:47:18.018 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1460, 2024-04-05T23:47:18.018 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 25498351, 2024-04-05T23:47:18.018 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 879537, 2024-04-05T23:47:18.018 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 8, 2024-04-05T23:47:18.019 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 6, 2024-04-05T23:47:18.019 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.173979324, 2024-04-05T23:47:18.019 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.046694871, 2024-04-05T23:47:18.019 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.081595104, 2024-04-05T23:47:18.019 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.032632124, 2024-04-05T23:47:18.019 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:18.019 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1457, 2024-04-05T23:47:18.019 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.069383221, 2024-04-05T23:47:18.019 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000047620 2024-04-05T23:47:18.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.020 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:18.020 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.020 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.020 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.020 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-05T23:47:18.020 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 483888 2024-04-05T23:47:18.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.021 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:18.021 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 811, 2024-04-05T23:47:18.021 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 744, 2024-04-05T23:47:18.021 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 14637462, 2024-04-05T23:47:18.021 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 291861, 2024-04-05T23:47:18.021 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-04-05T23:47:18.021 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 4, 2024-04-05T23:47:18.021 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.138502991, 2024-04-05T23:47:18.022 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.037015060, 2024-04-05T23:47:18.022 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.092446065, 2024-04-05T23:47:18.022 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.019445974, 2024-04-05T23:47:18.022 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:18.022 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 743, 2024-04-05T23:47:18.022 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.035262552, 2024-04-05T23:47:18.022 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000047459 2024-04-05T23:47:18.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1935216, 2024-04-05T23:47:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 145680 2024-04-05T23:47:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.023 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-05T23:47:18.024 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-04-05T23:47:18.024 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-05T23:47:18.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.024 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-04-05T23:47:18.024 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-05T23:47:18.024 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-05T23:47:18.024 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 5438, 2024-04-05T23:47:18.024 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.145398440, 2024-04-05T23:47:18.025 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000026737 2024-04-05T23:47:18.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:18.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.025 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:18.025 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-05T23:47:18.025 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-05T23:47:18.025 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.025 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.026 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:18.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.026 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-05T23:47:18.026 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 0, 2024-04-05T23:47:18.026 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 0, 2024-04-05T23:47:18.026 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-04-05T23:47:18.026 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-04-05T23:47:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 0, 2024-04-05T23:47:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-04-05T23:47:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-04-05T23:47:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 0, 2024-04-05T23:47:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-04-05T23:47:18.028 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-04-05T23:47:18.028 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 4910, 2024-04-05T23:47:18.028 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 28, 2024-04-05T23:47:18.028 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 4882, 2024-04-05T23:47:18.028 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-04-05T23:47:18.028 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 4870, 2024-04-05T23:47:18.028 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 0, 2024-04-05T23:47:18.028 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 0, 2024-04-05T23:47:18.029 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 0, 2024-04-05T23:47:18.029 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 5, 2024-04-05T23:47:18.029 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 0, 2024-04-05T23:47:18.029 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 0, 2024-04-05T23:47:18.029 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 0, 2024-04-05T23:47:18.029 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 0, 2024-04-05T23:47:18.029 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-04-05T23:47:18.029 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:18.029 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 0, 2024-04-05T23:47:18.030 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-04-05T23:47:18.030 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-04-05T23:47:18.030 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 0, 2024-04-05T23:47:18.030 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-04-05T23:47:18.030 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 0, 2024-04-05T23:47:18.030 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 0, 2024-04-05T23:47:18.030 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:18.030 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 0, 2024-04-05T23:47:18.031 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:18.031 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 0, 2024-04-05T23:47:18.031 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 0, 2024-04-05T23:47:18.031 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 0, 2024-04-05T23:47:18.031 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 0, 2024-04-05T23:47:18.031 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:18.031 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:18.031 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:18.031 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-04-05T23:47:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-04-05T23:47:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-04-05T23:47:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 0, 2024-04-05T23:47:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-04-05T23:47:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 0, 2024-04-05T23:47:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 0, 2024-04-05T23:47:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:18.032 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:18.033 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:18.033 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 0, 2024-04-05T23:47:18.033 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 0, 2024-04-05T23:47:18.033 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 0, 2024-04-05T23:47:18.033 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 0 2024-04-05T23:47:18.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.033 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-04-05T23:47:18.033 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-04-05T23:47:18.034 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 0, 2024-04-05T23:47:18.034 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 0, 2024-04-05T23:47:18.034 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 0, 2024-04-05T23:47:18.034 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 0, 2024-04-05T23:47:18.034 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-04-05T23:47:18.034 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:18.034 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:18.034 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 0, 2024-04-05T23:47:18.035 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 0, 2024-04-05T23:47:18.035 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:18.035 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-04-05T23:47:18.035 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:18.035 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:18.035 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:18.035 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-04-05T23:47:18.035 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-04-05T23:47:18.036 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-04-05T23:47:18.036 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 0, 2024-04-05T23:47:18.036 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-04-05T23:47:18.036 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-04-05T23:47:18.036 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:18.036 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0 2024-04-05T23:47:18.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.036 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-04-05T23:47:18.036 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 0, 2024-04-05T23:47:18.037 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-04-05T23:47:18.037 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-04-05T23:47:18.037 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 29952, 2024-04-05T23:47:18.037 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-04-05T23:47:18.037 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-04-05T23:47:18.037 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 0, 2024-04-05T23:47:18.037 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-04-05T23:47:18.037 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-04-05T23:47:18.037 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 36, 2024-04-05T23:47:18.038 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-04-05T23:47:18.038 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-04-05T23:47:18.038 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-04-05T23:47:18.038 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 57255710, 2024-04-05T23:47:18.038 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 57251466, 2024-04-05T23:47:18.038 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-04-05T23:47:18.038 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.038 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.038 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.039 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 29952 2024-04-05T23:47:18.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.039 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-04-05T23:47:18.039 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 327, 2024-04-05T23:47:18.039 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 4934, 2024-04-05T23:47:18.039 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 4607, 2024-04-05T23:47:18.039 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 44, 2024-04-05T23:47:18.040 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 64, 2024-04-05T23:47:18.040 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 20, 2024-04-05T23:47:18.040 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 4910, 2024-04-05T23:47:18.040 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 5175, 2024-04-05T23:47:18.040 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 265, 2024-04-05T23:47:18.040 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 0, 2024-04-05T23:47:18.040 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 0, 2024-04-05T23:47:18.040 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 0, 2024-04-05T23:47:18.041 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 56048, 2024-04-05T23:47:18.041 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 207156 2024-04-05T23:47:18.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.041 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-04-05T23:47:18.041 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 0, 2024-04-05T23:47:18.041 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 0, 2024-04-05T23:47:18.041 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 0, 2024-04-05T23:47:18.041 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 0, 2024-04-05T23:47:18.041 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 0, 2024-04-05T23:47:18.042 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-04-05T23:47:18.042 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.042 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.042 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.042 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-04-05T23:47:18.042 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.042 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.043 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.043 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:18.043 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.043 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.043 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.043 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-04-05T23:47:18.043 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.044 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.044 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.044 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-04-05T23:47:18.044 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.044 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.044 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.044 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:18.045 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.045 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.045 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.045 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-04-05T23:47:18.045 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.045 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.045 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.046 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-04-05T23:47:18.046 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.046 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.046 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.046 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:18.046 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.047 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.047 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.047 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:18.047 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.047 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.047 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.047 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-04-05T23:47:18.048 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.048 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.048 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.048 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-04-05T23:47:18.048 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.048 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.049 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.049 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-04-05T23:47:18.049 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.049 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.049 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.049 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-04-05T23:47:18.050 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.050 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.050 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.050 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-04-05T23:47:18.050 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.050 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.050 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.051 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-04-05T23:47:18.051 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.051 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.051 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.051 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-04-05T23:47:18.051 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.051 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.052 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.052 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:18.052 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.052 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.052 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.052 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:18.052 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.053 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.053 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.053 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-04-05T23:47:18.053 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.053 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.054 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.054 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:18.054 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.054 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.054 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.054 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:18.055 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.055 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.055 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.055 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-04-05T23:47:18.055 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.055 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.055 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.056 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:18.056 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.056 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.056 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.056 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:18.056 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.057 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.057 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.057 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-04-05T23:47:18.057 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.057 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.057 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.057 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-04-05T23:47:18.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.058 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.058 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-04-05T23:47:18.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.058 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.059 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-04-05T23:47:18.059 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.059 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.059 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.059 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:18.059 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:18.060 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:18.060 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.060 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.060 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.060 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:18.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-04-05T23:47:18.060 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 2, 2024-04-05T23:47:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 4, 2024-04-05T23:47:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 2, 2024-04-05T23:47:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 4, 2024-04-05T23:47:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-04-05T23:47:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 0, 2024-04-05T23:47:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 0, 2024-04-05T23:47:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 3793, 2024-04-05T23:47:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:18.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.062 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-05T23:47:18.062 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:18.062 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:18.062 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:18.062 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:18.062 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:18.062 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:18.062 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:18.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:18.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:18.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:18.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:18.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:18.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:18.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:18.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:18.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:18.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:18.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:18.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:18.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:18.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:18.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:18.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:18.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:18.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:18.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:18.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:18.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:18.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:18.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:18.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:18.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-05T23:47:18.066 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:18.066 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:18.066 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:18.066 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:18.066 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 242989, 2024-04-05T23:47:18.066 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 417, 2024-04-05T23:47:18.066 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-05T23:47:18.066 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-05T23:47:18.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-05T23:47:18.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-05T23:47:18.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:18.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:18.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:18.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:18.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:18.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 188, 2024-04-05T23:47:18.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:18.068 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:18.068 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:18.068 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-05T23:47:18.068 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 4350967, 2024-04-05T23:47:18.068 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 28990, 2024-04-05T23:47:18.068 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:18.068 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-05T23:47:18.068 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:18.068 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-05T23:47:18.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.069 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-04-05T23:47:18.069 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-04-05T23:47:18.069 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-04-05T23:47:18.069 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 3173, 2024-04-05T23:47:18.069 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 21, 2024-04-05T23:47:18.069 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 3, 2024-04-05T23:47:18.069 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 3173, 2024-04-05T23:47:18.070 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-04-05T23:47:18.070 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3173, 2024-04-05T23:47:18.070 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 1.841971500, 2024-04-05T23:47:18.070 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000580514 2024-04-05T23:47:18.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.070 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-04-05T23:47:18.070 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-04-05T23:47:18.070 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3173, 2024-04-05T23:47:18.070 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 3174 2024-04-05T23:47:18.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.071 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 3173, 2024-04-05T23:47:18.071 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 3173, 2024-04-05T23:47:18.071 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 0, 2024-04-05T23:47:18.071 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-04-05T23:47:18.071 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-04-05T23:47:18.071 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 1722, 2024-04-05T23:47:18.071 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 0, 2024-04-05T23:47:18.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 1449, 2024-04-05T23:47:18.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 0, 2024-04-05T23:47:18.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 0, 2024-04-05T23:47:18.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-04-05T23:47:18.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-04-05T23:47:18.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-04-05T23:47:18.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-04-05T23:47:18.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 0, 2024-04-05T23:47:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-04-05T23:47:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 1, 2024-04-05T23:47:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 0, 2024-04-05T23:47:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-04-05T23:47:18.074 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-04-05T23:47:18.074 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-04-05T23:47:18.074 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:18.074 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-04-05T23:47:18.074 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:18.074 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-04-05T23:47:18.074 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-04-05T23:47:18.074 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-04-05T23:47:18.075 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-04-05T23:47:18.075 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-04-05T23:47:18.075 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-04-05T23:47:18.075 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-04-05T23:47:18.075 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-04-05T23:47:18.075 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-04-05T23:47:18.075 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-04-05T23:47:18.075 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-04-05T23:47:18.076 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-04-05T23:47:18.076 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-04-05T23:47:18.076 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-04-05T23:47:18.076 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-04-05T23:47:18.076 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-04-05T23:47:18.076 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-04-05T23:47:18.076 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 101, 2024-04-05T23:47:18.076 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-04-05T23:47:18.076 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 22, 2024-04-05T23:47:18.077 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 10, 2024-04-05T23:47:18.077 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 11, 2024-04-05T23:47:18.077 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 1, 2024-04-05T23:47:18.077 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-04-05T23:47:18.077 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 0, 2024-04-05T23:47:18.077 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 2, 2024-04-05T23:47:18.077 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 0 2024-04-05T23:47:18.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-04-05T23:47:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 0, 2024-04-05T23:47:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-04-05T23:47:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-04-05T23:47:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: "osd-slow-ops": { 2024-04-05T23:47:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 0 2024-04-05T23:47:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-04-05T23:47:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 0, 2024-04-05T23:47:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-04-05T23:47:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 0, 2024-04-05T23:47:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 0, 2024-04-05T23:47:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:18.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.080 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:18.080 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:18.080 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-05T23:47:18.080 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:18.080 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 3516, 2024-04-05T23:47:18.080 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 55555090, 2024-04-05T23:47:18.080 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:18.081 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 3516, 2024-04-05T23:47:18.081 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-05T23:47:18.081 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-05T23:47:18.081 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 3516, 2024-04-05T23:47:18.081 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 55555090, 2024-04-05T23:47:18.081 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:18.081 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.081 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.082 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.082 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:18.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.082 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:18.082 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:18.082 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-05T23:47:18.082 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:18.082 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-05T23:47:18.083 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-05T23:47:18.083 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:18.083 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:18.083 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 3173, 2024-04-05T23:47:18.083 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 53074826, 2024-04-05T23:47:18.083 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 3173, 2024-04-05T23:47:18.083 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 53074826, 2024-04-05T23:47:18.083 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:18.084 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.084 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.084 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:18.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.084 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:18.084 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:18.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-04-05T23:47:18.085 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:18.085 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-05T23:47:18.085 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-05T23:47:18.085 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:18.085 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:18.085 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 3173, 2024-04-05T23:47:18.085 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 3173, 2024-04-05T23:47:18.085 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 3173, 2024-04-05T23:47:18.086 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 3173, 2024-04-05T23:47:18.086 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:18.086 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.086 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.086 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:18.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.086 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:18.087 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:18.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-05T23:47:18.087 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:18.087 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-05T23:47:18.087 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-05T23:47:18.087 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:18.087 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:18.087 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-05T23:47:18.088 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-05T23:47:18.088 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-04-05T23:47:18.088 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-04-05T23:47:18.088 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:18.088 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.088 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.088 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.088 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:18.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:18.089 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x55adba04e5a0": { 2024-04-05T23:47:18.089 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:18.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-05T23:47:18.089 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:18.089 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-05T23:47:18.089 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-05T23:47:18.090 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:18.090 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:18.090 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-05T23:47:18.090 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-05T23:47:18.090 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-04-05T23:47:18.090 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-04-05T23:47:18.091 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:18.091 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:18.091 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:18.091 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:18.092 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:18.092 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:18.092 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:47:18.439 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-05T23:47:18.439 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-05T23:47:18.440 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-04-05T23:47:18.440 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-05T23:47:18.440 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-05T23:47:18.440 DEBUG:tasks.check_counter:Getting stats from d 2024-04-05T23:47:18.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.d perf dump 2024-04-05T23:47:18.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:18 smithi016 ceph-mon[38248]: pgmap v433: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 30 MiB/s wr, 661 op/s 2024-04-05T23:47:18.851 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:47:18.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:18 smithi027 ceph-mon[78597]: pgmap v433: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 30 MiB/s wr, 661 op/s 2024-04-05T23:47:19.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:18 smithi130 ceph-mon[79417]: pgmap v433: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 30 MiB/s wr, 661 op/s 2024-04-05T23:47:20.029 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:47:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 7071, 2024-04-05T23:47:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 6762, 2024-04-05T23:47:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 4424090, 2024-04-05T23:47:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 20203500, 2024-04-05T23:47:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 20, 2024-04-05T23:47:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 4, 2024-04-05T23:47:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.648213437, 2024-04-05T23:47:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.255235322, 2024-04-05T23:47:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.243384904, 2024-04-05T23:47:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.139801889, 2024-04-05T23:47:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 6761, 2024-04-05T23:47:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.272195591, 2024-04-05T23:47:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000040259 2024-04-05T23:47:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2101, 2024-04-05T23:47:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000382712, 2024-04-05T23:47:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000182 2024-04-05T23:47:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1959888, 2024-04-05T23:47:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 154832 2024-04-05T23:47:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 6033, 2024-04-05T23:47:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 6221, 2024-04-05T23:47:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1716419, 2024-04-05T23:47:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 35696118, 2024-04-05T23:47:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 15, 2024-04-05T23:47:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 8, 2024-04-05T23:47:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.598651291, 2024-04-05T23:47:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.249333844, 2024-04-05T23:47:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.139699231, 2024-04-05T23:47:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.164113215, 2024-04-05T23:47:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 6219, 2024-04-05T23:47:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.281718757, 2024-04-05T23:47:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000045299 2024-04-05T23:47:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 181, 2024-04-05T23:47:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000031874, 2024-04-05T23:47:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000176 2024-04-05T23:47:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 4192, 2024-04-05T23:47:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 487648 2024-04-05T23:47:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 140541, 2024-04-05T23:47:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 142012, 2024-04-05T23:47:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 46734423, 2024-04-05T23:47:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 85083166, 2024-04-05T23:47:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 9, 2024-04-05T23:47:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 5, 2024-04-05T23:47:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 14.493155805, 2024-04-05T23:47:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 7.669066312, 2024-04-05T23:47:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 9.064945513, 2024-04-05T23:47:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.081667792, 2024-04-05T23:47:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 142010, 2024-04-05T23:47:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 4.219380911, 2024-04-05T23:47:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000029711 2024-04-05T23:47:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 263394, 2024-04-05T23:47:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.056722622, 2024-04-05T23:47:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000215 2024-04-05T23:47:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-05T23:47:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-04-05T23:47:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-05T23:47:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-04-05T23:47:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-05T23:47:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-04-05T23:47:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-05T23:47:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-05T23:47:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 16398, 2024-04-05T23:47:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 10.795760245, 2024-04-05T23:47:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000658358 2024-04-05T23:47:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-05T23:47:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-05T23:47:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 9443, 2024-04-05T23:47:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.521120044, 2024-04-05T23:47:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000055185 2024-04-05T23:47:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-05T23:47:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 35078, 2024-04-05T23:47:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 33250, 2024-04-05T23:47:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-04-05T23:47:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 33250, 2024-04-05T23:47:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 66.705040438, 2024-04-05T23:47:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.002006166 2024-04-05T23:47:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-04-05T23:47:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 1828, 2024-04-05T23:47:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 2, 2024-04-05T23:47:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-04-05T23:47:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 11, 2024-04-05T23:47:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-04-05T23:47:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-04-05T23:47:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 558, 2024-04-05T23:47:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 35, 2024-04-05T23:47:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 505, 2024-04-05T23:47:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 18, 2024-04-05T23:47:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 508, 2024-04-05T23:47:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 331, 2024-04-05T23:47:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 160, 2024-04-05T23:47:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 160, 2024-04-05T23:47:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 10, 2024-04-05T23:47:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 77622, 2024-04-05T23:47:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 42393, 2024-04-05T23:47:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 1823, 2024-04-05T23:47:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 63, 2024-04-05T23:47:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-04-05T23:47:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 2070, 2024-04-05T23:47:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 201100, 2024-04-05T23:47:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-04-05T23:47:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 5, 2024-04-05T23:47:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 211, 2024-04-05T23:47:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 8, 2024-04-05T23:47:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 443, 2024-04-05T23:47:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 0, 2024-04-05T23:47:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 702, 2024-04-05T23:47:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 26094429, 2024-04-05T23:47:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 6, 2024-04-05T23:47:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 0, 2024-04-05T23:47:20.047 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:20.047 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:20.047 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:20.047 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-04-05T23:47:20.047 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-04-05T23:47:20.047 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-04-05T23:47:20.048 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 0, 2024-04-05T23:47:20.048 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-04-05T23:47:20.048 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 2552, 2024-04-05T23:47:20.048 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 22315, 2024-04-05T23:47:20.048 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:20.048 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:20.048 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:20.048 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 10156, 2024-04-05T23:47:20.049 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 4266, 2024-04-05T23:47:20.049 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 2913, 2024-04-05T23:47:20.049 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 31763 2024-04-05T23:47:20.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.049 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-04-05T23:47:20.049 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-04-05T23:47:20.049 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 77, 2024-04-05T23:47:20.049 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 40, 2024-04-05T23:47:20.049 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 148, 2024-04-05T23:47:20.050 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 36, 2024-04-05T23:47:20.050 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 10, 2024-04-05T23:47:20.050 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:20.050 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:20.050 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 4695, 2024-04-05T23:47:20.050 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 4685, 2024-04-05T23:47:20.050 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:20.051 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-04-05T23:47:20.051 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:20.051 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:20.051 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:20.051 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-04-05T23:47:20.051 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-04-05T23:47:20.051 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-04-05T23:47:20.052 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 8, 2024-04-05T23:47:20.052 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-04-05T23:47:20.052 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-04-05T23:47:20.052 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:20.052 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0 2024-04-05T23:47:20.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.052 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-04-05T23:47:20.052 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 34079, 2024-04-05T23:47:20.053 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-04-05T23:47:20.053 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-04-05T23:47:20.053 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 34079, 2024-04-05T23:47:20.053 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-04-05T23:47:20.053 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-04-05T23:47:20.053 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 41, 2024-04-05T23:47:20.053 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-04-05T23:47:20.053 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-04-05T23:47:20.054 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 41, 2024-04-05T23:47:20.054 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-04-05T23:47:20.054 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-04-05T23:47:20.054 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-04-05T23:47:20.054 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 68583155, 2024-04-05T23:47:20.054 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 0, 2024-04-05T23:47:20.054 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-04-05T23:47:20.054 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 8179, 2024-04-05T23:47:20.055 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 17.948871312, 2024-04-05T23:47:20.055 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.002194506 2024-04-05T23:47:20.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.055 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 0 2024-04-05T23:47:20.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.055 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-04-05T23:47:20.055 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 365, 2024-04-05T23:47:20.055 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 5306, 2024-04-05T23:47:20.056 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 4941, 2024-04-05T23:47:20.056 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 62, 2024-04-05T23:47:20.056 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 126, 2024-04-05T23:47:20.056 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 64, 2024-04-05T23:47:20.056 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 567, 2024-04-05T23:47:20.056 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 8965, 2024-04-05T23:47:20.056 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 8398, 2024-04-05T23:47:20.056 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 160, 2024-04-05T23:47:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 5027, 2024-04-05T23:47:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 4867, 2024-04-05T23:47:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 66956, 2024-04-05T23:47:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 198964 2024-04-05T23:47:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-04-05T23:47:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 59706, 2024-04-05T23:47:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 534, 2024-04-05T23:47:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 35078, 2024-04-05T23:47:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 781, 2024-04-05T23:47:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 7215, 2024-04-05T23:47:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-04-05T23:47:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 4717, 2024-04-05T23:47:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 11.056085179, 2024-04-05T23:47:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.002343880 2024-04-05T23:47:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-04-05T23:47:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 4, 2024-04-05T23:47:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.017006510, 2024-04-05T23:47:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.004251627 2024-04-05T23:47:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.060 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-04-05T23:47:20.060 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.060 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.060 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.060 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-04-05T23:47:20.060 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 9836, 2024-04-05T23:47:20.061 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 16.778640837, 2024-04-05T23:47:20.061 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001705839 2024-04-05T23:47:20.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.061 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:20.061 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.061 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.061 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.062 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-04-05T23:47:20.062 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1, 2024-04-05T23:47:20.062 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000110758, 2024-04-05T23:47:20.062 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000110758 2024-04-05T23:47:20.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.062 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-04-05T23:47:20.062 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.063 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.063 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.063 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:20.063 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.063 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.063 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.063 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:20.064 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.064 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.064 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.064 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-04-05T23:47:20.064 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.064 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.064 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.065 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-04-05T23:47:20.065 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.065 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.065 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.065 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-04-05T23:47:20.065 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.066 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.066 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.066 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-04-05T23:47:20.066 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.066 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.066 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.067 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-04-05T23:47:20.067 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.067 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.067 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.067 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-04-05T23:47:20.067 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 11868, 2024-04-05T23:47:20.067 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 12.983197750, 2024-04-05T23:47:20.067 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001093966 2024-04-05T23:47:20.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.068 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-04-05T23:47:20.068 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2109, 2024-04-05T23:47:20.068 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 4.416303777, 2024-04-05T23:47:20.068 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.002094027 2024-04-05T23:47:20.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.068 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:20.068 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.070 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-04-05T23:47:20.070 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.070 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.070 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.070 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:20.070 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.070 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.071 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.071 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:20.071 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.071 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.071 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.071 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-04-05T23:47:20.072 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 12, 2024-04-05T23:47:20.072 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.044827278, 2024-04-05T23:47:20.072 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.003735606 2024-04-05T23:47:20.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.072 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:20.072 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.072 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.072 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.073 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:20.073 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 8, 2024-04-05T23:47:20.073 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.005932600, 2024-04-05T23:47:20.073 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000741575 2024-04-05T23:47:20.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.073 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-04-05T23:47:20.073 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.074 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.074 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.074 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-04-05T23:47:20.074 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.074 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.074 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.075 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-04-05T23:47:20.075 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.075 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.075 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.075 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-04-05T23:47:20.075 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 4695, 2024-04-05T23:47:20.075 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 21.402935749, 2024-04-05T23:47:20.076 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.004558665 2024-04-05T23:47:20.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.076 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:20.076 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:20.076 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:20.076 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.076 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.076 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:20.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.077 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-04-05T23:47:20.077 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 1, 2024-04-05T23:47:20.077 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 3, 2024-04-05T23:47:20.077 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 2, 2024-04-05T23:47:20.077 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 1, 2024-04-05T23:47:20.077 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-04-05T23:47:20.077 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 3692, 2024-04-05T23:47:20.078 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 3692, 2024-04-05T23:47:20.078 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 808, 2024-04-05T23:47:20.078 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:20.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.078 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-05T23:47:20.078 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:20.078 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:20.078 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:20.079 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:20.079 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:20.079 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:20.079 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:20.079 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:20.079 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:20.079 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:20.079 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:20.080 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:20.080 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:20.080 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:20.080 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:20.080 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:20.080 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:20.080 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:20.080 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:20.080 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:20.081 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:20.081 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:20.081 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:20.081 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:20.081 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:20.081 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:20.081 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:20.082 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:20.082 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:20.082 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:20.082 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:20.082 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-05T23:47:20.082 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:20.082 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:20.082 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:20.083 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:20.083 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 345960, 2024-04-05T23:47:20.083 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 396, 2024-04-05T23:47:20.083 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 440, 2024-04-05T23:47:20.083 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 5, 2024-04-05T23:47:20.083 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-05T23:47:20.083 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-05T23:47:20.084 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:20.084 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:20.084 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:20.084 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:20.084 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:20.084 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 188, 2024-04-05T23:47:20.084 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:20.084 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:20.085 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:20.085 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-05T23:47:20.085 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 1495269, 2024-04-05T23:47:20.085 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 13699, 2024-04-05T23:47:20.085 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:20.085 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-05T23:47:20.085 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:20.085 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-05T23:47:20.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.086 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-04-05T23:47:20.086 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-04-05T23:47:20.086 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-04-05T23:47:20.086 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 14470, 2024-04-05T23:47:20.086 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 65298505, 2024-04-05T23:47:20.087 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 27, 2024-04-05T23:47:20.087 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 14470, 2024-04-05T23:47:20.087 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-04-05T23:47:20.087 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 14470, 2024-04-05T23:47:20.087 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 31.760508854, 2024-04-05T23:47:20.087 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.002194921 2024-04-05T23:47:20.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.087 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-04-05T23:47:20.088 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-04-05T23:47:20.088 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 14470, 2024-04-05T23:47:20.088 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 14775 2024-04-05T23:47:20.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.088 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 14470, 2024-04-05T23:47:20.088 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 544, 2024-04-05T23:47:20.088 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 13926, 2024-04-05T23:47:20.088 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-04-05T23:47:20.089 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-04-05T23:47:20.089 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 0, 2024-04-05T23:47:20.089 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 10, 2024-04-05T23:47:20.089 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 542, 2024-04-05T23:47:20.089 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 8721, 2024-04-05T23:47:20.089 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 323, 2024-04-05T23:47:20.089 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-04-05T23:47:20.089 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-04-05T23:47:20.089 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-04-05T23:47:20.090 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-04-05T23:47:20.090 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 4712, 2024-04-05T23:47:20.090 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-04-05T23:47:20.090 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:20.090 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:20.090 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 0, 2024-04-05T23:47:20.090 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 20, 2024-04-05T23:47:20.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:20.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:20.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:20.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-04-05T23:47:20.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-04-05T23:47:20.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-04-05T23:47:20.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:20.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-04-05T23:47:20.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-04-05T23:47:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-04-05T23:47:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-04-05T23:47:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-04-05T23:47:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-04-05T23:47:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-04-05T23:47:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-04-05T23:47:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-04-05T23:47:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-04-05T23:47:20.093 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-04-05T23:47:20.093 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-04-05T23:47:20.093 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-04-05T23:47:20.093 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-04-05T23:47:20.093 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-04-05T23:47:20.093 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-04-05T23:47:20.093 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-04-05T23:47:20.093 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-04-05T23:47:20.094 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 101, 2024-04-05T23:47:20.094 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-04-05T23:47:20.094 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 22, 2024-04-05T23:47:20.094 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 12, 2024-04-05T23:47:20.094 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 12, 2024-04-05T23:47:20.094 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 0, 2024-04-05T23:47:20.094 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-04-05T23:47:20.094 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 301, 2024-04-05T23:47:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 4, 2024-04-05T23:47:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 142 2024-04-05T23:47:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-04-05T23:47:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 1, 2024-04-05T23:47:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-04-05T23:47:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-04-05T23:47:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.096 INFO:teuthology.orchestra.run.smithi016.stdout: "osd-slow-ops": { 2024-04-05T23:47:20.096 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 1 2024-04-05T23:47:20.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.096 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-04-05T23:47:20.096 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:20.096 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 6, 2024-04-05T23:47:20.096 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-04-05T23:47:20.096 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 6, 2024-04-05T23:47:20.097 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 4685, 2024-04-05T23:47:20.097 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:20.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.097 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:20.097 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:20.097 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-05T23:47:20.097 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 153641, 2024-04-05T23:47:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 39025268, 2024-04-05T23:47:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 153641, 2024-04-05T23:47:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-05T23:47:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-05T23:47:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 153641, 2024-04-05T23:47:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 39025268, 2024-04-05T23:47:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:20.099 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.099 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.099 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.099 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:20.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.099 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:20.099 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:20.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-05T23:47:20.100 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:20.100 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-05T23:47:20.100 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-05T23:47:20.100 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:20.100 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:20.100 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 14470, 2024-04-05T23:47:20.100 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 65773290, 2024-04-05T23:47:20.101 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 14470, 2024-04-05T23:47:20.101 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 65773290, 2024-04-05T23:47:20.101 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:20.101 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.101 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.101 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.101 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:20.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.101 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:20.102 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:20.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-04-05T23:47:20.102 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:20.102 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-05T23:47:20.102 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-05T23:47:20.102 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:20.102 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:20.102 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 14470, 2024-04-05T23:47:20.103 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 14470, 2024-04-05T23:47:20.103 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 14470, 2024-04-05T23:47:20.103 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 14470, 2024-04-05T23:47:20.103 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:20.103 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.103 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.103 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:20.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.104 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:20.104 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:20.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-05T23:47:20.104 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:20.104 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 4685, 2024-04-05T23:47:20.104 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 474817, 2024-04-05T23:47:20.104 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:20.104 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 4685, 2024-04-05T23:47:20.105 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-05T23:47:20.105 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-05T23:47:20.105 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 542, 2024-04-05T23:47:20.105 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 474817, 2024-04-05T23:47:20.105 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:20.105 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.105 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.105 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:20.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:20.106 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x557fa11c40a0": { 2024-04-05T23:47:20.106 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 4422, 2024-04-05T23:47:20.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-05T23:47:20.106 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:20.106 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 34079, 2024-04-05T23:47:20.106 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 64388851, 2024-04-05T23:47:20.107 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:20.107 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 34079, 2024-04-05T23:47:20.107 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-05T23:47:20.107 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-05T23:47:20.107 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 8179, 2024-04-05T23:47:20.107 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 64384429, 2024-04-05T23:47:20.107 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:20.107 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:20.107 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:20.108 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:20.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:20.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:20.108 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:47:20.406 INFO:tasks.check_counter:Daemon mds.d mds.exported=5 2024-04-05T23:47:20.406 INFO:tasks.check_counter:Daemon mds.d mds.imported=8 2024-04-05T23:47:20.406 INFO:tasks.check_counter:Daemon mds.d mds.root_rsnaps=6 2024-04-05T23:47:20.406 INFO:tasks.check_counter:Daemon mds.d mds_server.req_mksnap_latency.avgcount=0 2024-04-05T23:47:20.407 INFO:tasks.check_counter:Daemon mds.d mds_server.req_rmsnap_latency.avgcount=0 2024-04-05T23:47:20.407 DEBUG:tasks.check_counter:Getting stats from g 2024-04-05T23:47:20.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.g perf dump 2024-04-05T23:47:20.782 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:47:20.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:20 smithi016 ceph-mon[38248]: pgmap v434: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 26 MiB/s wr, 560 op/s 2024-04-05T23:47:20.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:20 smithi027 ceph-mon[78597]: pgmap v434: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 26 MiB/s wr, 560 op/s 2024-04-05T23:47:21.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:20 smithi130 ceph-mon[79417]: pgmap v434: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 26 MiB/s wr, 560 op/s 2024-04-05T23:47:22.187 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:47:22.187 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:22.187 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 260, 2024-04-05T23:47:22.187 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 249, 2024-04-05T23:47:22.188 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1570304, 2024-04-05T23:47:22.188 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 142160, 2024-04-05T23:47:22.188 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-05T23:47:22.188 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-05T23:47:22.188 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.064838740, 2024-04-05T23:47:22.188 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.020664273, 2024-04-05T23:47:22.188 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.052003510, 2024-04-05T23:47:22.188 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.005293175, 2024-04-05T23:47:22.189 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:22.189 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 248, 2024-04-05T23:47:22.189 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.013911939, 2024-04-05T23:47:22.189 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000056096 2024-04-05T23:47:22.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.189 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:22.189 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:22.189 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:22.189 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:22.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1570304, 2024-04-05T23:47:22.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 142160 2024-04-05T23:47:22.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.190 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:22.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2, 2024-04-05T23:47:22.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 190, 2024-04-05T23:47:22.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 320, 2024-04-05T23:47:22.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 185680, 2024-04-05T23:47:22.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-04-05T23:47:22.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-04-05T23:47:22.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.022355997, 2024-04-05T23:47:22.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.014465961, 2024-04-05T23:47:22.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.004272434, 2024-04-05T23:47:22.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-05T23:47:22.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:22.191 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 187, 2024-04-05T23:47:22.192 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.018596667, 2024-04-05T23:47:22.192 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000099447 2024-04-05T23:47:22.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.192 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:22.192 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:22.192 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:22.192 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:22.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-05T23:47:22.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 185680 2024-04-05T23:47:22.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.193 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:22.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 4, 2024-04-05T23:47:22.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 4, 2024-04-05T23:47:22.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 2944, 2024-04-05T23:47:22.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 448, 2024-04-05T23:47:22.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-05T23:47:22.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-05T23:47:22.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001327273, 2024-04-05T23:47:22.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000075919, 2024-04-05T23:47:22.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001201212, 2024-04-05T23:47:22.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-05T23:47:22.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:22.195 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3, 2024-04-05T23:47:22.195 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.001164209, 2024-04-05T23:47:22.195 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000388069 2024-04-05T23:47:22.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.195 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:22.195 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:22.195 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:22.195 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:22.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-05T23:47:22.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-04-05T23:47:22.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.196 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-05T23:47:22.196 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-05T23:47:22.196 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-05T23:47:22.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.196 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-05T23:47:22.196 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:22.197 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:22.197 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:22.197 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:22.197 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:22.197 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:22.197 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:22.197 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:22.197 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:22.198 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:22.198 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:22.198 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:22.198 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:22.198 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:22.198 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:22.198 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:22.198 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:22.198 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:22.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:22.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:22.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:22.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:22.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:22.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:22.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:22.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:22.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:22.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:22.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:22.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:22.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:22.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-05T23:47:22.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:22.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:22.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:22.201 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:22.201 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 151039, 2024-04-05T23:47:22.201 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 32, 2024-04-05T23:47:22.202 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-05T23:47:22.202 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-05T23:47:22.202 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-05T23:47:22.202 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-05T23:47:22.202 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:22.202 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:22.203 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:22.203 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:22.203 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-05T23:47:22.203 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-05T23:47:22.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:22.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:22.204 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:22.204 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-05T23:47:22.204 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-05T23:47:22.204 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-05T23:47:22.205 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:22.205 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-05T23:47:22.205 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:22.205 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-05T23:47:22.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:22.205 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:22.205 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:22.205 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-05T23:47:22.205 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:22.206 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 262, 2024-04-05T23:47:22.206 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1533588, 2024-04-05T23:47:22.206 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:22.206 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 262, 2024-04-05T23:47:22.206 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-05T23:47:22.206 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-05T23:47:22.206 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 262, 2024-04-05T23:47:22.206 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1533588, 2024-04-05T23:47:22.207 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:22.207 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:22.207 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:22.207 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:22.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:22.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:22.207 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:47:22.614 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-05T23:47:22.614 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-05T23:47:22.614 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-05T23:47:22.614 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-05T23:47:22.615 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-05T23:47:22.615 DEBUG:tasks.check_counter:Getting stats from j 2024-04-05T23:47:22.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.j perf dump 2024-04-05T23:47:22.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:22 smithi027 ceph-mon[78597]: pgmap v435: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 164 op/s 2024-04-05T23:47:22.993 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-05T23:47:23.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:22 smithi016 ceph-mon[38248]: pgmap v435: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 164 op/s 2024-04-05T23:47:23.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:22 smithi130 ceph-mon[79417]: pgmap v435: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 164 op/s 2024-04-05T23:47:23.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:23 smithi016 ceph-mon[38248]: pgmap v436: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 164 op/s 2024-04-05T23:47:23.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:23 smithi027 ceph-mon[78597]: pgmap v436: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 164 op/s 2024-04-05T23:47:24.082 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:23 smithi130 ceph-mon[79417]: pgmap v436: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 164 op/s 2024-04-05T23:47:24.189 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-05T23:47:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 264, 2024-04-05T23:47:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 253, 2024-04-05T23:47:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1571392, 2024-04-05T23:47:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 144528, 2024-04-05T23:47:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-05T23:47:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-05T23:47:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.061426446, 2024-04-05T23:47:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.020975790, 2024-04-05T23:47:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.044036443, 2024-04-05T23:47:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.005258487, 2024-04-05T23:47:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 252, 2024-04-05T23:47:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.015472465, 2024-04-05T23:47:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000061398 2024-04-05T23:47:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1571392, 2024-04-05T23:47:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 144528 2024-04-05T23:47:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2, 2024-04-05T23:47:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 193, 2024-04-05T23:47:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 320, 2024-04-05T23:47:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 186208, 2024-04-05T23:47:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-04-05T23:47:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-04-05T23:47:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.021228060, 2024-04-05T23:47:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.014318289, 2024-04-05T23:47:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.003274858, 2024-04-05T23:47:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-05T23:47:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 190, 2024-04-05T23:47:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.029907093, 2024-04-05T23:47:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000157405 2024-04-05T23:47:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-05T23:47:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 186208 2024-04-05T23:47:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 4, 2024-04-05T23:47:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 4, 2024-04-05T23:47:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 2944, 2024-04-05T23:47:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 448, 2024-04-05T23:47:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-05T23:47:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-05T23:47:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001726533, 2024-04-05T23:47:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000060047, 2024-04-05T23:47:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001593292, 2024-04-05T23:47:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-05T23:47:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3, 2024-04-05T23:47:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.001633179, 2024-04-05T23:47:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000544393 2024-04-05T23:47:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-05T23:47:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-04-05T23:47:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-05T23:47:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-05T23:47:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-05T23:47:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-05T23:47:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-05T23:47:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 151039, 2024-04-05T23:47:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 32, 2024-04-05T23:47:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-05T23:47:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-05T23:47:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-05T23:47:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-05T23:47:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-05T23:47:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-05T23:47:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-05T23:47:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-05T23:47:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-05T23:47:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-05T23:47:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-05T23:47:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-05T23:47:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-05T23:47:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-05T23:47:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-05T23:47:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 266, 2024-04-05T23:47:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1534108, 2024-04-05T23:47:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 266, 2024-04-05T23:47:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-05T23:47:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-05T23:47:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 266, 2024-04-05T23:47:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1534108, 2024-04-05T23:47:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-05T23:47:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-05T23:47:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-05T23:47:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-05T23:47:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-05T23:47:24.208 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-05T23:47:24.600 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-05T23:47:24.601 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-05T23:47:24.601 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-05T23:47:24.601 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-05T23:47:24.601 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-05T23:47:24.601 DEBUG:tasks.check_counter:Getting stats from b 2024-04-05T23:47:24.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.b perf dump 2024-04-05T23:47:24.856 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:47:25.965 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:25 smithi027 ceph-mon[78597]: pgmap v437: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 720 KiB/s wr, 161 op/s 2024-04-05T23:47:25.996 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-05T23:47:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 132721, 2024-04-05T23:47:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 159246, 2024-04-05T23:47:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 41727732, 2024-04-05T23:47:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 236544198, 2024-04-05T23:47:25.997 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 21, 2024-04-05T23:47:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 6, 2024-04-05T23:47:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 7.661055783, 2024-04-05T23:47:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 3.215721894, 2024-04-05T23:47:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 4.854029676, 2024-04-05T23:47:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.458068166, 2024-04-05T23:47:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 159238, 2024-04-05T23:47:25.998 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 11.228303624, 2024-04-05T23:47:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000070512 2024-04-05T23:47:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 208801, 2024-04-05T23:47:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.038857744, 2024-04-05T23:47:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000186 2024-04-05T23:47:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 51088, 2024-04-05T23:47:25.999 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 412368 2024-04-05T23:47:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 82745, 2024-04-05T23:47:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 102547, 2024-04-05T23:47:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 23338130, 2024-04-05T23:47:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 172208746, 2024-04-05T23:47:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 13, 2024-04-05T23:47:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 6, 2024-04-05T23:47:26.000 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 5.347019441, 2024-04-05T23:47:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 2.033541030, 2024-04-05T23:47:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 2.878604923, 2024-04-05T23:47:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.658479053, 2024-04-05T23:47:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 102548, 2024-04-05T23:47:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 9.025015532, 2024-04-05T23:47:26.001 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000088007 2024-04-05T23:47:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 139668, 2024-04-05T23:47:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.026096422, 2024-04-05T23:47:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000186 2024-04-05T23:47:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 2448480, 2024-04-05T23:47:26.002 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 296544 2024-04-05T23:47:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 166563, 2024-04-05T23:47:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 229965, 2024-04-05T23:47:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 40132305, 2024-04-05T23:47:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 101761629, 2024-04-05T23:47:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 148, 2024-04-05T23:47:26.003 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 6, 2024-04-05T23:47:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 9.021775653, 2024-04-05T23:47:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 3.527800091, 2024-04-05T23:47:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 6.222594623, 2024-04-05T23:47:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.185175662, 2024-04-05T23:47:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 229966, 2024-04-05T23:47:26.004 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 6.102641043, 2024-04-05T23:47:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000026537 2024-04-05T23:47:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 384049, 2024-04-05T23:47:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.061074633, 2024-04-05T23:47:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000159 2024-04-05T23:47:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 45328, 2024-04-05T23:47:26.005 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 115920 2024-04-05T23:47:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-05T23:47:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 1, 2024-04-05T23:47:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-05T23:47:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-MDSRank": { 2024-04-05T23:47:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-05T23:47:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-05T23:47:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 68338, 2024-04-05T23:47:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 39.912240885, 2024-04-05T23:47:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000584041 2024-04-05T23:47:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-05T23:47:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-05T23:47:26.007 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 10062, 2024-04-05T23:47:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 2.024531078, 2024-04-05T23:47:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000201205 2024-04-05T23:47:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-05T23:47:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "request": 72303, 2024-04-05T23:47:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "reply": 61116, 2024-04-05T23:47:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "reply_latency": { 2024-04-05T23:47:26.008 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 61116, 2024-04-05T23:47:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 477.612480880, 2024-04-05T23:47:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.007814851 2024-04-05T23:47:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_reply": 0, 2024-04-05T23:47:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "forward": 11187, 2024-04-05T23:47:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_complete": 23, 2024-04-05T23:47:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_keys": 2, 2024-04-05T23:47:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_commit": 1215, 2024-04-05T23:47:26.009 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_split": 3, 2024-04-05T23:47:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_merge": 0, 2024-04-05T23:47:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes": 1276, 2024-04-05T23:47:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_top": 0, 2024-04-05T23:47:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_bottom": 0, 2024-04-05T23:47:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pin_tail": 1276, 2024-04-05T23:47:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pinned": 1276, 2024-04-05T23:47:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_expired": 3123, 2024-04-05T23:47:26.010 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_with_caps": 127, 2024-04-05T23:47:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": 127, 2024-04-05T23:47:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "subtrees": 14, 2024-04-05T23:47:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse": 109996, 2024-04-05T23:47:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_hit": 83053, 2024-04-05T23:47:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_forward": 11169, 2024-04-05T23:47:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_discover": 36, 2024-04-05T23:47:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_dir_fetch": 0, 2024-04-05T23:47:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:26.011 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_lock": 109, 2024-04-05T23:47:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "load_cent": 98527, 2024-04-05T23:47:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "q": 32, 2024-04-05T23:47:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "exported": 42, 2024-04-05T23:47:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "exported_inodes": 2251, 2024-04-05T23:47:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "imported": 21, 2024-04-05T23:47:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "imported_inodes": 868, 2024-04-05T23:47:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_backtrace_fetch": 97, 2024-04-05T23:47:26.012 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rfiles": 614, 2024-04-05T23:47:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rbytes": 211488726, 2024-04-05T23:47:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rsnaps": 6, 2024-04-05T23:47:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_fetch": 97, 2024-04-05T23:47:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:26.013 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_inodes": 36, 2024-04-05T23:47:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_base_inodes": 3, 2024-04-05T23:47:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dirfrag_rstats": 33, 2024-04-05T23:47:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_file_inodes": 61, 2024-04-05T23:47:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_inode_file_caps": 19, 2024-04-05T23:47:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_revoke": 26406, 2024-04-05T23:47:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_grant": 20831, 2024-04-05T23:47:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:26.014 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps": 35392, 2024-04-05T23:47:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps_dirty": 5589, 2024-04-05T23:47:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_cap_release": 2796, 2024-04-05T23:47:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "process_request_cap_release": 24452 2024-04-05T23:47:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache": { 2024-04-05T23:47:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update": 5, 2024-04-05T23:47:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update_receipt": 0, 2024-04-05T23:47:26.015 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_try_discover": 0, 2024-04-05T23:47:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_send_discover": 36, 2024-04-05T23:47:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_handle_discover": 219, 2024-04-05T23:47:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays": 19, 2024-04-05T23:47:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_created": 5122, 2024-04-05T23:47:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_enqueued": 5103, 2024-04-05T23:47:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:26.016 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_migrated": 0, 2024-04-05T23:47:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_started": 0, 2024-04-05T23:47:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_completed": 0, 2024-04-05T23:47:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_enqueue_scrub": 3, 2024-04-05T23:47:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_exportdir": 43, 2024-04-05T23:47:26.017 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_flush": 0, 2024-04-05T23:47:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragmentdir": 3, 2024-04-05T23:47:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_inodestats": 0 2024-04-05T23:47:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log": { 2024-04-05T23:47:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "evadd": 53864, 2024-04-05T23:47:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "evex": 49884, 2024-04-05T23:47:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "evtrm": 49884, 2024-04-05T23:47:26.018 INFO:teuthology.orchestra.run.smithi027.stdout: "ev": 3980, 2024-04-05T23:47:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "evexg": 0, 2024-04-05T23:47:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "evexd": 7643, 2024-04-05T23:47:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "segadd": 139, 2024-04-05T23:47:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "segex": 132, 2024-04-05T23:47:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "segtrm": 132, 2024-04-05T23:47:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "seg": 7, 2024-04-05T23:47:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "segexg": 0, 2024-04-05T23:47:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "segexd": 10, 2024-04-05T23:47:26.019 INFO:teuthology.orchestra.run.smithi027.stdout: "expos": 274558983, 2024-04-05T23:47:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "wrpos": 285814467, 2024-04-05T23:47:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "rdpos": 0, 2024-04-05T23:47:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "jlat": { 2024-04-05T23:47:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 35189, 2024-04-05T23:47:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 98.900594437, 2024-04-05T23:47:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.002810554 2024-04-05T23:47:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: "replayed": 0 2024-04-05T23:47:26.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mem": { 2024-04-05T23:47:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 233, 2024-04-05T23:47:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "ino+": 7264, 2024-04-05T23:47:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "ino-": 7031, 2024-04-05T23:47:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "dir": 89, 2024-04-05T23:47:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "dir+": 194, 2024-04-05T23:47:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "dir-": 105, 2024-04-05T23:47:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "dn": 1325, 2024-04-05T23:47:26.021 INFO:teuthology.orchestra.run.smithi027.stdout: "dn+": 14674, 2024-04-05T23:47:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "dn-": 13349, 2024-04-05T23:47:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "cap": 122, 2024-04-05T23:47:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "cap+": 6108, 2024-04-05T23:47:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "cap-": 5986, 2024-04-05T23:47:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "rss": 82124, 2024-04-05T23:47:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "heap": 207156 2024-04-05T23:47:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server": { 2024-04-05T23:47:26.022 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_client_request": 109970, 2024-04-05T23:47:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_server_request": 25610, 2024-04-05T23:47:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_request": 72303, 2024-04-05T23:47:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_session": 844, 2024-04-05T23:47:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_peer_request": 51420, 2024-04-05T23:47:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "req_create_latency": { 2024-04-05T23:47:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 5717, 2024-04-05T23:47:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 36.788526724, 2024-04-05T23:47:26.023 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.006434935 2024-04-05T23:47:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getattr_latency": { 2024-04-05T23:47:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 26615, 2024-04-05T23:47:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 148.919549255, 2024-04-05T23:47:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.005595324 2024-04-05T23:47:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.024 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "req_link_latency": { 2024-04-05T23:47:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.025 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookup_latency": { 2024-04-05T23:47:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 7797, 2024-04-05T23:47:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 79.678015366, 2024-04-05T23:47:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.010219060 2024-04-05T23:47:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupino_latency": { 2024-04-05T23:47:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1, 2024-04-05T23:47:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000172333, 2024-04-05T23:47:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000172333 2024-04-05T23:47:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupname_latency": { 2024-04-05T23:47:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.027 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.028 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.028 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:26.028 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.028 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.028 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.028 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:26.028 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.029 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.029 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.029 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lssnap_latency": { 2024-04-05T23:47:26.029 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 14, 2024-04-05T23:47:26.029 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.002875802, 2024-04-05T23:47:26.029 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000205414 2024-04-05T23:47:26.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.029 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mkdir_latency": { 2024-04-05T23:47:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 119, 2024-04-05T23:47:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.720353531, 2024-04-05T23:47:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.006053391 2024-04-05T23:47:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mknod_latency": { 2024-04-05T23:47:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.030 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mksnap_latency": { 2024-04-05T23:47:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 14, 2024-04-05T23:47:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 10.595986906, 2024-04-05T23:47:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.756856207 2024-04-05T23:47:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "req_open_latency": { 2024-04-05T23:47:26.031 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 9, 2024-04-05T23:47:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.011937681, 2024-04-05T23:47:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.001326409 2024-04-05T23:47:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "req_readdir_latency": { 2024-04-05T23:47:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 14766, 2024-04-05T23:47:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 80.019220648, 2024-04-05T23:47:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.005419153 2024-04-05T23:47:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.032 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rename_latency": { 2024-04-05T23:47:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 817, 2024-04-05T23:47:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 71.663501282, 2024-04-05T23:47:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.087715423 2024-04-05T23:47:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmdir_latency": { 2024-04-05T23:47:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1, 2024-04-05T23:47:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000741169, 2024-04-05T23:47:26.034 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000741169 2024-04-05T23:47:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 8, 2024-04-05T23:47:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 11.262877393, 2024-04-05T23:47:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 1.407859674 2024-04-05T23:47:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:26.035 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setattr_latency": { 2024-04-05T23:47:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1, 2024-04-05T23:47:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.004586108, 2024-04-05T23:47:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.004586108 2024-04-05T23:47:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.036 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 108, 2024-04-05T23:47:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.637158918, 2024-04-05T23:47:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.005899619 2024-04-05T23:47:26.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setlayout_latency": { 2024-04-05T23:47:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setxattr_latency": { 2024-04-05T23:47:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 7, 2024-04-05T23:47:26.038 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.022252038, 2024-04-05T23:47:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.003178862 2024-04-05T23:47:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "req_symlink_latency": { 2024-04-05T23:47:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.039 INFO:teuthology.orchestra.run.smithi027.stdout: "req_unlink_latency": { 2024-04-05T23:47:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 5122, 2024-04-05T23:47:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 37.284725726, 2024-04-05T23:47:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.007279329 2024-04-05T23:47:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:26.040 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessions": { 2024-04-05T23:47:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "session_count": 2, 2024-04-05T23:47:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "session_add": 5, 2024-04-05T23:47:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "session_remove": 3, 2024-04-05T23:47:26.041 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_open": 2, 2024-04-05T23:47:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_stale": 0, 2024-04-05T23:47:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "total_load": 14035, 2024-04-05T23:47:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "average_load": 7017, 2024-04-05T23:47:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "avg_session_uptime": 815, 2024-04-05T23:47:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-05T23:47:26.042 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 360, 2024-04-05T23:47:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 360, 2024-04-05T23:47:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:26.043 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:26.044 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:26.045 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-05T23:47:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:26.046 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 355859, 2024-04-05T23:47:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 669, 2024-04-05T23:47:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 176, 2024-04-05T23:47:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 2, 2024-04-05T23:47:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-05T23:47:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-05T23:47:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:26.047 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 188, 2024-04-05T23:47:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-05T23:47:26.048 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 3557417, 2024-04-05T23:47:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 37496, 2024-04-05T23:47:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-05T23:47:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-05T23:47:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter": { 2024-04-05T23:47:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "op_active": 0, 2024-04-05T23:47:26.049 INFO:teuthology.orchestra.run.smithi027.stdout: "op_laggy": 0, 2024-04-05T23:47:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send": 47667, 2024-04-05T23:47:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send_bytes": 290992383, 2024-04-05T23:47:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "op_resend": 37, 2024-04-05T23:47:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "op_reply": 47667, 2024-04-05T23:47:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "op_latency": { 2024-04-05T23:47:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 47667, 2024-04-05T23:47:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 364.934631565, 2024-04-05T23:47:26.050 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.007655917 2024-04-05T23:47:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "op_inflight": 0, 2024-04-05T23:47:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "oplen_avg": { 2024-04-05T23:47:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 47667, 2024-04-05T23:47:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 58462 2024-04-05T23:47:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "op": 47667, 2024-04-05T23:47:26.051 INFO:teuthology.orchestra.run.smithi027.stdout: "op_r": 467, 2024-04-05T23:47:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "op_w": 47200, 2024-04-05T23:47:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "op_rmw": 0, 2024-04-05T23:47:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "op_pg": 0, 2024-04-05T23:47:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_stat": 1060, 2024-04-05T23:47:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_create": 3981, 2024-04-05T23:47:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_read": 342, 2024-04-05T23:47:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_write": 35528, 2024-04-05T23:47:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writefull": 447, 2024-04-05T23:47:26.052 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writesame": 0, 2024-04-05T23:47:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_append": 0, 2024-04-05T23:47:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_zero": 0, 2024-04-05T23:47:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_truncate": 3, 2024-04-05T23:47:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_delete": 5746, 2024-04-05T23:47:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_mapext": 0, 2024-04-05T23:47:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_getxattr": 116, 2024-04-05T23:47:26.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_setxattr": 7962, 2024-04-05T23:47:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_call": 0, 2024-04-05T23:47:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_watch": 0, 2024-04-05T23:47:26.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_notify": 0, 2024-04-05T23:47:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls": 0, 2024-04-05T23:47:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_other": 0, 2024-04-05T23:47:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_active": 0, 2024-04-05T23:47:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_send": 0, 2024-04-05T23:47:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_resend": 0, 2024-04-05T23:47:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_ping": 0, 2024-04-05T23:47:26.055 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_active": 0, 2024-04-05T23:47:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_send": 0, 2024-04-05T23:47:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_resend": 0, 2024-04-05T23:47:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_active": 0, 2024-04-05T23:47:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_send": 0, 2024-04-05T23:47:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_resend": 0, 2024-04-05T23:47:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_active": 0, 2024-04-05T23:47:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_send": 0, 2024-04-05T23:47:26.056 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_resend": 0, 2024-04-05T23:47:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "command_active": 0, 2024-04-05T23:47:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "command_send": 0, 2024-04-05T23:47:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "command_resend": 0, 2024-04-05T23:47:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "map_epoch": 101, 2024-04-05T23:47:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "map_full": 0, 2024-04-05T23:47:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "map_inc": 23, 2024-04-05T23:47:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_sessions": 12, 2024-04-05T23:47:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_open": 12, 2024-04-05T23:47:26.057 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_close": 0, 2024-04-05T23:47:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_laggy": 0, 2024-04-05T23:47:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_wr": 2726, 2024-04-05T23:47:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_rd": 50, 2024-04-05T23:47:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_del": 501 2024-04-05T23:47:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "oft": { 2024-04-05T23:47:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_objs": 1, 2024-04-05T23:47:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:26.058 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_updates": 0, 2024-04-05T23:47:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_removes": 0 2024-04-05T23:47:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "osd-slow-ops": { 2024-04-05T23:47:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_ops_count": 0 2024-04-05T23:47:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "purge_queue": { 2024-04-05T23:47:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:26.059 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops_high_water": 6, 2024-04-05T23:47:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing": 0, 2024-04-05T23:47:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_high_water": 6, 2024-04-05T23:47:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed": 5103, 2024-04-05T23:47:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-05T23:47:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 382025, 2024-04-05T23:47:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 70733789, 2024-04-05T23:47:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 382025, 2024-04-05T23:47:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 382025, 2024-04-05T23:47:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 70733789, 2024-04-05T23:47:26.061 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:26.062 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-05T23:47:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 47667, 2024-04-05T23:47:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 291508722, 2024-04-05T23:47:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 47667, 2024-04-05T23:47:26.063 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 291508722, 2024-04-05T23:47:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:26.064 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1024, 2024-04-05T23:47:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 47667, 2024-04-05T23:47:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 47667, 2024-04-05T23:47:26.065 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 47667, 2024-04-05T23:47:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 47667, 2024-04-05T23:47:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-05T23:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 5103, 2024-04-05T23:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 516715, 2024-04-05T23:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 5103, 2024-04-05T23:47:26.067 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 339, 2024-04-05T23:47:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 516715, 2024-04-05T23:47:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.068 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle-0x560428d1a0a0": { 2024-04-05T23:47:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-05T23:47:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 53864, 2024-04-05T23:47:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 281620163, 2024-04-05T23:47:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:26.069 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 53864, 2024-04-05T23:47:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 35189, 2024-04-05T23:47:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 281620163, 2024-04-05T23:47:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:26.070 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:26.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:26.071 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-05T23:47:26.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:25 smithi016 ceph-mon[38248]: pgmap v437: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 720 KiB/s wr, 161 op/s 2024-04-05T23:47:26.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:25 smithi130 ceph-mon[79417]: pgmap v437: 97 pgs: 97 active+clean; 503 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 720 KiB/s wr, 161 op/s 2024-04-05T23:47:26.362 INFO:tasks.check_counter:Daemon mds.b mds.exported=42 2024-04-05T23:47:26.362 INFO:tasks.check_counter:Daemon mds.b mds.imported=21 2024-04-05T23:47:26.363 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-05T23:47:26.363 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=14 2024-04-05T23:47:26.363 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=8 2024-04-05T23:47:26.363 DEBUG:tasks.check_counter:Getting stats from e 2024-04-05T23:47:26.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.e perf dump 2024-04-05T23:47:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:47:27.526 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-05T23:47:27.527 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:27.527 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 712, 2024-04-05T23:47:27.527 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 714, 2024-04-05T23:47:27.527 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 15153394, 2024-04-05T23:47:27.527 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 221496, 2024-04-05T23:47:27.527 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 8, 2024-04-05T23:47:27.527 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-05T23:47:27.528 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.067944731, 2024-04-05T23:47:27.528 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.018926215, 2024-04-05T23:47:27.528 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.024055266, 2024-04-05T23:47:27.528 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.017836497, 2024-04-05T23:47:27.528 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:27.528 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 712, 2024-04-05T23:47:27.528 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.035280307, 2024-04-05T23:47:27.528 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000049550 2024-04-05T23:47:27.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.529 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:27.529 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.529 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.529 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.529 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-05T23:47:27.529 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-05T23:47:27.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.530 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:27.530 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 945, 2024-04-05T23:47:27.530 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 1136, 2024-04-05T23:47:27.530 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 25408185, 2024-04-05T23:47:27.530 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 782750, 2024-04-05T23:47:27.530 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 8, 2024-04-05T23:47:27.530 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 6, 2024-04-05T23:47:27.531 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.099370608, 2024-04-05T23:47:27.531 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.025009145, 2024-04-05T23:47:27.531 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.039080022, 2024-04-05T23:47:27.531 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.025949540, 2024-04-05T23:47:27.531 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:27.531 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1133, 2024-04-05T23:47:27.531 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.045237451, 2024-04-05T23:47:27.531 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000039927 2024-04-05T23:47:27.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.532 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:27.532 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.532 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.532 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.532 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-05T23:47:27.532 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 489472 2024-04-05T23:47:27.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.533 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:27.533 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 1809, 2024-04-05T23:47:27.533 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 1735, 2024-04-05T23:47:27.533 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 19183560, 2024-04-05T23:47:27.533 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 601266, 2024-04-05T23:47:27.533 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-05T23:47:27.533 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 4, 2024-04-05T23:47:27.533 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.185357541, 2024-04-05T23:47:27.534 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.050342460, 2024-04-05T23:47:27.534 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.083383614, 2024-04-05T23:47:27.534 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.048384182, 2024-04-05T23:47:27.534 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:27.534 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1735, 2024-04-05T23:47:27.534 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.073892482, 2024-04-05T23:47:27.535 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000042589 2024-04-05T23:47:27.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.535 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:27.535 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.535 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.535 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.535 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1962224, 2024-04-05T23:47:27.536 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 149072 2024-04-05T23:47:27.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.536 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-05T23:47:27.536 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 1, 2024-04-05T23:47:27.536 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-05T23:47:27.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.536 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-MDSRank": { 2024-04-05T23:47:27.536 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-05T23:47:27.536 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-05T23:47:27.537 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 5286, 2024-04-05T23:47:27.537 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.124850800, 2024-04-05T23:47:27.537 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000023619 2024-04-05T23:47:27.537 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:27.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.537 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:27.537 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-05T23:47:27.537 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-05T23:47:27.538 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.538 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.538 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:27.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-05T23:47:27.538 INFO:teuthology.orchestra.run.smithi027.stdout: "request": 0, 2024-04-05T23:47:27.538 INFO:teuthology.orchestra.run.smithi027.stdout: "reply": 0, 2024-04-05T23:47:27.539 INFO:teuthology.orchestra.run.smithi027.stdout: "reply_latency": { 2024-04-05T23:47:27.539 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.539 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.539 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.539 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_reply": 0, 2024-04-05T23:47:27.540 INFO:teuthology.orchestra.run.smithi027.stdout: "forward": 0, 2024-04-05T23:47:27.540 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_complete": 0, 2024-04-05T23:47:27.540 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_keys": 0, 2024-04-05T23:47:27.540 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_commit": 0, 2024-04-05T23:47:27.540 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_split": 0, 2024-04-05T23:47:27.540 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_merge": 0, 2024-04-05T23:47:27.540 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes": 7796, 2024-04-05T23:47:27.541 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_top": 246, 2024-04-05T23:47:27.541 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_bottom": 7550, 2024-04-05T23:47:27.541 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pin_tail": 0, 2024-04-05T23:47:27.541 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pinned": 7445, 2024-04-05T23:47:27.541 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_expired": 0, 2024-04-05T23:47:27.541 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_with_caps": 0, 2024-04-05T23:47:27.541 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": 0, 2024-04-05T23:47:27.541 INFO:teuthology.orchestra.run.smithi027.stdout: "subtrees": 7, 2024-04-05T23:47:27.541 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse": 0, 2024-04-05T23:47:27.542 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_hit": 0, 2024-04-05T23:47:27.542 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_forward": 0, 2024-04-05T23:47:27.542 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_discover": 0, 2024-04-05T23:47:27.542 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_dir_fetch": 0, 2024-04-05T23:47:27.542 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:27.542 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_lock": 0, 2024-04-05T23:47:27.542 INFO:teuthology.orchestra.run.smithi027.stdout: "load_cent": 0, 2024-04-05T23:47:27.543 INFO:teuthology.orchestra.run.smithi027.stdout: "q": 0, 2024-04-05T23:47:27.543 INFO:teuthology.orchestra.run.smithi027.stdout: "exported": 0, 2024-04-05T23:47:27.543 INFO:teuthology.orchestra.run.smithi027.stdout: "exported_inodes": 0, 2024-04-05T23:47:27.543 INFO:teuthology.orchestra.run.smithi027.stdout: "imported": 0, 2024-04-05T23:47:27.543 INFO:teuthology.orchestra.run.smithi027.stdout: "imported_inodes": 0, 2024-04-05T23:47:27.543 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:27.543 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_backtrace_fetch": 0, 2024-04-05T23:47:27.543 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:27.544 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rfiles": 0, 2024-04-05T23:47:27.544 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rbytes": 0, 2024-04-05T23:47:27.544 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rsnaps": 0, 2024-04-05T23:47:27.544 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_fetch": 0, 2024-04-05T23:47:27.544 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:27.544 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:27.545 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:27.545 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_inodes": 0, 2024-04-05T23:47:27.545 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_base_inodes": 0, 2024-04-05T23:47:27.545 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dirfrag_rstats": 0, 2024-04-05T23:47:27.545 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_file_inodes": 0, 2024-04-05T23:47:27.546 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_inode_file_caps": 0, 2024-04-05T23:47:27.546 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_revoke": 0, 2024-04-05T23:47:27.546 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_grant": 0, 2024-04-05T23:47:27.546 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:27.546 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:27.546 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:27.546 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps": 0, 2024-04-05T23:47:27.547 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps_dirty": 0, 2024-04-05T23:47:27.547 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_cap_release": 0, 2024-04-05T23:47:27.547 INFO:teuthology.orchestra.run.smithi027.stdout: "process_request_cap_release": 0 2024-04-05T23:47:27.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.547 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache": { 2024-04-05T23:47:27.547 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update": 0, 2024-04-05T23:47:27.547 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update_receipt": 0, 2024-04-05T23:47:27.548 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_try_discover": 0, 2024-04-05T23:47:27.548 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_send_discover": 0, 2024-04-05T23:47:27.548 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_handle_discover": 0, 2024-04-05T23:47:27.548 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays": 0, 2024-04-05T23:47:27.548 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:27.548 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:27.548 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_created": 0, 2024-04-05T23:47:27.548 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_enqueued": 0, 2024-04-05T23:47:27.549 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:27.549 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_migrated": 0, 2024-04-05T23:47:27.549 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:27.549 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:27.549 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:27.549 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_started": 0, 2024-04-05T23:47:27.550 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_completed": 0, 2024-04-05T23:47:27.550 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_enqueue_scrub": 0, 2024-04-05T23:47:27.550 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_exportdir": 0, 2024-04-05T23:47:27.550 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_flush": 0, 2024-04-05T23:47:27.550 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragmentdir": 0, 2024-04-05T23:47:27.550 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:27.550 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_inodestats": 0 2024-04-05T23:47:27.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.551 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log": { 2024-04-05T23:47:27.551 INFO:teuthology.orchestra.run.smithi027.stdout: "evadd": 0, 2024-04-05T23:47:27.551 INFO:teuthology.orchestra.run.smithi027.stdout: "evex": 0, 2024-04-05T23:47:27.551 INFO:teuthology.orchestra.run.smithi027.stdout: "evtrm": 0, 2024-04-05T23:47:27.551 INFO:teuthology.orchestra.run.smithi027.stdout: "ev": 34040, 2024-04-05T23:47:27.551 INFO:teuthology.orchestra.run.smithi027.stdout: "evexg": 0, 2024-04-05T23:47:27.551 INFO:teuthology.orchestra.run.smithi027.stdout: "evexd": 0, 2024-04-05T23:47:27.552 INFO:teuthology.orchestra.run.smithi027.stdout: "segadd": 0, 2024-04-05T23:47:27.552 INFO:teuthology.orchestra.run.smithi027.stdout: "segex": 0, 2024-04-05T23:47:27.552 INFO:teuthology.orchestra.run.smithi027.stdout: "segtrm": 0, 2024-04-05T23:47:27.552 INFO:teuthology.orchestra.run.smithi027.stdout: "seg": 39, 2024-04-05T23:47:27.552 INFO:teuthology.orchestra.run.smithi027.stdout: "segexg": 0, 2024-04-05T23:47:27.552 INFO:teuthology.orchestra.run.smithi027.stdout: "segexd": 0, 2024-04-05T23:47:27.552 INFO:teuthology.orchestra.run.smithi027.stdout: "expos": 4194304, 2024-04-05T23:47:27.553 INFO:teuthology.orchestra.run.smithi027.stdout: "wrpos": 61107859, 2024-04-05T23:47:27.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rdpos": 61086917, 2024-04-05T23:47:27.553 INFO:teuthology.orchestra.run.smithi027.stdout: "jlat": { 2024-04-05T23:47:27.553 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.553 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.553 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.553 INFO:teuthology.orchestra.run.smithi027.stdout: "replayed": 34040 2024-04-05T23:47:27.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.554 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mem": { 2024-04-05T23:47:27.554 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 647, 2024-04-05T23:47:27.554 INFO:teuthology.orchestra.run.smithi027.stdout: "ino+": 7707, 2024-04-05T23:47:27.554 INFO:teuthology.orchestra.run.smithi027.stdout: "ino-": 7060, 2024-04-05T23:47:27.554 INFO:teuthology.orchestra.run.smithi027.stdout: "dir": 53, 2024-04-05T23:47:27.554 INFO:teuthology.orchestra.run.smithi027.stdout: "dir+": 69, 2024-04-05T23:47:27.555 INFO:teuthology.orchestra.run.smithi027.stdout: "dir-": 16, 2024-04-05T23:47:27.555 INFO:teuthology.orchestra.run.smithi027.stdout: "dn": 7796, 2024-04-05T23:47:27.555 INFO:teuthology.orchestra.run.smithi027.stdout: "dn+": 8010, 2024-04-05T23:47:27.555 INFO:teuthology.orchestra.run.smithi027.stdout: "dn-": 214, 2024-04-05T23:47:27.555 INFO:teuthology.orchestra.run.smithi027.stdout: "cap": 0, 2024-04-05T23:47:27.555 INFO:teuthology.orchestra.run.smithi027.stdout: "cap+": 0, 2024-04-05T23:47:27.556 INFO:teuthology.orchestra.run.smithi027.stdout: "cap-": 0, 2024-04-05T23:47:27.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rss": 59040, 2024-04-05T23:47:27.556 INFO:teuthology.orchestra.run.smithi027.stdout: "heap": 207156 2024-04-05T23:47:27.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.556 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server": { 2024-04-05T23:47:27.556 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_client_request": 0, 2024-04-05T23:47:27.556 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_server_request": 0, 2024-04-05T23:47:27.556 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_request": 0, 2024-04-05T23:47:27.557 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_session": 0, 2024-04-05T23:47:27.557 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_peer_request": 0, 2024-04-05T23:47:27.557 INFO:teuthology.orchestra.run.smithi027.stdout: "req_create_latency": { 2024-04-05T23:47:27.557 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.557 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.557 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.557 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getattr_latency": { 2024-04-05T23:47:27.558 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.558 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.558 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.558 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:27.558 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.558 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.558 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.559 INFO:teuthology.orchestra.run.smithi027.stdout: "req_link_latency": { 2024-04-05T23:47:27.559 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.559 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.559 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.559 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookup_latency": { 2024-04-05T23:47:27.559 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupino_latency": { 2024-04-05T23:47:27.561 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.561 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.561 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.561 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupname_latency": { 2024-04-05T23:47:27.561 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.561 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.561 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.562 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:27.562 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.562 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.562 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.562 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:27.562 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.563 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.563 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.563 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lssnap_latency": { 2024-04-05T23:47:27.563 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.563 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.563 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.564 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mkdir_latency": { 2024-04-05T23:47:27.564 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.564 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.564 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.564 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mknod_latency": { 2024-04-05T23:47:27.564 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.564 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.565 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.565 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mksnap_latency": { 2024-04-05T23:47:27.565 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.565 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.565 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.566 INFO:teuthology.orchestra.run.smithi027.stdout: "req_open_latency": { 2024-04-05T23:47:27.566 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.566 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.566 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.566 INFO:teuthology.orchestra.run.smithi027.stdout: "req_readdir_latency": { 2024-04-05T23:47:27.567 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.567 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.567 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.567 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rename_latency": { 2024-04-05T23:47:27.567 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.567 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.567 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.568 INFO:teuthology.orchestra.run.smithi027.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:27.568 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.568 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.568 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.568 INFO:teuthology.orchestra.run.smithi027.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:27.569 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.569 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.569 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.569 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmdir_latency": { 2024-04-05T23:47:27.569 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.569 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.570 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.570 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:27.570 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.570 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.570 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.570 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:27.571 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.571 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.571 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.571 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setattr_latency": { 2024-04-05T23:47:27.571 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.571 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.571 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.572 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:27.572 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.572 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.572 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.572 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:27.572 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.573 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.573 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.573 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setlayout_latency": { 2024-04-05T23:47:27.573 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.573 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.573 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.573 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setxattr_latency": { 2024-04-05T23:47:27.574 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.574 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.574 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.574 INFO:teuthology.orchestra.run.smithi027.stdout: "req_symlink_latency": { 2024-04-05T23:47:27.574 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.574 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout: "req_unlink_latency": { 2024-04-05T23:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:27.576 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:27.576 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.576 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.576 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.576 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:27.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.577 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessions": { 2024-04-05T23:47:27.577 INFO:teuthology.orchestra.run.smithi027.stdout: "session_count": 1, 2024-04-05T23:47:27.577 INFO:teuthology.orchestra.run.smithi027.stdout: "session_add": 3, 2024-04-05T23:47:27.577 INFO:teuthology.orchestra.run.smithi027.stdout: "session_remove": 2, 2024-04-05T23:47:27.577 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_open": 3, 2024-04-05T23:47:27.577 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_stale": 0, 2024-04-05T23:47:27.577 INFO:teuthology.orchestra.run.smithi027.stdout: "total_load": 0, 2024-04-05T23:47:27.578 INFO:teuthology.orchestra.run.smithi027.stdout: "average_load": 0, 2024-04-05T23:47:27.578 INFO:teuthology.orchestra.run.smithi027.stdout: "avg_session_uptime": 6446, 2024-04-05T23:47:27.578 INFO:teuthology.orchestra.run.smithi027.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:27.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.578 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-05T23:47:27.578 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:27.579 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:27.579 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:27.579 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:27.579 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:27.579 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:27.579 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:27.579 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:27.579 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:27.580 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:27.580 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:27.580 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:27.580 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:27.580 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:27.580 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:27.580 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:27.581 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:27.581 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:27.581 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:27.581 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:27.581 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:27.581 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:27.581 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:27.581 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:27.581 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:27.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:27.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:27.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:27.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:27.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:27.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:27.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-05T23:47:27.583 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:27.583 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:27.583 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:27.583 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:27.583 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 248715, 2024-04-05T23:47:27.583 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 799, 2024-04-05T23:47:27.583 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-05T23:47:27.583 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-05T23:47:27.583 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-05T23:47:27.584 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-05T23:47:27.584 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:27.584 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:27.584 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:27.584 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:27.584 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:27.584 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 188, 2024-04-05T23:47:27.584 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:27.585 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:27.585 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:27.585 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-05T23:47:27.585 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 7165718, 2024-04-05T23:47:27.585 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 49603, 2024-04-05T23:47:27.585 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:27.585 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-05T23:47:27.586 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:27.586 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-05T23:47:27.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.586 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter": { 2024-04-05T23:47:27.586 INFO:teuthology.orchestra.run.smithi027.stdout: "op_active": 0, 2024-04-05T23:47:27.586 INFO:teuthology.orchestra.run.smithi027.stdout: "op_laggy": 0, 2024-04-05T23:47:27.586 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send": 3109, 2024-04-05T23:47:27.587 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send_bytes": 21, 2024-04-05T23:47:27.587 INFO:teuthology.orchestra.run.smithi027.stdout: "op_resend": 3, 2024-04-05T23:47:27.587 INFO:teuthology.orchestra.run.smithi027.stdout: "op_reply": 3109, 2024-04-05T23:47:27.587 INFO:teuthology.orchestra.run.smithi027.stdout: "op_latency": { 2024-04-05T23:47:27.587 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3109, 2024-04-05T23:47:27.588 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 1.849952408, 2024-04-05T23:47:27.588 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000595031 2024-04-05T23:47:27.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.588 INFO:teuthology.orchestra.run.smithi027.stdout: "op_inflight": 0, 2024-04-05T23:47:27.589 INFO:teuthology.orchestra.run.smithi027.stdout: "oplen_avg": { 2024-04-05T23:47:27.589 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3109, 2024-04-05T23:47:27.589 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 3110 2024-04-05T23:47:27.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.590 INFO:teuthology.orchestra.run.smithi027.stdout: "op": 3109, 2024-04-05T23:47:27.590 INFO:teuthology.orchestra.run.smithi027.stdout: "op_r": 3109, 2024-04-05T23:47:27.590 INFO:teuthology.orchestra.run.smithi027.stdout: "op_w": 0, 2024-04-05T23:47:27.590 INFO:teuthology.orchestra.run.smithi027.stdout: "op_rmw": 0, 2024-04-05T23:47:27.590 INFO:teuthology.orchestra.run.smithi027.stdout: "op_pg": 0, 2024-04-05T23:47:27.590 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_stat": 1745, 2024-04-05T23:47:27.590 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_create": 0, 2024-04-05T23:47:27.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_read": 1362, 2024-04-05T23:47:27.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_write": 0, 2024-04-05T23:47:27.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writefull": 0, 2024-04-05T23:47:27.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writesame": 0, 2024-04-05T23:47:27.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_append": 0, 2024-04-05T23:47:27.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_zero": 0, 2024-04-05T23:47:27.592 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_truncate": 0, 2024-04-05T23:47:27.592 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_delete": 0, 2024-04-05T23:47:27.592 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_mapext": 0, 2024-04-05T23:47:27.592 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:27.592 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:27.592 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_getxattr": 1, 2024-04-05T23:47:27.592 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_setxattr": 0, 2024-04-05T23:47:27.592 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:27.593 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:27.593 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:27.593 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_call": 0, 2024-04-05T23:47:27.593 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_watch": 0, 2024-04-05T23:47:27.593 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_notify": 0, 2024-04-05T23:47:27.593 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:27.593 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls": 0, 2024-04-05T23:47:27.593 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:27.594 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_other": 0, 2024-04-05T23:47:27.594 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_active": 0, 2024-04-05T23:47:27.594 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_send": 0, 2024-04-05T23:47:27.594 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_resend": 0, 2024-04-05T23:47:27.594 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_ping": 0, 2024-04-05T23:47:27.594 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_active": 0, 2024-04-05T23:47:27.594 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_send": 0, 2024-04-05T23:47:27.595 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_resend": 0, 2024-04-05T23:47:27.595 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_active": 0, 2024-04-05T23:47:27.595 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_send": 0, 2024-04-05T23:47:27.595 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_resend": 0, 2024-04-05T23:47:27.595 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_active": 0, 2024-04-05T23:47:27.595 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_send": 0, 2024-04-05T23:47:27.595 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_resend": 0, 2024-04-05T23:47:27.595 INFO:teuthology.orchestra.run.smithi027.stdout: "command_active": 0, 2024-04-05T23:47:27.595 INFO:teuthology.orchestra.run.smithi027.stdout: "command_send": 0, 2024-04-05T23:47:27.596 INFO:teuthology.orchestra.run.smithi027.stdout: "command_resend": 0, 2024-04-05T23:47:27.596 INFO:teuthology.orchestra.run.smithi027.stdout: "map_epoch": 101, 2024-04-05T23:47:27.596 INFO:teuthology.orchestra.run.smithi027.stdout: "map_full": 0, 2024-04-05T23:47:27.596 INFO:teuthology.orchestra.run.smithi027.stdout: "map_inc": 22, 2024-04-05T23:47:27.596 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_sessions": 9, 2024-04-05T23:47:27.596 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_open": 10, 2024-04-05T23:47:27.596 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_close": 1, 2024-04-05T23:47:27.596 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_laggy": 0, 2024-04-05T23:47:27.597 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_wr": 0, 2024-04-05T23:47:27.597 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_rd": 2, 2024-04-05T23:47:27.597 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_del": 0 2024-04-05T23:47:27.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.597 INFO:teuthology.orchestra.run.smithi027.stdout: "oft": { 2024-04-05T23:47:27.597 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_objs": 0, 2024-04-05T23:47:27.598 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:27.598 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_updates": 0, 2024-04-05T23:47:27.598 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_removes": 0 2024-04-05T23:47:27.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.598 INFO:teuthology.orchestra.run.smithi027.stdout: "osd-slow-ops": { 2024-04-05T23:47:27.598 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_ops_count": 0 2024-04-05T23:47:27.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.598 INFO:teuthology.orchestra.run.smithi027.stdout: "purge_queue": { 2024-04-05T23:47:27.599 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:27.599 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops_high_water": 0, 2024-04-05T23:47:27.599 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing": 0, 2024-04-05T23:47:27.599 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_high_water": 0, 2024-04-05T23:47:27.599 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed": 0, 2024-04-05T23:47:27.599 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:27.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.599 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:27.600 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:27.600 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-05T23:47:27.600 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:27.600 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 3462, 2024-04-05T23:47:27.600 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 59412932, 2024-04-05T23:47:27.600 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:27.600 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 3462, 2024-04-05T23:47:27.600 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:27.601 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:27.601 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 3462, 2024-04-05T23:47:27.601 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 59412932, 2024-04-05T23:47:27.601 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:27.601 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.601 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.601 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:27.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.602 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:27.602 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:27.602 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-05T23:47:27.602 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:27.602 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:27.602 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:27.603 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:27.603 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:27.603 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 3109, 2024-04-05T23:47:27.603 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 56918209, 2024-04-05T23:47:27.603 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 3109, 2024-04-05T23:47:27.603 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 56918209, 2024-04-05T23:47:27.603 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:27.603 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.603 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.604 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:27.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.604 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:27.604 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:27.604 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1024, 2024-04-05T23:47:27.604 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:27.604 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:27.605 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:27.605 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:27.605 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:27.605 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 3109, 2024-04-05T23:47:27.605 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 3109, 2024-04-05T23:47:27.605 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 3109, 2024-04-05T23:47:27.605 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 3109, 2024-04-05T23:47:27.605 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:27.606 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.606 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.606 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.606 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:27.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.606 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:27.606 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:27.606 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-05T23:47:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-05T23:47:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-05T23:47:27.608 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:27.608 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.608 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.608 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.608 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:27.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:27.609 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle-0x559de2f9e5a0": { 2024-04-05T23:47:27.609 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:27.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-05T23:47:27.609 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:27.609 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:27.609 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:27.609 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:27.609 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:27.610 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:27.610 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:27.610 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-05T23:47:27.610 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-05T23:47:27.610 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:27.610 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:27.610 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:27.610 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:27.611 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:27.611 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:27.611 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-05T23:47:27.899 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-05T23:47:27.900 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-05T23:47:27.900 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-05T23:47:27.900 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-05T23:47:27.900 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-05T23:47:27.900 DEBUG:tasks.check_counter:Getting stats from h 2024-04-05T23:47:27.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.h perf dump 2024-04-05T23:47:27.967 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:27 smithi027 ceph-mon[78597]: pgmap v438: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 757 KiB/s wr, 145 op/s 2024-04-05T23:47:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:47:28.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:27 smithi016 ceph-mon[38248]: pgmap v438: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 757 KiB/s wr, 145 op/s 2024-04-05T23:47:28.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:27 smithi130 ceph-mon[79417]: pgmap v438: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 757 KiB/s wr, 145 op/s 2024-04-05T23:47:29.079 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-05T23:47:29.079 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:29.079 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 2109, 2024-04-05T23:47:29.079 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 2111, 2024-04-05T23:47:29.080 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 25789836, 2024-04-05T23:47:29.080 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 655906, 2024-04-05T23:47:29.080 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 7, 2024-04-05T23:47:29.080 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-05T23:47:29.080 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.176562617, 2024-04-05T23:47:29.080 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.050596470, 2024-04-05T23:47:29.080 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.057630594, 2024-04-05T23:47:29.080 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.051550966, 2024-04-05T23:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 2109, 2024-04-05T23:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.068394987, 2024-04-05T23:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000032430 2024-04-05T23:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.082 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-05T23:47:29.082 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-05T23:47:29.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.082 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:29.082 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 713, 2024-04-05T23:47:29.082 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 901, 2024-04-05T23:47:29.082 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 21155215, 2024-04-05T23:47:29.082 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 709683, 2024-04-05T23:47:29.083 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 8, 2024-04-05T23:47:29.083 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 6, 2024-04-05T23:47:29.083 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.070525226, 2024-04-05T23:47:29.083 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.021854391, 2024-04-05T23:47:29.083 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.025449398, 2024-04-05T23:47:29.083 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.016700341, 2024-04-05T23:47:29.083 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:29.083 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 899, 2024-04-05T23:47:29.084 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.047666119, 2024-04-05T23:47:29.084 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000053021 2024-04-05T23:47:29.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.084 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:29.084 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.084 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.084 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.084 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-05T23:47:29.085 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 489744 2024-04-05T23:47:29.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.085 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:29.085 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 734, 2024-04-05T23:47:29.085 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 662, 2024-04-05T23:47:29.085 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 11716458, 2024-04-05T23:47:29.085 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 267456, 2024-04-05T23:47:29.085 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 6, 2024-04-05T23:47:29.085 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 4, 2024-04-05T23:47:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.084216620, 2024-04-05T23:47:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.023172528, 2024-04-05T23:47:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.053877206, 2024-04-05T23:47:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.015002155, 2024-04-05T23:47:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 661, 2024-04-05T23:47:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.026065130, 2024-04-05T23:47:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000039432 2024-04-05T23:47:29.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.087 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:29.087 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.087 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.087 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.087 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1959008, 2024-04-05T23:47:29.087 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 148032 2024-04-05T23:47:29.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.088 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-05T23:47:29.088 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 1, 2024-04-05T23:47:29.088 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-05T23:47:29.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.088 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-MDSRank": { 2024-04-05T23:47:29.088 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-05T23:47:29.088 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-05T23:47:29.088 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 5516, 2024-04-05T23:47:29.088 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.115939588, 2024-04-05T23:47:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000021018 2024-04-05T23:47:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-05T23:47:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-05T23:47:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.090 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:29.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.090 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-05T23:47:29.090 INFO:teuthology.orchestra.run.smithi027.stdout: "request": 0, 2024-04-05T23:47:29.090 INFO:teuthology.orchestra.run.smithi027.stdout: "reply": 0, 2024-04-05T23:47:29.090 INFO:teuthology.orchestra.run.smithi027.stdout: "reply_latency": { 2024-04-05T23:47:29.090 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.090 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_reply": 0, 2024-04-05T23:47:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: "forward": 0, 2024-04-05T23:47:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_complete": 0, 2024-04-05T23:47:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_keys": 0, 2024-04-05T23:47:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_commit": 0, 2024-04-05T23:47:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_split": 0, 2024-04-05T23:47:29.091 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_merge": 0, 2024-04-05T23:47:29.092 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes": 4873, 2024-04-05T23:47:29.092 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_top": 25, 2024-04-05T23:47:29.092 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_bottom": 4848, 2024-04-05T23:47:29.092 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pin_tail": 0, 2024-04-05T23:47:29.092 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pinned": 4838, 2024-04-05T23:47:29.092 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_expired": 0, 2024-04-05T23:47:29.092 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_with_caps": 0, 2024-04-05T23:47:29.092 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": 0, 2024-04-05T23:47:29.092 INFO:teuthology.orchestra.run.smithi027.stdout: "subtrees": 7, 2024-04-05T23:47:29.093 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse": 0, 2024-04-05T23:47:29.093 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_hit": 0, 2024-04-05T23:47:29.093 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_forward": 0, 2024-04-05T23:47:29.093 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_discover": 0, 2024-04-05T23:47:29.093 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_dir_fetch": 0, 2024-04-05T23:47:29.093 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:29.093 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_lock": 0, 2024-04-05T23:47:29.093 INFO:teuthology.orchestra.run.smithi027.stdout: "load_cent": 0, 2024-04-05T23:47:29.094 INFO:teuthology.orchestra.run.smithi027.stdout: "q": 0, 2024-04-05T23:47:29.094 INFO:teuthology.orchestra.run.smithi027.stdout: "exported": 0, 2024-04-05T23:47:29.094 INFO:teuthology.orchestra.run.smithi027.stdout: "exported_inodes": 0, 2024-04-05T23:47:29.094 INFO:teuthology.orchestra.run.smithi027.stdout: "imported": 0, 2024-04-05T23:47:29.094 INFO:teuthology.orchestra.run.smithi027.stdout: "imported_inodes": 0, 2024-04-05T23:47:29.094 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:29.094 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_backtrace_fetch": 0, 2024-04-05T23:47:29.094 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:29.095 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rfiles": 0, 2024-04-05T23:47:29.095 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rbytes": 0, 2024-04-05T23:47:29.095 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rsnaps": 0, 2024-04-05T23:47:29.095 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_fetch": 0, 2024-04-05T23:47:29.095 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:29.095 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:29.095 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:29.095 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_inodes": 0, 2024-04-05T23:47:29.095 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_base_inodes": 0, 2024-04-05T23:47:29.096 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dirfrag_rstats": 0, 2024-04-05T23:47:29.096 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_file_inodes": 0, 2024-04-05T23:47:29.096 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_inode_file_caps": 0, 2024-04-05T23:47:29.096 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_revoke": 0, 2024-04-05T23:47:29.096 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_grant": 0, 2024-04-05T23:47:29.096 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:29.096 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:29.096 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:29.096 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps": 0, 2024-04-05T23:47:29.097 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps_dirty": 0, 2024-04-05T23:47:29.097 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_cap_release": 0, 2024-04-05T23:47:29.097 INFO:teuthology.orchestra.run.smithi027.stdout: "process_request_cap_release": 0 2024-04-05T23:47:29.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.097 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache": { 2024-04-05T23:47:29.097 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update": 0, 2024-04-05T23:47:29.097 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update_receipt": 0, 2024-04-05T23:47:29.097 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_try_discover": 0, 2024-04-05T23:47:29.098 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_send_discover": 0, 2024-04-05T23:47:29.098 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_handle_discover": 0, 2024-04-05T23:47:29.098 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays": 0, 2024-04-05T23:47:29.098 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:29.098 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:29.098 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_created": 0, 2024-04-05T23:47:29.098 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_enqueued": 0, 2024-04-05T23:47:29.098 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:29.098 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_migrated": 0, 2024-04-05T23:47:29.099 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:29.099 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:29.099 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:29.099 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_started": 0, 2024-04-05T23:47:29.099 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_completed": 0, 2024-04-05T23:47:29.099 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_enqueue_scrub": 0, 2024-04-05T23:47:29.099 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_exportdir": 0, 2024-04-05T23:47:29.099 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_flush": 0, 2024-04-05T23:47:29.100 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragmentdir": 0, 2024-04-05T23:47:29.100 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:29.100 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_inodestats": 0 2024-04-05T23:47:29.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.100 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log": { 2024-04-05T23:47:29.100 INFO:teuthology.orchestra.run.smithi027.stdout: "evadd": 0, 2024-04-05T23:47:29.100 INFO:teuthology.orchestra.run.smithi027.stdout: "evex": 0, 2024-04-05T23:47:29.100 INFO:teuthology.orchestra.run.smithi027.stdout: "evtrm": 0, 2024-04-05T23:47:29.100 INFO:teuthology.orchestra.run.smithi027.stdout: "ev": 31204, 2024-04-05T23:47:29.101 INFO:teuthology.orchestra.run.smithi027.stdout: "evexg": 0, 2024-04-05T23:47:29.101 INFO:teuthology.orchestra.run.smithi027.stdout: "evexd": 0, 2024-04-05T23:47:29.101 INFO:teuthology.orchestra.run.smithi027.stdout: "segadd": 0, 2024-04-05T23:47:29.101 INFO:teuthology.orchestra.run.smithi027.stdout: "segex": 0, 2024-04-05T23:47:29.101 INFO:teuthology.orchestra.run.smithi027.stdout: "segtrm": 0, 2024-04-05T23:47:29.101 INFO:teuthology.orchestra.run.smithi027.stdout: "seg": 40, 2024-04-05T23:47:29.101 INFO:teuthology.orchestra.run.smithi027.stdout: "segexg": 0, 2024-04-05T23:47:29.101 INFO:teuthology.orchestra.run.smithi027.stdout: "segexd": 0, 2024-04-05T23:47:29.101 INFO:teuthology.orchestra.run.smithi027.stdout: "expos": 4194304, 2024-04-05T23:47:29.102 INFO:teuthology.orchestra.run.smithi027.stdout: "wrpos": 59990648, 2024-04-05T23:47:29.102 INFO:teuthology.orchestra.run.smithi027.stdout: "rdpos": 59990587, 2024-04-05T23:47:29.102 INFO:teuthology.orchestra.run.smithi027.stdout: "jlat": { 2024-04-05T23:47:29.102 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.102 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.102 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.103 INFO:teuthology.orchestra.run.smithi027.stdout: "replayed": 31204 2024-04-05T23:47:29.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.103 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mem": { 2024-04-05T23:47:29.103 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 324, 2024-04-05T23:47:29.103 INFO:teuthology.orchestra.run.smithi027.stdout: "ino+": 4989, 2024-04-05T23:47:29.103 INFO:teuthology.orchestra.run.smithi027.stdout: "ino-": 4665, 2024-04-05T23:47:29.103 INFO:teuthology.orchestra.run.smithi027.stdout: "dir": 44, 2024-04-05T23:47:29.103 INFO:teuthology.orchestra.run.smithi027.stdout: "dir+": 59, 2024-04-05T23:47:29.103 INFO:teuthology.orchestra.run.smithi027.stdout: "dir-": 15, 2024-04-05T23:47:29.104 INFO:teuthology.orchestra.run.smithi027.stdout: "dn": 4873, 2024-04-05T23:47:29.104 INFO:teuthology.orchestra.run.smithi027.stdout: "dn+": 5196, 2024-04-05T23:47:29.104 INFO:teuthology.orchestra.run.smithi027.stdout: "dn-": 323, 2024-04-05T23:47:29.104 INFO:teuthology.orchestra.run.smithi027.stdout: "cap": 0, 2024-04-05T23:47:29.104 INFO:teuthology.orchestra.run.smithi027.stdout: "cap+": 0, 2024-04-05T23:47:29.104 INFO:teuthology.orchestra.run.smithi027.stdout: "cap-": 0, 2024-04-05T23:47:29.104 INFO:teuthology.orchestra.run.smithi027.stdout: "rss": 55248, 2024-04-05T23:47:29.104 INFO:teuthology.orchestra.run.smithi027.stdout: "heap": 207156 2024-04-05T23:47:29.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.105 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server": { 2024-04-05T23:47:29.105 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_client_request": 0, 2024-04-05T23:47:29.105 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_server_request": 0, 2024-04-05T23:47:29.105 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_request": 0, 2024-04-05T23:47:29.105 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_session": 0, 2024-04-05T23:47:29.105 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_peer_request": 0, 2024-04-05T23:47:29.105 INFO:teuthology.orchestra.run.smithi027.stdout: "req_create_latency": { 2024-04-05T23:47:29.105 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.106 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.106 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.106 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getattr_latency": { 2024-04-05T23:47:29.106 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.106 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.106 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.106 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:29.107 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.107 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.107 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.107 INFO:teuthology.orchestra.run.smithi027.stdout: "req_link_latency": { 2024-04-05T23:47:29.107 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.107 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.107 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.108 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookup_latency": { 2024-04-05T23:47:29.108 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.108 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.108 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.108 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:29.108 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.108 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.109 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.109 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupino_latency": { 2024-04-05T23:47:29.109 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.109 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.109 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.109 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupname_latency": { 2024-04-05T23:47:29.109 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.110 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.110 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.110 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:29.110 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.110 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.110 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.111 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:29.111 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.111 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.111 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.111 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lssnap_latency": { 2024-04-05T23:47:29.112 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.112 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.112 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.112 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mkdir_latency": { 2024-04-05T23:47:29.112 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.112 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.112 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mknod_latency": { 2024-04-05T23:47:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mksnap_latency": { 2024-04-05T23:47:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.114 INFO:teuthology.orchestra.run.smithi027.stdout: "req_open_latency": { 2024-04-05T23:47:29.114 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.114 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.114 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.114 INFO:teuthology.orchestra.run.smithi027.stdout: "req_readdir_latency": { 2024-04-05T23:47:29.114 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.115 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.115 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.115 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rename_latency": { 2024-04-05T23:47:29.115 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.115 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.115 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.115 INFO:teuthology.orchestra.run.smithi027.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:29.116 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.116 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.116 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.116 INFO:teuthology.orchestra.run.smithi027.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:29.116 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.116 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.116 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.117 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmdir_latency": { 2024-04-05T23:47:29.117 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.117 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.117 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.117 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:29.117 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.117 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.117 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setattr_latency": { 2024-04-05T23:47:29.118 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.119 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.119 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.119 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:29.119 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.119 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.119 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.120 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:29.120 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.120 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.120 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.120 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setlayout_latency": { 2024-04-05T23:47:29.120 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.120 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.120 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.121 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setxattr_latency": { 2024-04-05T23:47:29.121 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.121 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.121 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.121 INFO:teuthology.orchestra.run.smithi027.stdout: "req_symlink_latency": { 2024-04-05T23:47:29.121 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.121 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.122 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.122 INFO:teuthology.orchestra.run.smithi027.stdout: "req_unlink_latency": { 2024-04-05T23:47:29.122 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.122 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.122 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.122 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:29.123 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:29.123 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:29.123 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.123 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.123 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.123 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:29.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.123 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessions": { 2024-04-05T23:47:29.123 INFO:teuthology.orchestra.run.smithi027.stdout: "session_count": 1, 2024-04-05T23:47:29.124 INFO:teuthology.orchestra.run.smithi027.stdout: "session_add": 2, 2024-04-05T23:47:29.124 INFO:teuthology.orchestra.run.smithi027.stdout: "session_remove": 1, 2024-04-05T23:47:29.124 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_open": 2, 2024-04-05T23:47:29.124 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_stale": 0, 2024-04-05T23:47:29.124 INFO:teuthology.orchestra.run.smithi027.stdout: "total_load": 0, 2024-04-05T23:47:29.124 INFO:teuthology.orchestra.run.smithi027.stdout: "average_load": 0, 2024-04-05T23:47:29.124 INFO:teuthology.orchestra.run.smithi027.stdout: "avg_session_uptime": 4574, 2024-04-05T23:47:29.124 INFO:teuthology.orchestra.run.smithi027.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:29.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.125 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-05T23:47:29.125 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:29.125 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:29.125 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:29.125 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:29.125 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:29.125 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:29.125 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:29.125 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:29.126 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:29.126 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:29.126 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:29.126 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:29.126 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:29.126 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:29.126 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:29.126 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:29.127 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:29.127 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:29.127 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:29.127 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:29.127 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:29.127 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:29.127 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:29.127 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:29.127 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:29.128 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:29.128 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:29.128 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:29.128 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:29.128 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:29.128 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:29.128 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-05T23:47:29.128 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:29.128 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:29.129 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:29.129 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:29.129 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 236755, 2024-04-05T23:47:29.129 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 425, 2024-04-05T23:47:29.129 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-05T23:47:29.129 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-05T23:47:29.129 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-05T23:47:29.129 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-05T23:47:29.129 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:29.130 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:29.130 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:29.130 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:29.130 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:29.130 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 188, 2024-04-05T23:47:29.130 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:29.130 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:29.130 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:29.131 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-05T23:47:29.131 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 4328316, 2024-04-05T23:47:29.131 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 30719, 2024-04-05T23:47:29.131 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:29.131 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-05T23:47:29.131 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:29.131 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-05T23:47:29.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.132 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter": { 2024-04-05T23:47:29.132 INFO:teuthology.orchestra.run.smithi027.stdout: "op_active": 0, 2024-04-05T23:47:29.132 INFO:teuthology.orchestra.run.smithi027.stdout: "op_laggy": 0, 2024-04-05T23:47:29.132 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send": 3200, 2024-04-05T23:47:29.132 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send_bytes": 21, 2024-04-05T23:47:29.132 INFO:teuthology.orchestra.run.smithi027.stdout: "op_resend": 3, 2024-04-05T23:47:29.132 INFO:teuthology.orchestra.run.smithi027.stdout: "op_reply": 3200, 2024-04-05T23:47:29.132 INFO:teuthology.orchestra.run.smithi027.stdout: "op_latency": { 2024-04-05T23:47:29.132 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3200, 2024-04-05T23:47:29.133 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 1.707956036, 2024-04-05T23:47:29.133 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000533736 2024-04-05T23:47:29.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.133 INFO:teuthology.orchestra.run.smithi027.stdout: "op_inflight": 0, 2024-04-05T23:47:29.133 INFO:teuthology.orchestra.run.smithi027.stdout: "oplen_avg": { 2024-04-05T23:47:29.133 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3200, 2024-04-05T23:47:29.133 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 3201 2024-04-05T23:47:29.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.134 INFO:teuthology.orchestra.run.smithi027.stdout: "op": 3200, 2024-04-05T23:47:29.134 INFO:teuthology.orchestra.run.smithi027.stdout: "op_r": 3200, 2024-04-05T23:47:29.134 INFO:teuthology.orchestra.run.smithi027.stdout: "op_w": 0, 2024-04-05T23:47:29.134 INFO:teuthology.orchestra.run.smithi027.stdout: "op_rmw": 0, 2024-04-05T23:47:29.134 INFO:teuthology.orchestra.run.smithi027.stdout: "op_pg": 0, 2024-04-05T23:47:29.134 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_stat": 1746, 2024-04-05T23:47:29.134 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_create": 0, 2024-04-05T23:47:29.134 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_read": 1452, 2024-04-05T23:47:29.134 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_write": 0, 2024-04-05T23:47:29.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writefull": 0, 2024-04-05T23:47:29.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writesame": 0, 2024-04-05T23:47:29.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_append": 0, 2024-04-05T23:47:29.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_zero": 0, 2024-04-05T23:47:29.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_truncate": 0, 2024-04-05T23:47:29.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_delete": 0, 2024-04-05T23:47:29.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_mapext": 0, 2024-04-05T23:47:29.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:29.135 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:29.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_getxattr": 1, 2024-04-05T23:47:29.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_setxattr": 0, 2024-04-05T23:47:29.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:29.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:29.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:29.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_call": 0, 2024-04-05T23:47:29.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_watch": 0, 2024-04-05T23:47:29.136 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_notify": 0, 2024-04-05T23:47:29.137 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:29.137 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls": 0, 2024-04-05T23:47:29.137 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:29.137 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_other": 0, 2024-04-05T23:47:29.137 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_active": 0, 2024-04-05T23:47:29.137 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_send": 0, 2024-04-05T23:47:29.137 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_resend": 0, 2024-04-05T23:47:29.137 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_ping": 0, 2024-04-05T23:47:29.137 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_active": 0, 2024-04-05T23:47:29.138 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_send": 0, 2024-04-05T23:47:29.138 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_resend": 0, 2024-04-05T23:47:29.138 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_active": 0, 2024-04-05T23:47:29.138 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_send": 0, 2024-04-05T23:47:29.138 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_resend": 0, 2024-04-05T23:47:29.138 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_active": 0, 2024-04-05T23:47:29.138 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_send": 0, 2024-04-05T23:47:29.138 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_resend": 0, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "command_active": 0, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "command_send": 0, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "command_resend": 0, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "map_epoch": 101, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "map_full": 0, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "map_inc": 22, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_sessions": 9, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_open": 10, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_close": 1, 2024-04-05T23:47:29.139 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_laggy": 0, 2024-04-05T23:47:29.140 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_wr": 0, 2024-04-05T23:47:29.140 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_rd": 2, 2024-04-05T23:47:29.140 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_del": 0 2024-04-05T23:47:29.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.140 INFO:teuthology.orchestra.run.smithi027.stdout: "oft": { 2024-04-05T23:47:29.140 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_objs": 0, 2024-04-05T23:47:29.140 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:29.140 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_updates": 0, 2024-04-05T23:47:29.141 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_removes": 0 2024-04-05T23:47:29.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.141 INFO:teuthology.orchestra.run.smithi027.stdout: "osd-slow-ops": { 2024-04-05T23:47:29.141 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_ops_count": 0 2024-04-05T23:47:29.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.141 INFO:teuthology.orchestra.run.smithi027.stdout: "purge_queue": { 2024-04-05T23:47:29.141 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:29.141 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops_high_water": 0, 2024-04-05T23:47:29.141 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing": 0, 2024-04-05T23:47:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_high_water": 0, 2024-04-05T23:47:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed": 0, 2024-04-05T23:47:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-05T23:47:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 3552, 2024-04-05T23:47:29.143 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 58321254, 2024-04-05T23:47:29.143 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:29.143 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 3552, 2024-04-05T23:47:29.143 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:29.143 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:29.143 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 3552, 2024-04-05T23:47:29.143 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 58321254, 2024-04-05T23:47:29.143 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:29.143 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.144 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.144 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.144 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:29.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.144 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:29.144 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:29.144 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-05T23:47:29.144 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:29.144 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:29.145 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:29.145 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:29.145 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:29.145 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 3200, 2024-04-05T23:47:29.145 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 55812637, 2024-04-05T23:47:29.145 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 3200, 2024-04-05T23:47:29.145 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 55812637, 2024-04-05T23:47:29.145 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:29.146 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.146 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.146 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.146 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:29.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.146 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:29.146 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:29.146 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1024, 2024-04-05T23:47:29.146 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:29.147 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:29.147 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:29.147 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:29.147 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:29.147 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 3200, 2024-04-05T23:47:29.147 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 3200, 2024-04-05T23:47:29.147 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 3200, 2024-04-05T23:47:29.147 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 3200, 2024-04-05T23:47:29.147 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:29.148 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.148 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.148 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.148 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:29.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.148 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:29.148 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:29.148 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-05T23:47:29.148 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:29.149 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:29.149 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:29.149 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:29.149 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:29.149 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:29.149 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:29.150 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-05T23:47:29.150 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-05T23:47:29.150 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:29.150 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.150 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.150 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.150 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:29.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:29.151 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle-0x55bd5f1b8500": { 2024-04-05T23:47:29.151 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:29.151 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-05T23:47:29.151 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:29.151 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:29.151 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:29.151 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-05T23:47:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-05T23:47:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:29.152 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:29.153 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:29.153 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:29.153 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-05T23:47:29.562 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-05T23:47:29.563 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-05T23:47:29.563 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-05T23:47:29.563 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-05T23:47:29.563 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-05T23:47:29.563 DEBUG:tasks.check_counter:Getting stats from k 2024-04-05T23:47:29.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.k perf dump 2024-04-05T23:47:29.792 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-05T23:47:30.142 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:29 smithi027 ceph-mon[78597]: pgmap v439: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 163 KiB/s wr, 17 op/s 2024-04-05T23:47:30.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:29 smithi016 ceph-mon[38248]: pgmap v439: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 163 KiB/s wr, 17 op/s 2024-04-05T23:47:30.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:29 smithi130 ceph-mon[79417]: pgmap v439: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 163 KiB/s wr, 17 op/s 2024-04-05T23:47:31.265 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-05T23:47:31.266 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:31.266 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 1514, 2024-04-05T23:47:31.266 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 1516, 2024-04-05T23:47:31.266 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 14635567, 2024-04-05T23:47:31.266 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 470855, 2024-04-05T23:47:31.266 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 7, 2024-04-05T23:47:31.266 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 3, 2024-04-05T23:47:31.267 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.133166483, 2024-04-05T23:47:31.267 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.041780201, 2024-04-05T23:47:31.267 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.041032250, 2024-04-05T23:47:31.267 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.036558072, 2024-04-05T23:47:31.267 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:31.267 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1514, 2024-04-05T23:47:31.267 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.053196823, 2024-04-05T23:47:31.267 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000035136 2024-04-05T23:47:31.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.268 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:31.268 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.268 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.268 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.268 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-05T23:47:31.268 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 0 2024-04-05T23:47:31.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.269 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:31.269 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 1261, 2024-04-05T23:47:31.269 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 1454, 2024-04-05T23:47:31.269 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 38078238, 2024-04-05T23:47:31.269 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 883009, 2024-04-05T23:47:31.269 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 8, 2024-04-05T23:47:31.269 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 6, 2024-04-05T23:47:31.269 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.122376671, 2024-04-05T23:47:31.269 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.027196369, 2024-04-05T23:47:31.270 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.052507998, 2024-04-05T23:47:31.270 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.032403324, 2024-04-05T23:47:31.270 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:31.270 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 1451, 2024-04-05T23:47:31.270 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.064670324, 2024-04-05T23:47:31.270 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000044569 2024-04-05T23:47:31.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.270 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:31.271 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.271 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.271 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.271 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-05T23:47:31.272 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 491392 2024-04-05T23:47:31.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.272 INFO:teuthology.orchestra.run.smithi027.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:31.272 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_messages": 761, 2024-04-05T23:47:31.272 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages": 692, 2024-04-05T23:47:31.272 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_bytes": 14641429, 2024-04-05T23:47:31.272 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_bytes": 277736, 2024-04-05T23:47:31.272 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_created_connections": 5, 2024-04-05T23:47:31.273 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_active_connections": 4, 2024-04-05T23:47:31.273 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_total_time": 0.086538830, 2024-04-05T23:47:31.273 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_send_time": 0.022179323, 2024-04-05T23:47:31.273 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_recv_time": 0.056300658, 2024-04-05T23:47:31.273 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_running_fast_dispatch_time": 0.015817474, 2024-04-05T23:47:31.273 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:31.273 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 692, 2024-04-05T23:47:31.273 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.024414546, 2024-04-05T23:47:31.274 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000035281 2024-04-05T23:47:31.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.274 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:31.274 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.274 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.274 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.275 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_recv_encrypted_bytes": 1940336, 2024-04-05T23:47:31.275 INFO:teuthology.orchestra.run.smithi027.stdout: "msgr_send_encrypted_bytes": 150848 2024-04-05T23:47:31.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.275 INFO:teuthology.orchestra.run.smithi027.stdout: "cct": { 2024-04-05T23:47:31.275 INFO:teuthology.orchestra.run.smithi027.stdout: "total_workers": 1, 2024-04-05T23:47:31.275 INFO:teuthology.orchestra.run.smithi027.stdout: "unhealthy_workers": 0 2024-04-05T23:47:31.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.275 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-MDSRank": { 2024-04-05T23:47:31.276 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-05T23:47:31.276 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-05T23:47:31.276 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 5417, 2024-04-05T23:47:31.276 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.115626704, 2024-04-05T23:47:31.276 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000021345 2024-04-05T23:47:31.276 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:31.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.277 INFO:teuthology.orchestra.run.smithi027.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:31.277 INFO:teuthology.orchestra.run.smithi027.stdout: "queue_len": 0, 2024-04-05T23:47:31.277 INFO:teuthology.orchestra.run.smithi027.stdout: "complete_latency": { 2024-04-05T23:47:31.277 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.277 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.277 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.278 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:31.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.278 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-05T23:47:31.278 INFO:teuthology.orchestra.run.smithi027.stdout: "request": 0, 2024-04-05T23:47:31.278 INFO:teuthology.orchestra.run.smithi027.stdout: "reply": 0, 2024-04-05T23:47:31.278 INFO:teuthology.orchestra.run.smithi027.stdout: "reply_latency": { 2024-04-05T23:47:31.279 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.279 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.279 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.279 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_reply": 0, 2024-04-05T23:47:31.279 INFO:teuthology.orchestra.run.smithi027.stdout: "forward": 0, 2024-04-05T23:47:31.280 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_complete": 0, 2024-04-05T23:47:31.280 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_fetch_keys": 0, 2024-04-05T23:47:31.280 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_commit": 0, 2024-04-05T23:47:31.280 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_split": 0, 2024-04-05T23:47:31.280 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_merge": 0, 2024-04-05T23:47:31.280 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes": 5267, 2024-04-05T23:47:31.281 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_top": 26, 2024-04-05T23:47:31.281 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_bottom": 5241, 2024-04-05T23:47:31.281 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pin_tail": 0, 2024-04-05T23:47:31.281 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_pinned": 5230, 2024-04-05T23:47:31.281 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_expired": 0, 2024-04-05T23:47:31.281 INFO:teuthology.orchestra.run.smithi027.stdout: "inodes_with_caps": 0, 2024-04-05T23:47:31.282 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": 0, 2024-04-05T23:47:31.282 INFO:teuthology.orchestra.run.smithi027.stdout: "subtrees": 7, 2024-04-05T23:47:31.282 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse": 0, 2024-04-05T23:47:31.282 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_hit": 0, 2024-04-05T23:47:31.282 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_forward": 0, 2024-04-05T23:47:31.282 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_discover": 0, 2024-04-05T23:47:31.283 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_dir_fetch": 0, 2024-04-05T23:47:31.283 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:31.283 INFO:teuthology.orchestra.run.smithi027.stdout: "traverse_lock": 0, 2024-04-05T23:47:31.283 INFO:teuthology.orchestra.run.smithi027.stdout: "load_cent": 0, 2024-04-05T23:47:31.283 INFO:teuthology.orchestra.run.smithi027.stdout: "q": 0, 2024-04-05T23:47:31.284 INFO:teuthology.orchestra.run.smithi027.stdout: "exported": 0, 2024-04-05T23:47:31.284 INFO:teuthology.orchestra.run.smithi027.stdout: "exported_inodes": 0, 2024-04-05T23:47:31.284 INFO:teuthology.orchestra.run.smithi027.stdout: "imported": 0, 2024-04-05T23:47:31.284 INFO:teuthology.orchestra.run.smithi027.stdout: "imported_inodes": 0, 2024-04-05T23:47:31.284 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:31.284 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_backtrace_fetch": 0, 2024-04-05T23:47:31.285 INFO:teuthology.orchestra.run.smithi027.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:31.285 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rfiles": 0, 2024-04-05T23:47:31.285 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rbytes": 0, 2024-04-05T23:47:31.285 INFO:teuthology.orchestra.run.smithi027.stdout: "root_rsnaps": 0, 2024-04-05T23:47:31.285 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_fetch": 0, 2024-04-05T23:47:31.285 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:31.285 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:31.285 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:31.285 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_inodes": 0, 2024-04-05T23:47:31.286 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dir_base_inodes": 0, 2024-04-05T23:47:31.286 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_dirfrag_rstats": 0, 2024-04-05T23:47:31.286 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_file_inodes": 0, 2024-04-05T23:47:31.286 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_inode_file_caps": 0, 2024-04-05T23:47:31.286 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_revoke": 0, 2024-04-05T23:47:31.286 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_grant": 0, 2024-04-05T23:47:31.286 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:31.286 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:31.287 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:31.287 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps": 0, 2024-04-05T23:47:31.287 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_caps_dirty": 0, 2024-04-05T23:47:31.287 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_cap_release": 0, 2024-04-05T23:47:31.287 INFO:teuthology.orchestra.run.smithi027.stdout: "process_request_cap_release": 0 2024-04-05T23:47:31.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.287 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache": { 2024-04-05T23:47:31.287 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update": 0, 2024-04-05T23:47:31.287 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_update_receipt": 0, 2024-04-05T23:47:31.288 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_try_discover": 0, 2024-04-05T23:47:31.288 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_send_discover": 0, 2024-04-05T23:47:31.288 INFO:teuthology.orchestra.run.smithi027.stdout: "dir_handle_discover": 0, 2024-04-05T23:47:31.288 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays": 0, 2024-04-05T23:47:31.288 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:31.288 INFO:teuthology.orchestra.run.smithi027.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:31.288 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_created": 0, 2024-04-05T23:47:31.289 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_enqueued": 0, 2024-04-05T23:47:31.289 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:31.289 INFO:teuthology.orchestra.run.smithi027.stdout: "strays_migrated": 0, 2024-04-05T23:47:31.289 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:31.289 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:31.289 INFO:teuthology.orchestra.run.smithi027.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:31.290 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_started": 0, 2024-04-05T23:47:31.290 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_completed": 0, 2024-04-05T23:47:31.290 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_enqueue_scrub": 0, 2024-04-05T23:47:31.290 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_exportdir": 0, 2024-04-05T23:47:31.290 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_flush": 0, 2024-04-05T23:47:31.290 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragmentdir": 0, 2024-04-05T23:47:31.290 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:31.290 INFO:teuthology.orchestra.run.smithi027.stdout: "ireq_inodestats": 0 2024-04-05T23:47:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log": { 2024-04-05T23:47:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: "evadd": 0, 2024-04-05T23:47:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: "evex": 0, 2024-04-05T23:47:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: "evtrm": 0, 2024-04-05T23:47:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: "ev": 34105, 2024-04-05T23:47:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: "evexg": 0, 2024-04-05T23:47:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: "evexd": 0, 2024-04-05T23:47:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: "segadd": 0, 2024-04-05T23:47:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: "segex": 0, 2024-04-05T23:47:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: "segtrm": 0, 2024-04-05T23:47:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: "seg": 41, 2024-04-05T23:47:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: "segexg": 0, 2024-04-05T23:47:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: "segexd": 0, 2024-04-05T23:47:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: "expos": 4194304, 2024-04-05T23:47:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: "wrpos": 68699633, 2024-04-05T23:47:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: "rdpos": 68699572, 2024-04-05T23:47:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: "jlat": { 2024-04-05T23:47:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: "replayed": 34105 2024-04-05T23:47:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mem": { 2024-04-05T23:47:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 368, 2024-04-05T23:47:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: "ino+": 5282, 2024-04-05T23:47:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: "ino-": 4914, 2024-04-05T23:47:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: "dir": 48, 2024-04-05T23:47:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: "dir+": 63, 2024-04-05T23:47:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: "dir-": 15, 2024-04-05T23:47:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: "dn": 5267, 2024-04-05T23:47:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: "dn+": 5496, 2024-04-05T23:47:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: "dn-": 229, 2024-04-05T23:47:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: "cap": 0, 2024-04-05T23:47:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: "cap+": 0, 2024-04-05T23:47:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: "cap-": 0, 2024-04-05T23:47:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: "rss": 58524, 2024-04-05T23:47:31.296 INFO:teuthology.orchestra.run.smithi027.stdout: "heap": 207156 2024-04-05T23:47:31.296 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.296 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server": { 2024-04-05T23:47:31.296 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_client_request": 0, 2024-04-05T23:47:31.296 INFO:teuthology.orchestra.run.smithi027.stdout: "dispatch_server_request": 0, 2024-04-05T23:47:31.296 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_request": 0, 2024-04-05T23:47:31.296 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_client_session": 0, 2024-04-05T23:47:31.296 INFO:teuthology.orchestra.run.smithi027.stdout: "handle_peer_request": 0, 2024-04-05T23:47:31.296 INFO:teuthology.orchestra.run.smithi027.stdout: "req_create_latency": { 2024-04-05T23:47:31.297 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.297 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.297 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.297 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getattr_latency": { 2024-04-05T23:47:31.297 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.297 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.297 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.298 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:31.298 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.298 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.298 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.298 INFO:teuthology.orchestra.run.smithi027.stdout: "req_link_latency": { 2024-04-05T23:47:31.298 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.298 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.299 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.299 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookup_latency": { 2024-04-05T23:47:31.299 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.299 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.299 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.299 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:31.300 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.300 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.300 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.300 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupino_latency": { 2024-04-05T23:47:31.300 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.300 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.300 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.301 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupname_latency": { 2024-04-05T23:47:31.301 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.301 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.301 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.301 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:31.301 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.301 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.302 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.302 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:31.302 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.302 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.302 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.302 INFO:teuthology.orchestra.run.smithi027.stdout: "req_lssnap_latency": { 2024-04-05T23:47:31.303 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.303 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.303 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.303 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mkdir_latency": { 2024-04-05T23:47:31.303 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.303 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.303 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.304 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mknod_latency": { 2024-04-05T23:47:31.304 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.304 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.304 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.304 INFO:teuthology.orchestra.run.smithi027.stdout: "req_mksnap_latency": { 2024-04-05T23:47:31.304 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.304 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.305 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.305 INFO:teuthology.orchestra.run.smithi027.stdout: "req_open_latency": { 2024-04-05T23:47:31.305 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.305 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.305 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.305 INFO:teuthology.orchestra.run.smithi027.stdout: "req_readdir_latency": { 2024-04-05T23:47:31.306 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.306 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.306 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.306 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rename_latency": { 2024-04-05T23:47:31.306 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.306 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.306 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.307 INFO:teuthology.orchestra.run.smithi027.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:31.307 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.307 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.307 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.307 INFO:teuthology.orchestra.run.smithi027.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmdir_latency": { 2024-04-05T23:47:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setattr_latency": { 2024-04-05T23:47:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setlayout_latency": { 2024-04-05T23:47:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: "req_setxattr_latency": { 2024-04-05T23:47:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: "req_symlink_latency": { 2024-04-05T23:47:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: "req_unlink_latency": { 2024-04-05T23:47:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessions": { 2024-04-05T23:47:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: "session_count": 1, 2024-04-05T23:47:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: "session_add": 3, 2024-04-05T23:47:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: "session_remove": 2, 2024-04-05T23:47:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_open": 3, 2024-04-05T23:47:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: "sessions_stale": 0, 2024-04-05T23:47:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: "total_load": 0, 2024-04-05T23:47:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: "average_load": 0, 2024-04-05T23:47:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: "avg_session_uptime": 6449, 2024-04-05T23:47:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool": { 2024-04-05T23:47:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:31.321 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:31.321 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:31.321 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:31.321 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:31.321 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:31.321 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:31.321 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:31.321 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:31.322 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:31.322 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:31.322 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:31.322 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:31.322 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:31.322 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:31.322 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_items": 0, 2024-04-05T23:47:31.323 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:31.323 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:31.323 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:31.323 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:31.323 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_bytes": 237342, 2024-04-05T23:47:31.323 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_anon_items": 464, 2024-04-05T23:47:31.323 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_bytes": 0, 2024-04-05T23:47:31.324 INFO:teuthology.orchestra.run.smithi027.stdout: "buffer_meta_items": 0, 2024-04-05T23:47:31.324 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bytes": 0, 2024-04-05T23:47:31.324 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_items": 0, 2024-04-05T23:47:31.324 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:31.324 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:31.324 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:31.324 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:31.324 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:31.324 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_items": 188, 2024-04-05T23:47:31.325 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:31.325 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:31.325 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:31.325 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap_items": 0, 2024-04-05T23:47:31.325 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_bytes": 4710297, 2024-04-05T23:47:31.325 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_co_items": 31263, 2024-04-05T23:47:31.325 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:31.325 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_1_items": 0, 2024-04-05T23:47:31.326 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:31.326 INFO:teuthology.orchestra.run.smithi027.stdout: "unittest_2_items": 0 2024-04-05T23:47:31.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.326 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter": { 2024-04-05T23:47:31.326 INFO:teuthology.orchestra.run.smithi027.stdout: "op_active": 0, 2024-04-05T23:47:31.326 INFO:teuthology.orchestra.run.smithi027.stdout: "op_laggy": 0, 2024-04-05T23:47:31.326 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send": 3181, 2024-04-05T23:47:31.326 INFO:teuthology.orchestra.run.smithi027.stdout: "op_send_bytes": 21, 2024-04-05T23:47:31.327 INFO:teuthology.orchestra.run.smithi027.stdout: "op_resend": 3, 2024-04-05T23:47:31.327 INFO:teuthology.orchestra.run.smithi027.stdout: "op_reply": 3181, 2024-04-05T23:47:31.327 INFO:teuthology.orchestra.run.smithi027.stdout: "op_latency": { 2024-04-05T23:47:31.327 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3181, 2024-04-05T23:47:31.327 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 1.740955197, 2024-04-05T23:47:31.327 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000547298 2024-04-05T23:47:31.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.327 INFO:teuthology.orchestra.run.smithi027.stdout: "op_inflight": 0, 2024-04-05T23:47:31.327 INFO:teuthology.orchestra.run.smithi027.stdout: "oplen_avg": { 2024-04-05T23:47:31.328 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 3181, 2024-04-05T23:47:31.328 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 3182 2024-04-05T23:47:31.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.328 INFO:teuthology.orchestra.run.smithi027.stdout: "op": 3181, 2024-04-05T23:47:31.328 INFO:teuthology.orchestra.run.smithi027.stdout: "op_r": 3181, 2024-04-05T23:47:31.328 INFO:teuthology.orchestra.run.smithi027.stdout: "op_w": 0, 2024-04-05T23:47:31.328 INFO:teuthology.orchestra.run.smithi027.stdout: "op_rmw": 0, 2024-04-05T23:47:31.328 INFO:teuthology.orchestra.run.smithi027.stdout: "op_pg": 0, 2024-04-05T23:47:31.329 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_stat": 1748, 2024-04-05T23:47:31.329 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_create": 0, 2024-04-05T23:47:31.329 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_read": 1431, 2024-04-05T23:47:31.329 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_write": 0, 2024-04-05T23:47:31.329 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writefull": 0, 2024-04-05T23:47:31.329 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_writesame": 0, 2024-04-05T23:47:31.329 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_append": 0, 2024-04-05T23:47:31.329 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_zero": 0, 2024-04-05T23:47:31.329 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_truncate": 0, 2024-04-05T23:47:31.330 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_delete": 0, 2024-04-05T23:47:31.330 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_mapext": 0, 2024-04-05T23:47:31.330 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:31.330 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:31.330 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_getxattr": 1, 2024-04-05T23:47:31.330 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_setxattr": 0, 2024-04-05T23:47:31.330 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:31.330 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:31.331 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:31.331 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_call": 0, 2024-04-05T23:47:31.331 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_watch": 0, 2024-04-05T23:47:31.331 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_notify": 0, 2024-04-05T23:47:31.331 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:31.331 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls": 0, 2024-04-05T23:47:31.331 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:31.331 INFO:teuthology.orchestra.run.smithi027.stdout: "osdop_other": 0, 2024-04-05T23:47:31.331 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_active": 0, 2024-04-05T23:47:31.332 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_send": 0, 2024-04-05T23:47:31.332 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_resend": 0, 2024-04-05T23:47:31.332 INFO:teuthology.orchestra.run.smithi027.stdout: "linger_ping": 0, 2024-04-05T23:47:31.332 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_active": 0, 2024-04-05T23:47:31.332 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_send": 0, 2024-04-05T23:47:31.332 INFO:teuthology.orchestra.run.smithi027.stdout: "poolop_resend": 0, 2024-04-05T23:47:31.332 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_active": 0, 2024-04-05T23:47:31.332 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_send": 0, 2024-04-05T23:47:31.332 INFO:teuthology.orchestra.run.smithi027.stdout: "poolstat_resend": 0, 2024-04-05T23:47:31.333 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_active": 0, 2024-04-05T23:47:31.333 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_send": 0, 2024-04-05T23:47:31.333 INFO:teuthology.orchestra.run.smithi027.stdout: "statfs_resend": 0, 2024-04-05T23:47:31.333 INFO:teuthology.orchestra.run.smithi027.stdout: "command_active": 0, 2024-04-05T23:47:31.333 INFO:teuthology.orchestra.run.smithi027.stdout: "command_send": 0, 2024-04-05T23:47:31.333 INFO:teuthology.orchestra.run.smithi027.stdout: "command_resend": 0, 2024-04-05T23:47:31.333 INFO:teuthology.orchestra.run.smithi027.stdout: "map_epoch": 101, 2024-04-05T23:47:31.333 INFO:teuthology.orchestra.run.smithi027.stdout: "map_full": 0, 2024-04-05T23:47:31.333 INFO:teuthology.orchestra.run.smithi027.stdout: "map_inc": 22, 2024-04-05T23:47:31.334 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_sessions": 9, 2024-04-05T23:47:31.334 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_open": 9, 2024-04-05T23:47:31.334 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_session_close": 0, 2024-04-05T23:47:31.334 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_laggy": 0, 2024-04-05T23:47:31.334 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_wr": 0, 2024-04-05T23:47:31.334 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_rd": 2, 2024-04-05T23:47:31.334 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_del": 0 2024-04-05T23:47:31.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "oft": { 2024-04-05T23:47:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_objs": 0, 2024-04-05T23:47:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_updates": 0, 2024-04-05T23:47:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "omap_total_removes": 0 2024-04-05T23:47:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "osd-slow-ops": { 2024-04-05T23:47:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "slow_ops_count": 0 2024-04-05T23:47:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "purge_queue": { 2024-04-05T23:47:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_ops_high_water": 0, 2024-04-05T23:47:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing": 0, 2024-04-05T23:47:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executing_high_water": 0, 2024-04-05T23:47:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_executed": 0, 2024-04-05T23:47:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-05T23:47:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 3532, 2024-04-05T23:47:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 67016790, 2024-04-05T23:47:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 3532, 2024-04-05T23:47:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 3532, 2024-04-05T23:47:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 67016790, 2024-04-05T23:47:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.339 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:31.339 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:31.339 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 104857600, 2024-04-05T23:47:31.339 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:31.339 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:31.339 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:31.339 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:31.339 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:31.340 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 3181, 2024-04-05T23:47:31.340 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 64532556, 2024-04-05T23:47:31.340 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 3181, 2024-04-05T23:47:31.340 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 64532556, 2024-04-05T23:47:31.340 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:31.340 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.340 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.340 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.340 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:31.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.341 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:31.341 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:31.341 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1024, 2024-04-05T23:47:31.341 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:31.341 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:31.341 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:31.341 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:31.342 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:31.342 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 3181, 2024-04-05T23:47:31.342 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 3181, 2024-04-05T23:47:31.342 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 3181, 2024-04-05T23:47:31.342 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 3181, 2024-04-05T23:47:31.342 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:31.342 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.343 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.343 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.343 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:31.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.343 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:31.343 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:31.343 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-05T23:47:31.343 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:31.344 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:31.344 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:31.344 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:31.344 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:31.344 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:31.345 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:31.345 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-05T23:47:31.345 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-05T23:47:31.345 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:31.345 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.345 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.345 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.345 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:31.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-05T23:47:31.346 INFO:teuthology.orchestra.run.smithi027.stdout: "throttle-write_buf_throttle-0x55f0695805a0": { 2024-04-05T23:47:31.346 INFO:teuthology.orchestra.run.smithi027.stdout: "val": 0, 2024-04-05T23:47:31.346 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 3758096384, 2024-04-05T23:47:31.346 INFO:teuthology.orchestra.run.smithi027.stdout: "get_started": 0, 2024-04-05T23:47:31.346 INFO:teuthology.orchestra.run.smithi027.stdout: "get": 0, 2024-04-05T23:47:31.346 INFO:teuthology.orchestra.run.smithi027.stdout: "get_sum": 0, 2024-04-05T23:47:31.347 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:31.347 INFO:teuthology.orchestra.run.smithi027.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:31.347 INFO:teuthology.orchestra.run.smithi027.stdout: "take": 0, 2024-04-05T23:47:31.347 INFO:teuthology.orchestra.run.smithi027.stdout: "take_sum": 0, 2024-04-05T23:47:31.347 INFO:teuthology.orchestra.run.smithi027.stdout: "put": 0, 2024-04-05T23:47:31.347 INFO:teuthology.orchestra.run.smithi027.stdout: "put_sum": 0, 2024-04-05T23:47:31.347 INFO:teuthology.orchestra.run.smithi027.stdout: "wait": { 2024-04-05T23:47:31.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgcount": 0, 2024-04-05T23:47:31.348 INFO:teuthology.orchestra.run.smithi027.stdout: "sum": 0.000000000, 2024-04-05T23:47:31.348 INFO:teuthology.orchestra.run.smithi027.stdout: "avgtime": 0.000000000 2024-04-05T23:47:31.348 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:31.348 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-05T23:47:31.348 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-05T23:47:31.620 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-05T23:47:31.621 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-05T23:47:31.621 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-04-05T23:47:31.621 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-05T23:47:31.621 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-05T23:47:31.621 DEBUG:tasks.check_counter:Getting stats from c 2024-04-05T23:47:31.621 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.c perf dump 2024-04-05T23:47:31.870 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:47:31.977 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:31 smithi130 ceph-mon[79417]: pgmap v440: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 178 KiB/s wr, 24 op/s 2024-04-05T23:47:32.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:31 smithi027 ceph-mon[78597]: pgmap v440: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 178 KiB/s wr, 24 op/s 2024-04-05T23:47:32.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:31 smithi016 ceph-mon[38248]: pgmap v440: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 178 KiB/s wr, 24 op/s 2024-04-05T23:47:32.852 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-05T23:47:32.852 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:32.852 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 67222, 2024-04-05T23:47:32.852 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 105778, 2024-04-05T23:47:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 24581496, 2024-04-05T23:47:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 89729835, 2024-04-05T23:47:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 15, 2024-04-05T23:47:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 3, 2024-04-05T23:47:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 4.015714675, 2024-04-05T23:47:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 1.742879124, 2024-04-05T23:47:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 2.377050207, 2024-04-05T23:47:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.159387848, 2024-04-05T23:47:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 105777, 2024-04-05T23:47:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2.208585600, 2024-04-05T23:47:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000020879 2024-04-05T23:47:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 140731, 2024-04-05T23:47:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.022525424, 2024-04-05T23:47:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000160 2024-04-05T23:47:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1966032, 2024-04-05T23:47:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 164448 2024-04-05T23:47:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 7627, 2024-04-05T23:47:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 7660, 2024-04-05T23:47:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1962178, 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 28160627, 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 20, 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 10, 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.501571618, 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.179514136, 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.111621315, 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.171949747, 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 7655, 2024-04-05T23:47:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.214843644, 2024-04-05T23:47:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000028065 2024-04-05T23:47:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 42, 2024-04-05T23:47:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000005352, 2024-04-05T23:47:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000127 2024-04-05T23:47:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 4480, 2024-04-05T23:47:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 119552 2024-04-05T23:47:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 87195, 2024-04-05T23:47:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 66828, 2024-04-05T23:47:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 30259928, 2024-04-05T23:47:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 31001629, 2024-04-05T23:47:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 9, 2024-04-05T23:47:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-04-05T23:47:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 3.929297907, 2024-04-05T23:47:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 1.521277413, 2024-04-05T23:47:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 2.920246477, 2024-04-05T23:47:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.121249738, 2024-04-05T23:47:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 66829, 2024-04-05T23:47:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1.236297579, 2024-04-05T23:47:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000018499 2024-04-05T23:47:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 143031, 2024-04-05T23:47:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.020471936, 2024-04-05T23:47:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000143 2024-04-05T23:47:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-04-05T23:47:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 379040 2024-04-05T23:47:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-04-05T23:47:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-04-05T23:47:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-04-05T23:47:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-04-05T23:47:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-05T23:47:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-05T23:47:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 16483, 2024-04-05T23:47:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 9.243005525, 2024-04-05T23:47:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000560759 2024-04-05T23:47:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-05T23:47:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-05T23:47:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 13732, 2024-04-05T23:47:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.607375297, 2024-04-05T23:47:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000044230 2024-04-05T23:47:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-04-05T23:47:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 44495, 2024-04-05T23:47:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 42155, 2024-04-05T23:47:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-04-05T23:47:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 42155, 2024-04-05T23:47:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 64.963533520, 2024-04-05T23:47:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001541063 2024-04-05T23:47:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-04-05T23:47:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 2340, 2024-04-05T23:47:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 5, 2024-04-05T23:47:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-04-05T23:47:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 11, 2024-04-05T23:47:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-04-05T23:47:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-04-05T23:47:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 899, 2024-04-05T23:47:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 199, 2024-04-05T23:47:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 655, 2024-04-05T23:47:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 45, 2024-04-05T23:47:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 569, 2024-04-05T23:47:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 424, 2024-04-05T23:47:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 207, 2024-04-05T23:47:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 207, 2024-04-05T23:47:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 8, 2024-04-05T23:47:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 79248, 2024-04-05T23:47:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 34554, 2024-04-05T23:47:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 2336, 2024-04-05T23:47:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 50, 2024-04-05T23:47:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-04-05T23:47:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 18, 2024-04-05T23:47:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 34428, 2024-04-05T23:47:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 6, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 206, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 14, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 941, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 12, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 614, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 211488726, 2024-04-05T23:47:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-04-05T23:47:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 12, 2024-04-05T23:47:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-04-05T23:47:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-04-05T23:47:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-04-05T23:47:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 12, 2024-04-05T23:47:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-04-05T23:47:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 6195, 2024-04-05T23:47:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 28343, 2024-04-05T23:47:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 8662, 2024-04-05T23:47:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 1258, 2024-04-05T23:47:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 814, 2024-04-05T23:47:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 40479 2024-04-05T23:47:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-04-05T23:47:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 84, 2024-04-05T23:47:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 25, 2024-04-05T23:47:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 14, 2024-04-05T23:47:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 73, 2024-04-05T23:47:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 153, 2024-04-05T23:47:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 26, 2024-04-05T23:47:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 6914, 2024-04-05T23:47:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 6888, 2024-04-05T23:47:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-04-05T23:47:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-04-05T23:47:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-04-05T23:47:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-04-05T23:47:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 11, 2024-04-05T23:47:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-04-05T23:47:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-04-05T23:47:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-04-05T23:47:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-04-05T23:47:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 34049, 2024-04-05T23:47:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-04-05T23:47:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-04-05T23:47:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 34049, 2024-04-05T23:47:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-04-05T23:47:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-04-05T23:47:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 39, 2024-04-05T23:47:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-04-05T23:47:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-04-05T23:47:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 39, 2024-04-05T23:47:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-04-05T23:47:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-04-05T23:47:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-04-05T23:47:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 61148610, 2024-04-05T23:47:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-04-05T23:47:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-04-05T23:47:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8168, 2024-04-05T23:47:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 16.797327728, 2024-04-05T23:47:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002056479 2024-04-05T23:47:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-04-05T23:47:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-04-05T23:47:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 641, 2024-04-05T23:47:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 7823, 2024-04-05T23:47:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 7182, 2024-04-05T23:47:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 57, 2024-04-05T23:47:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 94, 2024-04-05T23:47:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 37, 2024-04-05T23:47:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 899, 2024-04-05T23:47:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 12184, 2024-04-05T23:47:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 11285, 2024-04-05T23:47:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 207, 2024-04-05T23:47:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 7304, 2024-04-05T23:47:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 7097, 2024-04-05T23:47:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 62040, 2024-04-05T23:47:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 198964 2024-04-05T23:47:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-04-05T23:47:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 71945, 2024-04-05T23:47:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 773, 2024-04-05T23:47:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 44495, 2024-04-05T23:47:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 195, 2024-04-05T23:47:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 7596, 2024-04-05T23:47:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-04-05T23:47:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 6967, 2024-04-05T23:47:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 16.934816671, 2024-04-05T23:47:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002430718 2024-04-05T23:47:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-04-05T23:47:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 140, 2024-04-05T23:47:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2.732281845, 2024-04-05T23:47:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.019516298 2024-04-05T23:47:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-04-05T23:47:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-04-05T23:47:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8161, 2024-04-05T23:47:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 23.360062459, 2024-04-05T23:47:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002862401 2024-04-05T23:47:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-04-05T23:47:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.883 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.883 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.883 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.883 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-04-05T23:47:32.883 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.883 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.883 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.883 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.884 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:32.884 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.884 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.884 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.884 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.884 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:32.884 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.884 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.885 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.885 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.885 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-04-05T23:47:32.885 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.885 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.885 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.885 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-04-05T23:47:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-04-05T23:47:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-04-05T23:47:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-04-05T23:47:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-04-05T23:47:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 19929, 2024-04-05T23:47:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 13.673292639, 2024-04-05T23:47:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000686100 2024-04-05T23:47:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-04-05T23:47:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 40, 2024-04-05T23:47:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.074922805, 2024-04-05T23:47:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001873070 2024-04-05T23:47:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-04-05T23:47:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-04-05T23:47:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4, 2024-04-05T23:47:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.001458288, 2024-04-05T23:47:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000364572 2024-04-05T23:47:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-04-05T23:47:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-04-05T23:47:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 6914, 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 8.186698813, 2024-04-05T23:47:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001184075 2024-04-05T23:47:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-04-05T23:47:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 1, 2024-04-05T23:47:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 3, 2024-04-05T23:47:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 2, 2024-04-05T23:47:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 1, 2024-04-05T23:47:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-04-05T23:47:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 4041, 2024-04-05T23:47:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 4041, 2024-04-05T23:47:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 840, 2024-04-05T23:47:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-04-05T23:47:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-04-05T23:47:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 361160, 2024-04-05T23:47:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 778, 2024-04-05T23:47:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 176, 2024-04-05T23:47:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 2, 2024-04-05T23:47:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-04-05T23:47:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-04-05T23:47:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 188, 2024-04-05T23:47:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-04-05T23:47:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 2435608, 2024-04-05T23:47:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 24904, 2024-04-05T23:47:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-04-05T23:47:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-04-05T23:47:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-04-05T23:47:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-04-05T23:47:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-04-05T23:47:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 16702, 2024-04-05T23:47:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 58062670, 2024-04-05T23:47:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 27, 2024-04-05T23:47:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 16702, 2024-04-05T23:47:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-04-05T23:47:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 16702, 2024-04-05T23:47:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 33.652397286, 2024-04-05T23:47:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002014872 2024-04-05T23:47:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-04-05T23:47:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-04-05T23:47:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 16702, 2024-04-05T23:47:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 17010 2024-04-05T23:47:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 16702, 2024-04-05T23:47:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 567, 2024-04-05T23:47:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 16135, 2024-04-05T23:47:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-04-05T23:47:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-04-05T23:47:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 0, 2024-04-05T23:47:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10, 2024-04-05T23:47:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 550, 2024-04-05T23:47:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 8718, 2024-04-05T23:47:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 331, 2024-04-05T23:47:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-04-05T23:47:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-04-05T23:47:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-04-05T23:47:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-04-05T23:47:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 6913, 2024-04-05T23:47:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-04-05T23:47:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 12, 2024-04-05T23:47:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 20, 2024-04-05T23:47:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-04-05T23:47:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-04-05T23:47:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-04-05T23:47:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-04-05T23:47:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-04-05T23:47:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-04-05T23:47:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-04-05T23:47:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-04-05T23:47:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-04-05T23:47:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-04-05T23:47:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-04-05T23:47:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-04-05T23:47:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-04-05T23:47:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-04-05T23:47:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-04-05T23:47:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-04-05T23:47:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-04-05T23:47:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-04-05T23:47:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-04-05T23:47:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-04-05T23:47:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-04-05T23:47:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 101, 2024-04-05T23:47:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-04-05T23:47:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 22, 2024-04-05T23:47:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 12, 2024-04-05T23:47:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 12, 2024-04-05T23:47:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-04-05T23:47:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-04-05T23:47:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 304, 2024-04-05T23:47:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 10, 2024-04-05T23:47:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 142 2024-04-05T23:47:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-04-05T23:47:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-04-05T23:47:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-04-05T23:47:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-04-05T23:47:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-04-05T23:47:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-04-05T23:47:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-04-05T23:47:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 6, 2024-04-05T23:47:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-04-05T23:47:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 6, 2024-04-05T23:47:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 6888, 2024-04-05T23:47:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-05T23:47:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 162040, 2024-04-05T23:47:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 42197399, 2024-04-05T23:47:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 162040, 2024-04-05T23:47:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 162040, 2024-04-05T23:47:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 42197399, 2024-04-05T23:47:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-05T23:47:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 16702, 2024-04-05T23:47:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 58758878, 2024-04-05T23:47:32.922 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 16702, 2024-04-05T23:47:32.922 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 58758878, 2024-04-05T23:47:32.922 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:32.922 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.922 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.922 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-04-05T23:47:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 16702, 2024-04-05T23:47:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 16702, 2024-04-05T23:47:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 16702, 2024-04-05T23:47:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 16702, 2024-04-05T23:47:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-05T23:47:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 6888, 2024-04-05T23:47:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 696288, 2024-04-05T23:47:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 6888, 2024-04-05T23:47:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 550, 2024-04-05T23:47:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 696288, 2024-04-05T23:47:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x55e6887dc0a0": { 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 8800, 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 34049, 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 56954306, 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 34049, 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 8168, 2024-04-05T23:47:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 56945506, 2024-04-05T23:47:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:32.929 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-05T23:47:33.199 INFO:tasks.check_counter:Daemon mds.c mds.exported=6 2024-04-05T23:47:33.199 INFO:tasks.check_counter:Daemon mds.c mds.imported=14 2024-04-05T23:47:33.199 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-05T23:47:33.199 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-05T23:47:33.199 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-05T23:47:33.199 DEBUG:tasks.check_counter:Getting stats from f 2024-04-05T23:47:33.200 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.f perf dump 2024-04-05T23:47:33.421 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:47:34.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:33 smithi027 ceph-mon[78597]: pgmap v441: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 159 KiB/s wr, 16 op/s 2024-04-05T23:47:34.236 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:33 smithi130 ceph-mon[79417]: pgmap v441: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 159 KiB/s wr, 16 op/s 2024-04-05T23:47:34.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:33 smithi016 ceph-mon[38248]: pgmap v441: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 159 KiB/s wr, 16 op/s 2024-04-05T23:47:34.441 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-05T23:47:34.442 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:34.442 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 1181, 2024-04-05T23:47:34.442 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 1143, 2024-04-05T23:47:34.442 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 51057642, 2024-04-05T23:47:34.442 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 390159, 2024-04-05T23:47:34.442 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 7, 2024-04-05T23:47:34.443 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 3, 2024-04-05T23:47:34.443 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.131595624, 2024-04-05T23:47:34.443 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.031009745, 2024-04-05T23:47:34.443 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.061942635, 2024-04-05T23:47:34.443 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.030892151, 2024-04-05T23:47:34.443 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:34.443 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1142, 2024-04-05T23:47:34.443 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.046109374, 2024-04-05T23:47:34.443 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000040375 2024-04-05T23:47:34.444 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.444 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:34.444 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.444 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.444 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.444 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.444 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 431088, 2024-04-05T23:47:34.444 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 87552 2024-04-05T23:47:34.445 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.445 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:34.445 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 689, 2024-04-05T23:47:34.445 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 800, 2024-04-05T23:47:34.445 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 84755115, 2024-04-05T23:47:34.445 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 489625, 2024-04-05T23:47:34.445 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 6, 2024-04-05T23:47:34.445 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-04-05T23:47:34.445 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.089349799, 2024-04-05T23:47:34.446 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.019114615, 2024-04-05T23:47:34.446 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.049563918, 2024-04-05T23:47:34.446 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.015359505, 2024-04-05T23:47:34.446 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:34.446 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 798, 2024-04-05T23:47:34.446 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.026470246, 2024-04-05T23:47:34.446 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000033170 2024-04-05T23:47:34.446 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.447 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:34.447 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.447 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.447 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.447 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.447 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-05T23:47:34.447 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 275600 2024-04-05T23:47:34.447 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.448 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:34.448 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 469, 2024-04-05T23:47:34.448 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 469, 2024-04-05T23:47:34.448 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 58842949, 2024-04-05T23:47:34.448 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 145079, 2024-04-05T23:47:34.448 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 6, 2024-04-05T23:47:34.448 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-04-05T23:47:34.448 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.062677482, 2024-04-05T23:47:34.448 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.011087093, 2024-04-05T23:47:34.449 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.037313102, 2024-04-05T23:47:34.449 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.010701099, 2024-04-05T23:47:34.449 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:34.449 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 468, 2024-04-05T23:47:34.449 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.020633120, 2024-04-05T23:47:34.449 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000044087 2024-04-05T23:47:34.449 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.449 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:34.450 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.450 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.450 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.450 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.450 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-05T23:47:34.450 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 448 2024-04-05T23:47:34.450 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.451 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-04-05T23:47:34.451 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-04-05T23:47:34.451 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-04-05T23:47:34.451 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.451 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-04-05T23:47:34.452 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-05T23:47:34.452 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-05T23:47:34.452 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3575, 2024-04-05T23:47:34.452 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.130889890, 2024-04-05T23:47:34.452 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000036612 2024-04-05T23:47:34.452 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:34.452 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.452 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:34.453 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-05T23:47:34.453 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-05T23:47:34.453 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.453 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.453 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.453 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:34.453 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.453 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-04-05T23:47:34.453 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 0, 2024-04-05T23:47:34.454 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 0, 2024-04-05T23:47:34.454 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-04-05T23:47:34.454 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.454 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.454 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.454 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.454 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-04-05T23:47:34.454 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 0, 2024-04-05T23:47:34.455 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-04-05T23:47:34.455 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-04-05T23:47:34.455 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 0, 2024-04-05T23:47:34.455 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-04-05T23:47:34.455 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-04-05T23:47:34.455 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 3558, 2024-04-05T23:47:34.455 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 1397, 2024-04-05T23:47:34.455 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 2161, 2024-04-05T23:47:34.455 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2024-04-05T23:47:34.456 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 1561, 2024-04-05T23:47:34.456 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 0, 2024-04-05T23:47:34.456 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 0, 2024-04-05T23:47:34.456 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 0, 2024-04-05T23:47:34.456 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 20, 2024-04-05T23:47:34.456 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 0, 2024-04-05T23:47:34.456 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 0, 2024-04-05T23:47:34.456 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2024-04-05T23:47:34.457 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 0, 2024-04-05T23:47:34.457 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-04-05T23:47:34.457 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:34.457 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2024-04-05T23:47:34.457 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-04-05T23:47:34.457 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-04-05T23:47:34.457 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2024-04-05T23:47:34.457 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2024-04-05T23:47:34.458 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 0, 2024-04-05T23:47:34.458 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2024-04-05T23:47:34.458 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:34.458 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2024-04-05T23:47:34.458 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:34.458 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 614, 2024-04-05T23:47:34.458 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 211488726, 2024-04-05T23:47:34.458 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-04-05T23:47:34.459 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2024-04-05T23:47:34.459 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:34.459 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:34.459 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:34.459 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-04-05T23:47:34.459 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-04-05T23:47:34.459 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-04-05T23:47:34.459 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2024-04-05T23:47:34.460 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-04-05T23:47:34.460 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2024-04-05T23:47:34.460 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2024-04-05T23:47:34.460 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:34.460 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:34.460 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:34.460 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 0, 2024-04-05T23:47:34.461 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 0, 2024-04-05T23:47:34.461 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2024-04-05T23:47:34.461 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2024-04-05T23:47:34.461 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.461 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-04-05T23:47:34.461 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-04-05T23:47:34.461 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-04-05T23:47:34.461 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-04-05T23:47:34.462 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 0, 2024-04-05T23:47:34.462 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2024-04-05T23:47:34.462 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-04-05T23:47:34.462 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:34.462 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:34.462 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2024-04-05T23:47:34.462 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2024-04-05T23:47:34.462 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:34.463 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-04-05T23:47:34.463 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:34.463 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:34.463 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:34.463 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-04-05T23:47:34.463 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-04-05T23:47:34.463 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-04-05T23:47:34.463 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2024-04-05T23:47:34.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-04-05T23:47:34.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-04-05T23:47:34.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:34.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-04-05T23:47:34.464 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.464 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-04-05T23:47:34.464 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 0, 2024-04-05T23:47:34.464 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-04-05T23:47:34.464 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-04-05T23:47:34.465 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 31913, 2024-04-05T23:47:34.465 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-04-05T23:47:34.465 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-04-05T23:47:34.465 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 0, 2024-04-05T23:47:34.465 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-04-05T23:47:34.465 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-04-05T23:47:34.466 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 7, 2024-04-05T23:47:34.466 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-04-05T23:47:34.466 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-04-05T23:47:34.466 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 274558983, 2024-04-05T23:47:34.466 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 286008294, 2024-04-05T23:47:34.466 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 286003794, 2024-04-05T23:47:34.466 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-04-05T23:47:34.467 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.467 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.467 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.467 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.467 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 31913 2024-04-05T23:47:34.467 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.467 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-04-05T23:47:34.467 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 199, 2024-04-05T23:47:34.467 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 4933, 2024-04-05T23:47:34.468 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 4734, 2024-04-05T23:47:34.468 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 65, 2024-04-05T23:47:34.468 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 154, 2024-04-05T23:47:34.468 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 89, 2024-04-05T23:47:34.468 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 3558, 2024-04-05T23:47:34.468 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 5815, 2024-04-05T23:47:34.468 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 2257, 2024-04-05T23:47:34.468 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 0, 2024-04-05T23:47:34.468 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 0, 2024-04-05T23:47:34.469 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 0, 2024-04-05T23:47:34.469 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 58216, 2024-04-05T23:47:34.469 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 207132 2024-04-05T23:47:34.469 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.469 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-04-05T23:47:34.469 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 0, 2024-04-05T23:47:34.469 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2024-04-05T23:47:34.469 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 0, 2024-04-05T23:47:34.470 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 0, 2024-04-05T23:47:34.470 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2024-04-05T23:47:34.470 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-04-05T23:47:34.470 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.470 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.470 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.470 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.470 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-04-05T23:47:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.472 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.472 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-04-05T23:47:34.472 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.472 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.472 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.472 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.472 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-04-05T23:47:34.472 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.473 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.473 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.473 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.473 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:34.473 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.473 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.473 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.473 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.473 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-04-05T23:47:34.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.474 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.474 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.474 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-04-05T23:47:34.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.474 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.475 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.475 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:34.475 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.475 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.475 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.475 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.475 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:34.475 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.475 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.476 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.476 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.476 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-04-05T23:47:34.476 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.476 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.476 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.476 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.476 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-04-05T23:47:34.476 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.477 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.477 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.477 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-04-05T23:47:34.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.477 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.477 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.478 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-04-05T23:47:34.478 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.478 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.478 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.478 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.478 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-04-05T23:47:34.479 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.479 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.479 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.479 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.479 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-04-05T23:47:34.479 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.479 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.479 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.480 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.480 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-04-05T23:47:34.480 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.480 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.480 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.480 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.480 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:34.480 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.480 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.481 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.481 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.481 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:34.481 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.481 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.481 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.481 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.481 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-04-05T23:47:34.482 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.482 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.482 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.482 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.482 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:34.482 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.482 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.482 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.482 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.483 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:34.483 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.483 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.483 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.483 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.483 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-04-05T23:47:34.483 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.483 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.484 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.484 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.484 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:34.484 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.484 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.484 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.484 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.484 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:34.484 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.485 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.485 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.485 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.485 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-04-05T23:47:34.485 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.485 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.485 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.485 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.486 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-04-05T23:47:34.486 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.486 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.486 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.486 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.486 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-04-05T23:47:34.486 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.486 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.487 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.487 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.487 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-04-05T23:47:34.487 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.487 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.487 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.487 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.487 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:34.487 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:34.488 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:34.488 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.488 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.488 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.488 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:34.488 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.488 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-04-05T23:47:34.488 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 2, 2024-04-05T23:47:34.489 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 2, 2024-04-05T23:47:34.489 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 0, 2024-04-05T23:47:34.489 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 0, 2024-04-05T23:47:34.489 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-04-05T23:47:34.489 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 0, 2024-04-05T23:47:34.489 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 0, 2024-04-05T23:47:34.489 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 2779, 2024-04-05T23:47:34.489 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:34.490 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.490 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-04-05T23:47:34.490 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:34.490 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:34.490 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:34.490 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:34.490 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:34.490 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:34.490 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:34.491 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:34.491 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:34.491 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:34.491 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:34.491 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:34.491 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:34.491 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:34.491 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:34.492 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:34.492 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:34.492 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:34.492 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:34.492 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:34.492 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:34.492 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:34.492 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:34.493 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:34.493 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:34.493 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:34.493 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:34.493 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:34.493 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:34.493 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:34.493 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:34.493 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-04-05T23:47:34.494 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:34.494 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:34.494 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:34.494 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:34.494 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 239084, 2024-04-05T23:47:34.494 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 323, 2024-04-05T23:47:34.494 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-04-05T23:47:34.494 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-04-05T23:47:34.495 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-04-05T23:47:34.495 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-04-05T23:47:34.495 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:34.495 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:34.495 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:34.495 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:34.495 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:34.495 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 188, 2024-04-05T23:47:34.495 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:34.496 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:34.496 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:34.496 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-04-05T23:47:34.496 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 3146444, 2024-04-05T23:47:34.496 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 22433, 2024-04-05T23:47:34.496 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:34.496 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-04-05T23:47:34.496 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:34.497 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-04-05T23:47:34.497 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.497 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-04-05T23:47:34.497 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-04-05T23:47:34.497 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-04-05T23:47:34.497 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 2126, 2024-04-05T23:47:34.497 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 26, 2024-04-05T23:47:34.497 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 4, 2024-04-05T23:47:34.497 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 2126, 2024-04-05T23:47:34.498 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-04-05T23:47:34.498 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2126, 2024-04-05T23:47:34.498 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1.643970818, 2024-04-05T23:47:34.498 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000773269 2024-04-05T23:47:34.498 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.498 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-04-05T23:47:34.498 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-04-05T23:47:34.498 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2126, 2024-04-05T23:47:34.499 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2127 2024-04-05T23:47:34.499 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.499 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 2126, 2024-04-05T23:47:34.499 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 2126, 2024-04-05T23:47:34.499 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 0, 2024-04-05T23:47:34.499 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-04-05T23:47:34.499 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-04-05T23:47:34.499 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 1110, 2024-04-05T23:47:34.499 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 0, 2024-04-05T23:47:34.500 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 1013, 2024-04-05T23:47:34.500 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 0, 2024-04-05T23:47:34.500 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 0, 2024-04-05T23:47:34.500 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-04-05T23:47:34.500 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-04-05T23:47:34.500 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-04-05T23:47:34.500 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-04-05T23:47:34.500 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 0, 2024-04-05T23:47:34.501 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-04-05T23:47:34.501 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:34.501 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:34.501 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 2, 2024-04-05T23:47:34.501 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 0, 2024-04-05T23:47:34.501 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:34.501 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:34.501 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:34.502 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-04-05T23:47:34.502 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-04-05T23:47:34.502 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-04-05T23:47:34.502 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:34.502 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-04-05T23:47:34.502 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:34.502 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-04-05T23:47:34.502 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-04-05T23:47:34.503 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-04-05T23:47:34.503 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-04-05T23:47:34.503 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-04-05T23:47:34.503 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-04-05T23:47:34.503 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-04-05T23:47:34.503 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-04-05T23:47:34.503 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-04-05T23:47:34.503 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-04-05T23:47:34.503 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-04-05T23:47:34.504 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-04-05T23:47:34.504 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-04-05T23:47:34.504 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-04-05T23:47:34.504 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-04-05T23:47:34.504 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-04-05T23:47:34.504 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-04-05T23:47:34.504 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 101, 2024-04-05T23:47:34.504 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-04-05T23:47:34.505 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 16, 2024-04-05T23:47:34.505 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 10, 2024-04-05T23:47:34.505 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 10, 2024-04-05T23:47:34.505 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-04-05T23:47:34.505 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-04-05T23:47:34.505 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 0, 2024-04-05T23:47:34.505 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 2, 2024-04-05T23:47:34.505 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 0 2024-04-05T23:47:34.505 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.506 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-04-05T23:47:34.506 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 0, 2024-04-05T23:47:34.506 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:34.506 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-04-05T23:47:34.506 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-04-05T23:47:34.506 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.506 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-04-05T23:47:34.506 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-04-05T23:47:34.507 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.507 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-04-05T23:47:34.507 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:34.507 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2024-04-05T23:47:34.507 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-04-05T23:47:34.507 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2024-04-05T23:47:34.507 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2024-04-05T23:47:34.507 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:34.508 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.508 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:34.508 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:34.508 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-05T23:47:34.508 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:34.508 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 2335, 2024-04-05T23:47:34.508 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 194432130, 2024-04-05T23:47:34.508 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:34.508 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 2335, 2024-04-05T23:47:34.509 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:34.509 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:34.509 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2335, 2024-04-05T23:47:34.509 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 194432130, 2024-04-05T23:47:34.509 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:34.509 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.509 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.509 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.510 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:34.510 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.510 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:34.510 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:34.510 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-05T23:47:34.510 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:34.510 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:34.510 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:34.510 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:34.511 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:34.511 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 2126, 2024-04-05T23:47:34.511 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 193630680, 2024-04-05T23:47:34.511 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2126, 2024-04-05T23:47:34.511 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 193630680, 2024-04-05T23:47:34.511 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:34.511 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.511 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.512 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.512 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:34.512 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.512 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:34.512 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:34.512 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-04-05T23:47:34.512 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:34.512 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:34.512 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:34.513 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:34.513 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:34.513 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 2126, 2024-04-05T23:47:34.513 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 2126, 2024-04-05T23:47:34.513 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2126, 2024-04-05T23:47:34.513 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 2126, 2024-04-05T23:47:34.513 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:34.513 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.514 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.514 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.514 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:34.514 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.514 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:34.514 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:34.514 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-05T23:47:34.514 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:34.515 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:34.515 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:34.515 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:34.515 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:34.515 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:34.515 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:34.515 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-04-05T23:47:34.515 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-04-05T23:47:34.515 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:34.516 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.516 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.516 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.516 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:34.516 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:34.516 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x55cf23648500": { 2024-04-05T23:47:34.516 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:34.516 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-05T23:47:34.517 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:34.517 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:34.517 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:34.517 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:34.517 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:34.517 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:34.517 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:34.517 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-04-05T23:47:34.518 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-04-05T23:47:34.518 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:34.518 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:34.518 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:34.518 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:34.518 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:34.518 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:34.518 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-05T23:47:34.805 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-05T23:47:34.805 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-05T23:47:34.805 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-05T23:47:34.806 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-05T23:47:34.806 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-05T23:47:34.806 DEBUG:tasks.check_counter:Getting stats from i 2024-04-05T23:47:34.806 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.i perf dump 2024-04-05T23:47:35.028 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:47:36.002 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-05T23:47:36.002 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:36.002 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 7944, 2024-04-05T23:47:36.002 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 8843, 2024-04-05T23:47:36.003 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 4086158, 2024-04-05T23:47:36.003 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 24859604, 2024-04-05T23:47:36.003 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 20, 2024-04-05T23:47:36.003 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 8, 2024-04-05T23:47:36.003 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.565228501, 2024-04-05T23:47:36.003 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.219063059, 2024-04-05T23:47:36.003 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.195880163, 2024-04-05T23:47:36.003 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.136132605, 2024-04-05T23:47:36.004 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:36.004 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8839, 2024-04-05T23:47:36.004 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.214382377, 2024-04-05T23:47:36.004 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000024254 2024-04-05T23:47:36.004 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.004 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:36.004 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4013, 2024-04-05T23:47:36.004 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000650449, 2024-04-05T23:47:36.004 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000162 2024-04-05T23:47:36.005 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.005 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1970512, 2024-04-05T23:47:36.005 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 156160 2024-04-05T23:47:36.005 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.005 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:36.005 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 76609, 2024-04-05T23:47:36.005 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 58891, 2024-04-05T23:47:36.005 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 25864480, 2024-04-05T23:47:36.006 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 28007291, 2024-04-05T23:47:36.006 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 11, 2024-04-05T23:47:36.006 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-04-05T23:47:36.006 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 3.502010768, 2024-04-05T23:47:36.006 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 1.359239322, 2024-04-05T23:47:36.006 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 2.608217696, 2024-04-05T23:47:36.006 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.091765739, 2024-04-05T23:47:36.006 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:36.007 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 58892, 2024-04-05T23:47:36.007 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1.171474312, 2024-04-05T23:47:36.007 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000019891 2024-04-05T23:47:36.007 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.007 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:36.007 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 126576, 2024-04-05T23:47:36.007 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.018468151, 2024-04-05T23:47:36.007 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000145 2024-04-05T23:47:36.007 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.008 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-05T23:47:36.008 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 493152 2024-04-05T23:47:36.008 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.008 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:36.008 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 50364, 2024-04-05T23:47:36.008 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 75217, 2024-04-05T23:47:36.008 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 17127025, 2024-04-05T23:47:36.008 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 79041379, 2024-04-05T23:47:36.009 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 12, 2024-04-05T23:47:36.009 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 7, 2024-04-05T23:47:36.009 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 2.933143626, 2024-04-05T23:47:36.009 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 1.246180840, 2024-04-05T23:47:36.009 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 1.825628623, 2024-04-05T23:47:36.009 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.148228464, 2024-04-05T23:47:36.009 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:36.009 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 75215, 2024-04-05T23:47:36.010 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2.119446775, 2024-04-05T23:47:36.010 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000028178 2024-04-05T23:47:36.010 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.010 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:36.010 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 99346, 2024-04-05T23:47:36.010 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.015951185, 2024-04-05T23:47:36.010 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000160 2024-04-05T23:47:36.010 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.011 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-05T23:47:36.011 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 0 2024-04-05T23:47:36.011 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.011 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-04-05T23:47:36.011 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-04-05T23:47:36.011 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-04-05T23:47:36.011 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.011 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-04-05T23:47:36.012 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-05T23:47:36.012 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-05T23:47:36.012 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 19685, 2024-04-05T23:47:36.012 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 9.361489562, 2024-04-05T23:47:36.012 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000475564 2024-04-05T23:47:36.012 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:36.012 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.012 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:36.012 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-05T23:47:36.013 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-05T23:47:36.013 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 9188, 2024-04-05T23:47:36.013 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.404545617, 2024-04-05T23:47:36.013 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000044029 2024-04-05T23:47:36.013 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:36.013 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.013 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-04-05T23:47:36.013 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 27562, 2024-04-05T23:47:36.013 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 25845, 2024-04-05T23:47:36.014 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-04-05T23:47:36.014 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 25845, 2024-04-05T23:47:36.014 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 194.274203180, 2024-04-05T23:47:36.014 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.007516897 2024-04-05T23:47:36.014 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.014 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-04-05T23:47:36.014 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 1716, 2024-04-05T23:47:36.014 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-04-05T23:47:36.015 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-04-05T23:47:36.015 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 11, 2024-04-05T23:47:36.015 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-04-05T23:47:36.015 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-04-05T23:47:36.015 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 507, 2024-04-05T23:47:36.015 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 31, 2024-04-05T23:47:36.015 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 330, 2024-04-05T23:47:36.015 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 146, 2024-04-05T23:47:36.016 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 462, 2024-04-05T23:47:36.016 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 402, 2024-04-05T23:47:36.016 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 176, 2024-04-05T23:47:36.016 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 176, 2024-04-05T23:47:36.016 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 9, 2024-04-05T23:47:36.016 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 60631, 2024-04-05T23:47:36.016 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 32298, 2024-04-05T23:47:36.016 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 1712, 2024-04-05T23:47:36.016 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 81, 2024-04-05T23:47:36.017 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-04-05T23:47:36.017 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:36.017 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 1384, 2024-04-05T23:47:36.017 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 51357, 2024-04-05T23:47:36.017 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-04-05T23:47:36.017 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 7, 2024-04-05T23:47:36.017 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 309, 2024-04-05T23:47:36.017 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 8, 2024-04-05T23:47:36.017 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 468, 2024-04-05T23:47:36.018 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:36.018 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 19, 2024-04-05T23:47:36.018 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:36.018 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 614, 2024-04-05T23:47:36.018 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 211488726, 2024-04-05T23:47:36.018 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-04-05T23:47:36.018 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 19, 2024-04-05T23:47:36.018 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:36.018 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:36.019 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:36.019 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-04-05T23:47:36.019 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-04-05T23:47:36.019 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-04-05T23:47:36.019 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 19, 2024-04-05T23:47:36.019 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-04-05T23:47:36.019 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 2494, 2024-04-05T23:47:36.019 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 19457, 2024-04-05T23:47:36.019 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:36.020 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:36.020 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:36.020 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 11616, 2024-04-05T23:47:36.020 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 6732, 2024-04-05T23:47:36.020 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 1685, 2024-04-05T23:47:36.020 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 25752 2024-04-05T23:47:36.021 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.021 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-04-05T23:47:36.021 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 27, 2024-04-05T23:47:36.021 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 60, 2024-04-05T23:47:36.021 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 31, 2024-04-05T23:47:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 147, 2024-04-05T23:47:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 93, 2024-04-05T23:47:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 28, 2024-04-05T23:47:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 4410, 2024-04-05T23:47:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 4382, 2024-04-05T23:47:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-04-05T23:47:36.023 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:36.023 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:36.023 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:36.023 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-04-05T23:47:36.023 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-04-05T23:47:36.023 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-04-05T23:47:36.023 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 7, 2024-04-05T23:47:36.023 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-04-05T23:47:36.024 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-04-05T23:47:36.024 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:36.024 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-04-05T23:47:36.024 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.024 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-04-05T23:47:36.024 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 31208, 2024-04-05T23:47:36.024 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-04-05T23:47:36.024 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-04-05T23:47:36.024 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 31208, 2024-04-05T23:47:36.025 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-04-05T23:47:36.025 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-04-05T23:47:36.025 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 40, 2024-04-05T23:47:36.025 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-04-05T23:47:36.025 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-04-05T23:47:36.025 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 40, 2024-04-05T23:47:36.025 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-04-05T23:47:36.025 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-04-05T23:47:36.025 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-04-05T23:47:36.026 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 59996848, 2024-04-05T23:47:36.026 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-04-05T23:47:36.026 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-04-05T23:47:36.026 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 10087, 2024-04-05T23:47:36.026 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 20.473045333, 2024-04-05T23:47:36.026 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002029646 2024-04-05T23:47:36.026 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.026 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-04-05T23:47:36.026 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.027 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-04-05T23:47:36.027 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 341, 2024-04-05T23:47:36.027 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 5017, 2024-04-05T23:47:36.027 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 4676, 2024-04-05T23:47:36.027 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 57, 2024-04-05T23:47:36.027 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 113, 2024-04-05T23:47:36.027 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 56, 2024-04-05T23:47:36.027 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 507, 2024-04-05T23:47:36.027 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 5884, 2024-04-05T23:47:36.028 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 5377, 2024-04-05T23:47:36.028 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 176, 2024-04-05T23:47:36.028 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 4719, 2024-04-05T23:47:36.028 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 4543, 2024-04-05T23:47:36.028 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 63180, 2024-04-05T23:47:36.028 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 198964 2024-04-05T23:47:36.028 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.028 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-04-05T23:47:36.029 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 50695, 2024-04-05T23:47:36.029 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 15, 2024-04-05T23:47:36.029 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 27562, 2024-04-05T23:47:36.029 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 1094, 2024-04-05T23:47:36.029 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 6457, 2024-04-05T23:47:36.029 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-04-05T23:47:36.029 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4364, 2024-04-05T23:47:36.029 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 10.630662090, 2024-04-05T23:47:36.030 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002435990 2024-04-05T23:47:36.030 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.030 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-04-05T23:47:36.030 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 84, 2024-04-05T23:47:36.030 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.062802200, 2024-04-05T23:47:36.030 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000747645 2024-04-05T23:47:36.030 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.031 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:36.031 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.031 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.031 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.031 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.031 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-04-05T23:47:36.031 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.033 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.033 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.033 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.033 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-04-05T23:47:36.033 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 6317, 2024-04-05T23:47:36.034 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 11.558085149, 2024-04-05T23:47:36.034 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001829679 2024-04-05T23:47:36.034 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.034 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:36.034 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.034 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.034 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.034 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.035 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-04-05T23:47:36.035 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.035 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.035 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.035 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.035 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-04-05T23:47:36.035 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.035 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.035 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.036 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.036 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:36.036 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.036 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.036 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.036 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.036 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:36.036 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.036 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.037 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.037 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.037 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-04-05T23:47:36.037 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.037 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.037 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.037 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.037 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-04-05T23:47:36.038 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.038 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.038 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.038 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.038 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-04-05T23:47:36.038 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.038 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.038 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.038 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.039 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-04-05T23:47:36.039 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.039 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.039 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.039 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.039 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-04-05T23:47:36.039 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.039 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.039 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.040 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.040 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-04-05T23:47:36.040 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 9156, 2024-04-05T23:47:36.040 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 6.754276524, 2024-04-05T23:47:36.040 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000737688 2024-04-05T23:47:36.040 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.040 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-04-05T23:47:36.040 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1500, 2024-04-05T23:47:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 153.713468590, 2024-04-05T23:47:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.102475645 2024-04-05T23:47:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:36.042 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.042 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.042 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.042 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.042 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-04-05T23:47:36.042 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.042 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.042 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.042 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.044 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.044 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-04-05T23:47:36.044 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 6, 2024-04-05T23:47:36.044 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.020285758, 2024-04-05T23:47:36.044 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.003380959 2024-04-05T23:47:36.044 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.044 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:36.044 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.044 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.045 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.045 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.045 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:36.045 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8, 2024-04-05T23:47:36.045 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.004350923, 2024-04-05T23:47:36.045 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000543865 2024-04-05T23:47:36.045 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.045 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-04-05T23:47:36.046 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.046 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.046 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.046 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.046 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-04-05T23:47:36.046 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.046 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.046 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.046 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.047 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-04-05T23:47:36.047 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.047 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.047 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.047 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.047 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-04-05T23:47:36.047 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4410, 2024-04-05T23:47:36.047 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 11.530271946, 2024-04-05T23:47:36.048 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002614574 2024-04-05T23:47:36.048 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.048 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:36.048 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:36.048 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:36.048 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.048 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.048 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.048 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 1, 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 2, 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 1, 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 1, 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 3365, 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 3365, 2024-04-05T23:47:36.049 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 768, 2024-04-05T23:47:36.050 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:36.050 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.050 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-04-05T23:47:36.050 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:36.050 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:36.050 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:36.050 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:36.050 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:36.051 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:36.051 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:36.051 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:36.051 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:36.051 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:36.051 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:36.051 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:36.051 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:36.051 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:36.052 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:36.052 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:36.052 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:36.052 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:36.052 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:36.052 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:36.052 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:36.052 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:36.054 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-04-05T23:47:36.054 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:36.054 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:36.054 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:36.054 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:36.054 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 292814, 2024-04-05T23:47:36.054 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 370, 2024-04-05T23:47:36.055 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-04-05T23:47:36.055 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-04-05T23:47:36.055 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-04-05T23:47:36.055 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-04-05T23:47:36.055 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:36.055 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:36.055 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:36.055 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:36.056 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:36.056 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 188, 2024-04-05T23:47:36.056 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:36.056 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:36.056 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:36.056 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-04-05T23:47:36.056 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 1385168, 2024-04-05T23:47:36.056 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 15613, 2024-04-05T23:47:36.056 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:36.057 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-04-05T23:47:36.057 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:36.057 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-04-05T23:47:36.057 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.057 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-04-05T23:47:36.057 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-04-05T23:47:36.057 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-04-05T23:47:36.057 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 16114, 2024-04-05T23:47:36.058 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 56619102, 2024-04-05T23:47:36.058 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 27, 2024-04-05T23:47:36.058 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 16114, 2024-04-05T23:47:36.058 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-04-05T23:47:36.058 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 16114, 2024-04-05T23:47:36.058 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 32.935410053, 2024-04-05T23:47:36.058 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002043900 2024-04-05T23:47:36.058 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.058 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-04-05T23:47:36.059 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-04-05T23:47:36.059 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 16114, 2024-04-05T23:47:36.059 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 16428 2024-04-05T23:47:36.059 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.059 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 16114, 2024-04-05T23:47:36.059 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 565, 2024-04-05T23:47:36.059 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 15549, 2024-04-05T23:47:36.059 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-04-05T23:47:36.060 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-04-05T23:47:36.060 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 0, 2024-04-05T23:47:36.060 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10, 2024-04-05T23:47:36.060 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 546, 2024-04-05T23:47:36.060 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 10633, 2024-04-05T23:47:36.060 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 335, 2024-04-05T23:47:36.060 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-04-05T23:47:36.060 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-04-05T23:47:36.061 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-04-05T23:47:36.061 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-04-05T23:47:36.061 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 4407, 2024-04-05T23:47:36.061 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-04-05T23:47:36.061 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:36.061 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:36.061 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 19, 2024-04-05T23:47:36.061 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 20, 2024-04-05T23:47:36.061 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:36.062 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:36.062 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:36.062 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-04-05T23:47:36.062 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-04-05T23:47:36.062 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-04-05T23:47:36.062 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:36.062 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-04-05T23:47:36.062 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:36.062 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-04-05T23:47:36.063 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-04-05T23:47:36.063 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-04-05T23:47:36.063 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-04-05T23:47:36.063 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-04-05T23:47:36.063 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-04-05T23:47:36.063 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-04-05T23:47:36.063 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-04-05T23:47:36.063 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-04-05T23:47:36.064 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-04-05T23:47:36.064 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-04-05T23:47:36.064 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-04-05T23:47:36.064 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-04-05T23:47:36.064 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-04-05T23:47:36.064 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-04-05T23:47:36.064 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-04-05T23:47:36.064 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-04-05T23:47:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 101, 2024-04-05T23:47:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-04-05T23:47:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 22, 2024-04-05T23:47:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 12, 2024-04-05T23:47:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 12, 2024-04-05T23:47:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-04-05T23:47:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-04-05T23:47:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 311, 2024-04-05T23:47:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 0, 2024-04-05T23:47:36.066 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 147 2024-04-05T23:47:36.066 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.066 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-04-05T23:47:36.066 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-04-05T23:47:36.066 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:36.066 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-04-05T23:47:36.066 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-04-05T23:47:36.066 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.067 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-04-05T23:47:36.067 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-04-05T23:47:36.067 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.067 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-04-05T23:47:36.067 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:36.067 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 6, 2024-04-05T23:47:36.067 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-04-05T23:47:36.067 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 6, 2024-04-05T23:47:36.067 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 4382, 2024-04-05T23:47:36.068 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:36.068 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.068 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:36.068 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:36.068 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-05T23:47:36.068 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:36.068 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 134913, 2024-04-05T23:47:36.068 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 34913334, 2024-04-05T23:47:36.068 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:36.069 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 134913, 2024-04-05T23:47:36.069 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:36.069 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:36.069 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 134913, 2024-04-05T23:47:36.069 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 34913334, 2024-04-05T23:47:36.069 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:36.069 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.069 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.070 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.070 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:36.070 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.070 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:36.070 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:36.070 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-05T23:47:36.070 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:36.070 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:36.071 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:36.071 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:36.071 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:36.071 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 16114, 2024-04-05T23:47:36.071 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 57063708, 2024-04-05T23:47:36.071 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 16114, 2024-04-05T23:47:36.071 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 57063708, 2024-04-05T23:47:36.071 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:36.071 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.072 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.072 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.072 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:36.072 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.072 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:36.072 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:36.072 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-04-05T23:47:36.072 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:36.072 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 16114, 2024-04-05T23:47:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 16114, 2024-04-05T23:47:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 16114, 2024-04-05T23:47:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 16114, 2024-04-05T23:47:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:36.074 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.074 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.074 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.074 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:36.074 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.074 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:36.074 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:36.074 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-05T23:47:36.075 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:36.075 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 4382, 2024-04-05T23:47:36.075 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 444606, 2024-04-05T23:47:36.075 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:36.075 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 4382, 2024-04-05T23:47:36.075 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:36.075 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:36.075 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 546, 2024-04-05T23:47:36.076 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 444606, 2024-04-05T23:47:36.076 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:36.076 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.076 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.076 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.076 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:36.076 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:36.076 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x561b541820a0": { 2024-04-05T23:47:36.077 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:36.077 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-05T23:47:36.077 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:36.077 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 31208, 2024-04-05T23:47:36.077 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 55802544, 2024-04-05T23:47:36.077 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:36.077 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 31208, 2024-04-05T23:47:36.077 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:36.077 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:36.078 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 10087, 2024-04-05T23:47:36.078 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 55802544, 2024-04-05T23:47:36.078 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:36.078 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:36.078 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:36.078 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:36.078 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:36.078 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:36.079 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-05T23:47:36.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:35 smithi130 ceph-mon[79417]: pgmap v442: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 159 KiB/s wr, 16 op/s 2024-04-05T23:47:36.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:35 smithi027 ceph-mon[78597]: pgmap v442: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 159 KiB/s wr, 16 op/s 2024-04-05T23:47:36.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:35 smithi016 ceph-mon[38248]: pgmap v442: 97 pgs: 97 active+clean; 505 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 159 KiB/s wr, 16 op/s 2024-04-05T23:47:36.393 INFO:tasks.check_counter:Daemon mds.i mds.exported=7 2024-04-05T23:47:36.393 INFO:tasks.check_counter:Daemon mds.i mds.imported=8 2024-04-05T23:47:36.394 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-05T23:47:36.394 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-05T23:47:36.394 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-05T23:47:36.394 DEBUG:tasks.check_counter:Getting stats from l 2024-04-05T23:47:36.394 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e shell --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 -- ceph daemon mds.l perf dump 2024-04-05T23:47:36.615 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/mon.c/config 2024-04-05T23:47:37.754 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-05T23:47:37.755 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-04-05T23:47:37.755 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 6393, 2024-04-05T23:47:37.755 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 6410, 2024-04-05T23:47:37.755 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 2322707, 2024-04-05T23:47:37.755 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 27702161, 2024-04-05T23:47:37.755 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 20, 2024-04-05T23:47:37.755 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 8, 2024-04-05T23:47:37.755 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.422239577, 2024-04-05T23:47:37.756 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.142596984, 2024-04-05T23:47:37.756 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.104972310, 2024-04-05T23:47:37.756 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.144424602, 2024-04-05T23:47:37.756 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:37.756 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 6406, 2024-04-05T23:47:37.756 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.179829885, 2024-04-05T23:47:37.756 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000028072 2024-04-05T23:47:37.756 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.757 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:37.757 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 326, 2024-04-05T23:47:37.757 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000055655, 2024-04-05T23:47:37.757 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000170 2024-04-05T23:47:37.757 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.757 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 5920, 2024-04-05T23:47:37.757 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 9536 2024-04-05T23:47:37.757 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.758 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-04-05T23:47:37.758 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 115325, 2024-04-05T23:47:37.758 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 119239, 2024-04-05T23:47:37.758 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 39375622, 2024-04-05T23:47:37.758 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 69919565, 2024-04-05T23:47:37.758 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 16, 2024-04-05T23:47:37.758 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 7, 2024-04-05T23:47:37.758 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 5.551016145, 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 2.259642607, 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 4.155730095, 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.048156434, 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 119239, 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2.541823272, 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000021317 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:37.759 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 216524, 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.032167776, 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000148 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 114288 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 5332, 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 5442, 2024-04-05T23:47:37.760 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 3431173, 2024-04-05T23:47:37.761 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 26550156, 2024-04-05T23:47:37.761 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 9, 2024-04-05T23:47:37.761 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 7, 2024-04-05T23:47:37.761 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.406622116, 2024-04-05T23:47:37.761 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.160581828, 2024-04-05T23:47:37.761 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.118552951, 2024-04-05T23:47:37.761 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.116223224, 2024-04-05T23:47:37.761 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-05T23:47:37.761 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 5441, 2024-04-05T23:47:37.762 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.184998653, 2024-04-05T23:47:37.762 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000034000 2024-04-05T23:47:37.762 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.762 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-05T23:47:37.762 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 165, 2024-04-05T23:47:37.762 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000029462, 2024-04-05T23:47:37.762 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000178 2024-04-05T23:47:37.763 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.763 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1951888, 2024-04-05T23:47:37.763 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 537216 2024-04-05T23:47:37.763 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.763 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-04-05T23:47:37.763 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-04-05T23:47:37.763 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-04-05T23:47:37.763 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.764 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-04-05T23:47:37.764 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-05T23:47:37.764 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-05T23:47:37.764 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 14345, 2024-04-05T23:47:37.764 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 8.213204412, 2024-04-05T23:47:37.764 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000572548 2024-04-05T23:47:37.764 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:37.765 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.765 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-04-05T23:47:37.765 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-05T23:47:37.765 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-05T23:47:37.765 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8940, 2024-04-05T23:47:37.765 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.429818503, 2024-04-05T23:47:37.765 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000048078 2024-04-05T23:47:37.765 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:37.766 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.766 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-04-05T23:47:37.766 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 28121, 2024-04-05T23:47:37.766 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 27945, 2024-04-05T23:47:37.766 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-04-05T23:47:37.766 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 27945, 2024-04-05T23:47:37.766 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 63.433442211, 2024-04-05T23:47:37.766 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002269938 2024-04-05T23:47:37.767 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.767 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-04-05T23:47:37.767 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 176, 2024-04-05T23:47:37.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 6, 2024-04-05T23:47:37.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-04-05T23:47:37.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 11, 2024-04-05T23:47:37.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-04-05T23:47:37.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-04-05T23:47:37.768 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 612, 2024-04-05T23:47:37.768 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 36, 2024-04-05T23:47:37.768 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 554, 2024-04-05T23:47:37.768 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 22, 2024-04-05T23:47:37.768 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 560, 2024-04-05T23:47:37.768 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 319, 2024-04-05T23:47:37.769 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 162, 2024-04-05T23:47:37.769 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 162, 2024-04-05T23:47:37.769 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 12, 2024-04-05T23:47:37.769 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 56418, 2024-04-05T23:47:37.769 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 30269, 2024-04-05T23:47:37.769 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 168, 2024-04-05T23:47:37.770 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 21, 2024-04-05T23:47:37.770 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 1, 2024-04-05T23:47:37.770 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-04-05T23:47:37.770 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 876, 2024-04-05T23:47:37.770 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 7467, 2024-04-05T23:47:37.771 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-04-05T23:47:37.771 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 9, 2024-04-05T23:47:37.771 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 272, 2024-04-05T23:47:37.771 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 16, 2024-04-05T23:47:37.771 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 488, 2024-04-05T23:47:37.771 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-04-05T23:47:37.771 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2024-04-05T23:47:37.771 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-04-05T23:47:37.771 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 614, 2024-04-05T23:47:37.772 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 211488726, 2024-04-05T23:47:37.772 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-04-05T23:47:37.772 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2024-04-05T23:47:37.772 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-04-05T23:47:37.772 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-04-05T23:47:37.772 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-04-05T23:47:37.773 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-04-05T23:47:37.773 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-04-05T23:47:37.773 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-04-05T23:47:37.773 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2024-04-05T23:47:37.773 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 2, 2024-04-05T23:47:37.773 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 2465, 2024-04-05T23:47:37.773 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 19467, 2024-04-05T23:47:37.774 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-04-05T23:47:37.774 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-05T23:47:37.774 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-05T23:47:37.774 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 9433, 2024-04-05T23:47:37.774 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 4109, 2024-04-05T23:47:37.774 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 1871, 2024-04-05T23:47:37.774 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 27188 2024-04-05T23:47:37.775 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.775 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-04-05T23:47:37.775 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-04-05T23:47:37.775 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 81, 2024-04-05T23:47:37.775 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 42, 2024-04-05T23:47:37.775 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 101, 2024-04-05T23:47:37.775 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 6, 2024-04-05T23:47:37.776 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 12, 2024-04-05T23:47:37.776 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-04-05T23:47:37.776 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-04-05T23:47:37.776 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 4395, 2024-04-05T23:47:37.776 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 4383, 2024-04-05T23:47:37.776 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-04-05T23:47:37.777 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-04-05T23:47:37.777 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-04-05T23:47:37.777 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-04-05T23:47:37.777 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-04-05T23:47:37.777 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-04-05T23:47:37.777 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-04-05T23:47:37.777 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-04-05T23:47:37.777 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 14, 2024-04-05T23:47:37.778 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-04-05T23:47:37.778 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-04-05T23:47:37.778 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-04-05T23:47:37.778 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-04-05T23:47:37.778 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.778 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-04-05T23:47:37.779 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 29996, 2024-04-05T23:47:37.779 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-04-05T23:47:37.779 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-04-05T23:47:37.779 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 29996, 2024-04-05T23:47:37.779 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-04-05T23:47:37.779 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-04-05T23:47:37.780 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 36, 2024-04-05T23:47:37.780 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-04-05T23:47:37.780 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-04-05T23:47:37.780 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 36, 2024-04-05T23:47:37.780 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-04-05T23:47:37.781 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-04-05T23:47:37.781 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-04-05T23:47:37.781 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 57453792, 2024-04-05T23:47:37.781 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-04-05T23:47:37.781 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-04-05T23:47:37.781 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 7200, 2024-04-05T23:47:37.781 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 15.000700108, 2024-04-05T23:47:37.782 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002083430 2024-04-05T23:47:37.782 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.782 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-04-05T23:47:37.782 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.782 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-04-05T23:47:37.782 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 387, 2024-04-05T23:47:37.782 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 5046, 2024-04-05T23:47:37.782 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 4659, 2024-04-05T23:47:37.783 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 69, 2024-04-05T23:47:37.783 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 121, 2024-04-05T23:47:37.783 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 52, 2024-04-05T23:47:37.783 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 612, 2024-04-05T23:47:37.783 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 6699, 2024-04-05T23:47:37.783 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 6087, 2024-04-05T23:47:37.783 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 160, 2024-04-05T23:47:37.783 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 4925, 2024-04-05T23:47:37.783 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 4765, 2024-04-05T23:47:37.784 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 63356, 2024-04-05T23:47:37.784 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 198964 2024-04-05T23:47:37.784 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.784 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-04-05T23:47:37.784 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 47951, 2024-04-05T23:47:37.784 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 8, 2024-04-05T23:47:37.784 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 28121, 2024-04-05T23:47:37.784 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 937, 2024-04-05T23:47:37.785 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 5595, 2024-04-05T23:47:37.785 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-04-05T23:47:37.785 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4385, 2024-04-05T23:47:37.785 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 32.919784394, 2024-04-05T23:47:37.785 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.007507362 2024-04-05T23:47:37.785 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.786 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-04-05T23:47:37.786 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 9, 2024-04-05T23:47:37.786 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.030072515, 2024-04-05T23:47:37.786 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.003341390 2024-04-05T23:47:37.786 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.786 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-04-05T23:47:37.786 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.787 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.787 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.787 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.787 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-04-05T23:47:37.787 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.787 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.787 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.787 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.787 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-04-05T23:47:37.788 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 5824, 2024-04-05T23:47:37.788 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 12.900548225, 2024-04-05T23:47:37.788 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002215066 2024-04-05T23:47:37.788 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.788 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-04-05T23:47:37.788 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.788 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.788 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.788 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.789 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-04-05T23:47:37.789 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.789 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.789 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.789 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.789 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-04-05T23:47:37.789 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.789 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.789 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.790 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.790 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-04-05T23:47:37.790 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.790 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.790 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.790 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.790 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-04-05T23:47:37.790 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.791 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.791 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.791 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.791 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-04-05T23:47:37.791 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.791 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.791 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.791 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.791 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-04-05T23:47:37.792 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.792 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.792 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.792 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.792 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-04-05T23:47:37.792 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.792 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.792 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.793 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.794 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.794 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-04-05T23:47:37.794 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 11736, 2024-04-05T23:47:37.794 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 8.455407974, 2024-04-05T23:47:37.794 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000720467 2024-04-05T23:47:37.794 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.794 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-04-05T23:47:37.794 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 908, 2024-04-05T23:47:37.795 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2.182662479, 2024-04-05T23:47:37.795 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002403813 2024-04-05T23:47:37.795 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.795 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-04-05T23:47:37.795 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.795 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.795 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.795 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.795 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-04-05T23:47:37.796 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.796 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.796 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.796 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.796 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-04-05T23:47:37.796 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.796 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.796 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.797 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.797 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-04-05T23:47:37.797 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.797 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.797 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.797 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.797 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-04-05T23:47:37.797 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.797 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.798 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.798 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.798 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-04-05T23:47:37.798 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.798 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.798 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.798 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.798 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-04-05T23:47:37.799 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.799 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.799 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.799 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.799 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-04-05T23:47:37.799 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 688, 2024-04-05T23:47:37.799 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.246620859, 2024-04-05T23:47:37.799 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000358460 2024-04-05T23:47:37.799 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.800 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-04-05T23:47:37.800 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.800 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.800 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.800 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.800 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-04-05T23:47:37.800 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.800 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.800 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.801 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.801 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-04-05T23:47:37.801 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.801 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.801 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.801 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.801 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-04-05T23:47:37.801 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4395, 2024-04-05T23:47:37.801 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 6.698345765, 2024-04-05T23:47:37.802 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001524083 2024-04-05T23:47:37.802 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.802 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-04-05T23:47:37.802 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-04-05T23:47:37.802 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-04-05T23:47:37.802 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.802 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.802 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.803 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:37.803 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.803 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-04-05T23:47:37.803 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 2, 2024-04-05T23:47:37.803 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 4, 2024-04-05T23:47:37.803 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 2, 2024-04-05T23:47:37.803 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 2, 2024-04-05T23:47:37.803 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-04-05T23:47:37.803 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 3715, 2024-04-05T23:47:37.804 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 1857, 2024-04-05T23:47:37.804 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 823, 2024-04-05T23:47:37.804 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-04-05T23:47:37.804 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.804 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-04-05T23:47:37.804 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-04-05T23:47:37.804 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-04-05T23:47:37.804 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-04-05T23:47:37.804 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-04-05T23:47:37.805 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-04-05T23:47:37.805 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-04-05T23:47:37.805 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-05T23:47:37.805 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-04-05T23:47:37.805 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-05T23:47:37.805 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-04-05T23:47:37.805 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-04-05T23:47:37.805 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-04-05T23:47:37.805 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-05T23:47:37.806 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-04-05T23:47:37.806 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-04-05T23:47:37.806 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-04-05T23:47:37.806 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-04-05T23:47:37.806 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-04-05T23:47:37.806 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-05T23:47:37.806 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-04-05T23:47:37.806 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-05T23:47:37.807 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-04-05T23:47:37.807 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-04-05T23:47:37.807 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-04-05T23:47:37.807 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-04-05T23:47:37.807 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-04-05T23:47:37.807 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-05T23:47:37.808 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-04-05T23:47:37.808 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-04-05T23:47:37.808 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-04-05T23:47:37.808 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-04-05T23:47:37.808 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-04-05T23:47:37.808 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-04-05T23:47:37.808 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-04-05T23:47:37.808 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-04-05T23:47:37.809 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-04-05T23:47:37.809 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 329226, 2024-04-05T23:47:37.809 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 426, 2024-04-05T23:47:37.809 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 440, 2024-04-05T23:47:37.809 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 5, 2024-04-05T23:47:37.809 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-04-05T23:47:37.809 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-04-05T23:47:37.809 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-04-05T23:47:37.810 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-04-05T23:47:37.810 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-04-05T23:47:37.810 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-04-05T23:47:37.810 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8256, 2024-04-05T23:47:37.810 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 188, 2024-04-05T23:47:37.810 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-04-05T23:47:37.811 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-04-05T23:47:37.811 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-04-05T23:47:37.811 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-04-05T23:47:37.811 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 1581584, 2024-04-05T23:47:37.811 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 15163, 2024-04-05T23:47:37.811 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-04-05T23:47:37.811 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-04-05T23:47:37.811 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-04-05T23:47:37.811 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-04-05T23:47:37.812 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.812 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-04-05T23:47:37.812 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-04-05T23:47:37.812 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-04-05T23:47:37.812 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 13104, 2024-04-05T23:47:37.812 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 54058405, 2024-04-05T23:47:37.813 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 27, 2024-04-05T23:47:37.813 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 13104, 2024-04-05T23:47:37.813 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-04-05T23:47:37.813 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 13104, 2024-04-05T23:47:37.813 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 27.939500191, 2024-04-05T23:47:37.813 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002132135 2024-04-05T23:47:37.813 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.814 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-04-05T23:47:37.814 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-04-05T23:47:37.814 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 13104, 2024-04-05T23:47:37.814 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 13414 2024-04-05T23:47:37.814 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.814 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 13104, 2024-04-05T23:47:37.814 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 511, 2024-04-05T23:47:37.815 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 12593, 2024-04-05T23:47:37.815 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-04-05T23:47:37.815 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-04-05T23:47:37.815 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 0, 2024-04-05T23:47:37.815 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10, 2024-04-05T23:47:37.815 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 505, 2024-04-05T23:47:37.815 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 7705, 2024-04-05T23:47:37.816 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 313, 2024-04-05T23:47:37.816 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-04-05T23:47:37.816 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-04-05T23:47:37.816 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-04-05T23:47:37.816 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-04-05T23:47:37.816 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 4407, 2024-04-05T23:47:37.816 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-04-05T23:47:37.817 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-04-05T23:47:37.817 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-04-05T23:47:37.817 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 2, 2024-04-05T23:47:37.817 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 20, 2024-04-05T23:47:37.817 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-04-05T23:47:37.817 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-04-05T23:47:37.817 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-04-05T23:47:37.817 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-04-05T23:47:37.818 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-04-05T23:47:37.818 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-04-05T23:47:37.818 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-04-05T23:47:37.818 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-04-05T23:47:37.818 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-04-05T23:47:37.818 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-04-05T23:47:37.819 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-04-05T23:47:37.819 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-04-05T23:47:37.819 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-04-05T23:47:37.819 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-04-05T23:47:37.819 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-04-05T23:47:37.820 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-04-05T23:47:37.820 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-04-05T23:47:37.820 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-04-05T23:47:37.820 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-04-05T23:47:37.820 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-04-05T23:47:37.820 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-04-05T23:47:37.820 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-04-05T23:47:37.821 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-04-05T23:47:37.821 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-04-05T23:47:37.821 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-04-05T23:47:37.821 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-04-05T23:47:37.821 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 101, 2024-04-05T23:47:37.821 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-04-05T23:47:37.822 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 22, 2024-04-05T23:47:37.822 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 12, 2024-04-05T23:47:37.822 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 12, 2024-04-05T23:47:37.823 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-04-05T23:47:37.823 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-04-05T23:47:37.823 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 299, 2024-04-05T23:47:37.823 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 12, 2024-04-05T23:47:37.823 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 141 2024-04-05T23:47:37.823 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.823 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-04-05T23:47:37.823 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-04-05T23:47:37.824 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-04-05T23:47:37.824 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-04-05T23:47:37.824 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-04-05T23:47:37.824 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.824 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-04-05T23:47:37.824 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-04-05T23:47:37.824 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.824 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-04-05T23:47:37.825 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-04-05T23:47:37.825 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 6, 2024-04-05T23:47:37.825 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-04-05T23:47:37.825 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 6, 2024-04-05T23:47:37.825 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 4383, 2024-04-05T23:47:37.825 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-04-05T23:47:37.825 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.825 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-05T23:47:37.826 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:37.826 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-05T23:47:37.826 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:37.826 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 127046, 2024-04-05T23:47:37.826 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 33672700, 2024-04-05T23:47:37.826 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:37.826 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 127046, 2024-04-05T23:47:37.826 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 127046, 2024-04-05T23:47:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 33672700, 2024-04-05T23:47:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.828 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-04-05T23:47:37.828 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:37.828 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-05T23:47:37.828 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:37.828 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:37.828 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:37.828 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:37.828 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:37.828 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 13104, 2024-04-05T23:47:37.829 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 54502712, 2024-04-05T23:47:37.829 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 13104, 2024-04-05T23:47:37.829 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 54502712, 2024-04-05T23:47:37.829 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:37.829 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.829 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.829 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.829 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:37.829 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.830 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-04-05T23:47:37.830 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:37.830 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-04-05T23:47:37.830 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:37.830 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-05T23:47:37.830 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-05T23:47:37.830 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:37.830 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-05T23:47:37.831 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 13104, 2024-04-05T23:47:37.831 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 13104, 2024-04-05T23:47:37.831 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 13104, 2024-04-05T23:47:37.831 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 13104, 2024-04-05T23:47:37.831 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:37.831 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.831 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.831 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.831 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:37.832 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.832 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-04-05T23:47:37.832 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-05T23:47:37.832 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-05T23:47:37.832 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:37.832 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 4383, 2024-04-05T23:47:37.832 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 444403, 2024-04-05T23:47:37.832 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:37.832 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 4383, 2024-04-05T23:47:37.833 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:37.833 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:37.833 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 505, 2024-04-05T23:47:37.833 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 444403, 2024-04-05T23:47:37.833 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:37.833 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.833 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.833 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.833 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:37.834 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-05T23:47:37.834 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x5574cd2960a0": { 2024-04-05T23:47:37.834 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 18550, 2024-04-05T23:47:37.834 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-05T23:47:37.834 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-05T23:47:37.834 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 29996, 2024-04-05T23:47:37.834 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 53259488, 2024-04-05T23:47:37.834 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-05T23:47:37.835 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 29996, 2024-04-05T23:47:37.835 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-05T23:47:37.835 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-05T23:47:37.835 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 7200, 2024-04-05T23:47:37.835 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 53240938, 2024-04-05T23:47:37.835 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-05T23:47:37.835 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-05T23:47:37.835 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-05T23:47:37.835 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-05T23:47:37.836 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:37.836 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-05T23:47:37.836 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-05T23:47:38.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:37 smithi130 ceph-mon[79417]: pgmap v443: 97 pgs: 97 active+clean; 506 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 233 KiB/s wr, 23 op/s 2024-04-05T23:47:38.122 INFO:tasks.check_counter:Daemon mds.l mds.exported=9 2024-04-05T23:47:38.123 INFO:tasks.check_counter:Daemon mds.l mds.imported=16 2024-04-05T23:47:38.123 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=6 2024-04-05T23:47:38.123 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-05T23:47:38.123 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-05T23:47:38.123 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-05T23:47:38.136 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-05T23:47:38.136 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:47:38.136 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T23:47:38.189 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:47:38.190 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T23:47:38.214 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:37 smithi027 ceph-mon[78597]: pgmap v443: 97 pgs: 97 active+clean; 506 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 233 KiB/s wr, 23 op/s 2024-04-05T23:47:38.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:47:38.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:37 smithi016 ceph-mon[38248]: pgmap v443: 97 pgs: 97 active+clean; 506 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 233 KiB/s wr, 23 op/s 2024-04-05T23:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.488+0000 7f28aaa00700 1 -- 172.21.15.16:0/3185011054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a41060b0 msgr2=0x7f28a410dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.488+0000 7f28aaa00700 1 --2- 172.21.15.16:0/3185011054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a41060b0 0x7f28a410dc30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f289c00b0a0 tx=0x7f289c00b3b0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.488+0000 7f28aaa00700 1 -- 172.21.15.16:0/3185011054 shutdown_connections 2024-04-05T23:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.488+0000 7f28aaa00700 1 --2- 172.21.15.16:0/3185011054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a41060b0 0x7f28a410dc30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.488+0000 7f28aaa00700 1 --2- 172.21.15.16:0/3185011054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28a4100f90 0x7f28a4101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.488+0000 7f28aaa00700 1 --2- 172.21.15.16:0/3185011054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a4101940 0x7f28a4105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.488+0000 7f28aaa00700 1 -- 172.21.15.16:0/3185011054 >> 172.21.15.16:0/3185011054 conn(0x7f28a40fc830 msgr2=0x7f28a40fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.488+0000 7f28aaa00700 1 -- 172.21.15.16:0/3185011054 shutdown_connections 2024-04-05T23:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.488+0000 7f28aaa00700 1 -- 172.21.15.16:0/3185011054 wait complete. 2024-04-05T23:47:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.489+0000 7f28aaa00700 1 Processor -- start 2024-04-05T23:47:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.489+0000 7f28aaa00700 1 -- start start 2024-04-05T23:47:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28aaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a4100f90 0x7f28a4199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28aaa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28a41060b0 0x7f28a4199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28aaa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a419de50 0x7f28a419e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28aaa00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a419e8d0 con 0x7f28a419de50 2024-04-05T23:47:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28aaa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a419ea40 con 0x7f28a4100f90 2024-04-05T23:47:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28aaa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a419ec00 con 0x7f28a41060b0 2024-04-05T23:47:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28a8a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a419de50 0x7f28a419e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28a3400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28a41060b0 0x7f28a4199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28a8a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a419de50 0x7f28a419e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59206/0 (socket says 172.21.15.16:59206) 2024-04-05T23:47:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28a8a00700 1 -- 172.21.15.16:0/2975545638 learned_addr learned my addr 172.21.15.16:0/2975545638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.490+0000 7f28a3e00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a4100f90 0x7f28a4199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28a8a00700 1 -- 172.21.15.16:0/2975545638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28a41060b0 msgr2=0x7f28a4199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28a8a00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28a41060b0 0x7f28a4199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28a8a00700 1 -- 172.21.15.16:0/2975545638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a4100f90 msgr2=0x7f28a4199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28a8a00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a4100f90 0x7f28a4199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28a8a00700 1 -- 172.21.15.16:0/2975545638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c009d00 con 0x7f28a419de50 2024-04-05T23:47:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28a8a00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a419de50 0x7f28a419e2d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f289c00b070 tx=0x7f289c007cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28a0c00700 1 -- 172.21.15.16:0/2975545638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c00e040 con 0x7f28a419de50 2024-04-05T23:47:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28a0c00700 1 -- 172.21.15.16:0/2975545638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f289c00bdc0 con 0x7f28a419de50 2024-04-05T23:47:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a41a6100 con 0x7f28a419de50 2024-04-05T23:47:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28a0c00700 1 -- 172.21.15.16:0/2975545638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c0052a0 con 0x7f28a419de50 2024-04-05T23:47:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.491+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a41a64d0 con 0x7f28a419de50 2024-04-05T23:47:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.492+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a4102460 con 0x7f28a419de50 2024-04-05T23:47:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.494+0000 7f28a0c00700 1 -- 172.21.15.16:0/2975545638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f289c01d480 con 0x7f28a419de50 2024-04-05T23:47:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.496+0000 7f28a0c00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28880a1290 0x7f28880a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.496+0000 7f28a0c00700 1 -- 172.21.15.16:0/2975545638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f289c01de40 con 0x7f28a419de50 2024-04-05T23:47:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.496+0000 7f28a3e00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28880a1290 0x7f28880a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.496+0000 7f28a3e00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28880a1290 0x7f28880a3750 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2894009c60 tx=0x7f2894009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.498+0000 7f28a0c00700 1 -- 172.21.15.16:0/2975545638 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f289c097480 con 0x7f28a419de50 2024-04-05T23:47:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.720+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28a4067b50 con 0x7f28a419de50 2024-04-05T23:47:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.722+0000 7f28a0c00700 1 -- 172.21.15.16:0/2975545638 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+23052 (secure 0 0 0) 0x7f289c09aaa0 con 0x7f28a419de50 2024-04-05T23:47:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:47:38.714 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:47:14.602437+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/985211078":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1740984099":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/4157043736":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/1565223355":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2846240830":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/4184690359":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/1565223355":"2024-04-06T23:33:34.593227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":23,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:47:38.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.724+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28880a1290 msgr2=0x7f28880a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:38.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.724+0000 7f28aaa00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28880a1290 0x7f28880a3750 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2894009c60 tx=0x7f2894009260 comp rx=0 tx=0).stop 2024-04-05T23:47:38.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a419de50 msgr2=0x7f28a419e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:38.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a419de50 0x7f28a419e2d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f289c00b070 tx=0x7f289c007cf0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 shutdown_connections 2024-04-05T23:47:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f28880a1290 0x7f28880a3750 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28a419de50 0x7f28a419e2d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a4100f90 0x7f28a4199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 --2- 172.21.15.16:0/2975545638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28a41060b0 0x7f28a4199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 >> 172.21.15.16:0/2975545638 conn(0x7f28a40fc830 msgr2=0x7f28a40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 shutdown_connections 2024-04-05T23:47:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.725+0000 7f28aaa00700 1 -- 172.21.15.16:0/2975545638 wait complete. 2024-04-05T23:47:38.735 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T23:28:33.862433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-05T23:32:02.641673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-05T23:32:03.972521+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 100, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T23:47:38.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T23:47:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.950+0000 7f9f70e00700 1 -- 172.21.15.16:0/704013724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 msgr2=0x7f9f6c1014a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.950+0000 7f9f70e00700 1 --2- 172.21.15.16:0/704013724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 0x7f9f6c1014a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c0098f0 tx=0x7f9f5c009c00 comp rx=0 tx=0).stop 2024-04-05T23:47:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.950+0000 7f9f70e00700 1 -- 172.21.15.16:0/704013724 shutdown_connections 2024-04-05T23:47:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.950+0000 7f9f70e00700 1 --2- 172.21.15.16:0/704013724 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f6c1061e0 0x7f9f6c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.950+0000 7f9f70e00700 1 --2- 172.21.15.16:0/704013724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 0x7f9f6c1014a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.950+0000 7f9f70e00700 1 --2- 172.21.15.16:0/704013724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c101a70 0x7f9f6c105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.950+0000 7f9f70e00700 1 -- 172.21.15.16:0/704013724 >> 172.21.15.16:0/704013724 conn(0x7f9f6c0fc920 msgr2=0x7f9f6c0fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.950+0000 7f9f70e00700 1 -- 172.21.15.16:0/704013724 shutdown_connections 2024-04-05T23:47:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.950+0000 7f9f70e00700 1 -- 172.21.15.16:0/704013724 wait complete. 2024-04-05T23:47:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.951+0000 7f9f70e00700 1 Processor -- start 2024-04-05T23:47:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.951+0000 7f9f70e00700 1 -- start start 2024-04-05T23:47:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.951+0000 7f9f70e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 0x7f9f6c197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.951+0000 7f9f70e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c1061e0 0x7f9f6c197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.951+0000 7f9f70e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f6c19bc10 0x7f9f6c19c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.951+0000 7f9f70e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c19c690 con 0x7f9f6c1061e0 2024-04-05T23:47:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.951+0000 7f9f70e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c19c800 con 0x7f9f6c1010c0 2024-04-05T23:47:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.951+0000 7f9f70e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c19c9c0 con 0x7f9f6c19bc10 2024-04-05T23:47:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c1061e0 0x7f9f6c197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 0x7f9f6c197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c1061e0 0x7f9f6c197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59230/0 (socket says 172.21.15.16:59230) 2024-04-05T23:47:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 0x7f9f6c197330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51390/0 (socket says 172.21.15.16:51390) 2024-04-05T23:47:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69400700 1 -- 172.21.15.16:0/2093192238 learned_addr learned my addr 172.21.15.16:0/2093192238 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f6a800700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f6c19bc10 0x7f9f6c19c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69e00700 1 -- 172.21.15.16:0/2093192238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f6c19bc10 msgr2=0x7f9f6c19c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69e00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f6c19bc10 0x7f9f6c19c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69e00700 1 -- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c1061e0 msgr2=0x7f9f6c197870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69e00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c1061e0 0x7f9f6c197870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69e00700 1 -- 172.21.15.16:0/2093192238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c009520 con 0x7f9f6c1010c0 2024-04-05T23:47:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69400700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c1061e0 0x7f9f6c197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:47:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.952+0000 7f9f69e00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 0x7f9f6c197330 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009fd0 tx=0x7f9f5c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.953+0000 7f9f62a00700 1 -- 172.21.15.16:0/2093192238 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c01c070 con 0x7f9f6c1010c0 2024-04-05T23:47:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.953+0000 7f9f62a00700 1 -- 172.21.15.16:0/2093192238 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f5c00bb20 con 0x7f9f6c1010c0 2024-04-05T23:47:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.953+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c1a3ec0 con 0x7f9f6c1010c0 2024-04-05T23:47:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.953+0000 7f9f62a00700 1 -- 172.21.15.16:0/2093192238 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c0038e0 con 0x7f9f6c1010c0 2024-04-05T23:47:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.953+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c1a42e0 con 0x7f9f6c1010c0 2024-04-05T23:47:38.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.954+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c109400 con 0x7f9f6c1010c0 2024-04-05T23:47:38.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.956+0000 7f9f62a00700 1 -- 172.21.15.16:0/2093192238 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f9f5c020030 con 0x7f9f6c1010c0 2024-04-05T23:47:38.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.957+0000 7f9f62a00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9f500a1290 0x7f9f500a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:38.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.957+0000 7f9f62a00700 1 -- 172.21.15.16:0/2093192238 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f9f5c0cdf20 con 0x7f9f6c1010c0 2024-04-05T23:47:38.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.958+0000 7f9f69400700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9f500a1290 0x7f9f500a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:38.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.958+0000 7f9f69400700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9f500a1290 0x7f9f500a3750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c198880 tx=0x7f9f58009310 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:38.960+0000 7f9f62a00700 1 -- 172.21.15.16:0/2093192238 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9f5c097380 con 0x7f9f6c1010c0 2024-04-05T23:47:39.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:38 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2975545638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:47:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.179+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f6c198630 con 0x7f9f6c1010c0 2024-04-05T23:47:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.179+0000 7f9f62a00700 1 -- 172.21.15.16:0/2093192238 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f5c09a9a0 con 0x7f9f6c1010c0 2024-04-05T23:47:39.171 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-05T23:47:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9f500a1290 msgr2=0x7f9f500a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9f500a1290 0x7f9f500a3750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c198880 tx=0x7f9f58009310 comp rx=0 tx=0).stop 2024-04-05T23:47:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 msgr2=0x7f9f6c197330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 0x7f9f6c197330 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009fd0 tx=0x7f9f5c00fa00 comp rx=0 tx=0).stop 2024-04-05T23:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 shutdown_connections 2024-04-05T23:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f9f500a1290 0x7f9f500a3750 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f6c19bc10 0x7f9f6c19c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f6c1010c0 0x7f9f6c197330 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 --2- 172.21.15.16:0/2093192238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f6c1061e0 0x7f9f6c197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 >> 172.21.15.16:0/2093192238 conn(0x7f9f6c0fc920 msgr2=0x7f9f6c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 shutdown_connections 2024-04-05T23:47:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.182+0000 7f9f70e00700 1 -- 172.21.15.16:0/2093192238 wait complete. 2024-04-05T23:47:39.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-05T23:47:39.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:38 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2975545638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:47:39.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:38 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2975545638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:47:39.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.406+0000 7fddb6c00700 1 -- 172.21.15.16:0/242625111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddb01061c0 msgr2=0x7fddb010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.406+0000 7fddb6c00700 1 --2- 172.21.15.16:0/242625111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddb01061c0 0x7fddb010dd40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdda800b0a0 tx=0x7fdda800b3b0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.406+0000 7fddb6c00700 1 -- 172.21.15.16:0/242625111 shutdown_connections 2024-04-05T23:47:39.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.406+0000 7fddb6c00700 1 --2- 172.21.15.16:0/242625111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddb01061c0 0x7fddb010dd40 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.406+0000 7fddb6c00700 1 --2- 172.21.15.16:0/242625111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb01010a0 0x7fddb0101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.406+0000 7fddb6c00700 1 --2- 172.21.15.16:0/242625111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddb0101a50 0x7fddb0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.406+0000 7fddb6c00700 1 -- 172.21.15.16:0/242625111 >> 172.21.15.16:0/242625111 conn(0x7fddb00fc900 msgr2=0x7fddb00fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:39.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.406+0000 7fddb6c00700 1 -- 172.21.15.16:0/242625111 shutdown_connections 2024-04-05T23:47:39.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.406+0000 7fddb6c00700 1 -- 172.21.15.16:0/242625111 wait complete. 2024-04-05T23:47:39.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.407+0000 7fddb6c00700 1 Processor -- start 2024-04-05T23:47:39.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.407+0000 7fddb6c00700 1 -- start start 2024-04-05T23:47:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.407+0000 7fddb6c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddb01010a0 0x7fddb00731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.407+0000 7fddb6c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb01061c0 0x7fddb0073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.407+0000 7fddb6c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddb0073c60 0x7fddb01a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.407+0000 7fddb6c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb01a5c30 con 0x7fddb0073c60 2024-04-05T23:47:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.407+0000 7fddb6c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb01a5da0 con 0x7fddb01061c0 2024-04-05T23:47:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.407+0000 7fddb6c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb01a5f60 con 0x7fddb01010a0 2024-04-05T23:47:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddb4c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddb0073c60 0x7fddb01a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddaf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb01061c0 0x7fddb0073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddaf400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb01061c0 0x7fddb0073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51402/0 (socket says 172.21.15.16:51402) 2024-04-05T23:47:39.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddaf400700 1 -- 172.21.15.16:0/2551890949 learned_addr learned my addr 172.21.15.16:0/2551890949 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:39.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddafe00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddb01010a0 0x7fddb00731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:39.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddb4c00700 1 -- 172.21.15.16:0/2551890949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddb01010a0 msgr2=0x7fddb00731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddb4c00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddb01010a0 0x7fddb00731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddb4c00700 1 -- 172.21.15.16:0/2551890949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb01061c0 msgr2=0x7fddb0073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddb4c00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb01061c0 0x7fddb0073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddb4c00700 1 -- 172.21.15.16:0/2551890949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda8009d00 con 0x7fddb0073c60 2024-04-05T23:47:39.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddaf400700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb01061c0 0x7fddb0073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:47:39.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddb4c00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddb0073c60 0x7fddb01a5660 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdda800b070 tx=0x7fdda800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:39.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.408+0000 7fddacc00700 1 -- 172.21.15.16:0/2551890949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda800e040 con 0x7fddb0073c60 2024-04-05T23:47:39.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.409+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb01a61e0 con 0x7fddb0073c60 2024-04-05T23:47:39.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.409+0000 7fddacc00700 1 -- 172.21.15.16:0/2551890949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdda801d480 con 0x7fddb0073c60 2024-04-05T23:47:39.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.409+0000 7fddacc00700 1 -- 172.21.15.16:0/2551890949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda80052a0 con 0x7fddb0073c60 2024-04-05T23:47:39.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.409+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb01a6600 con 0x7fddb0073c60 2024-04-05T23:47:39.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.410+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb0109490 con 0x7fddb0073c60 2024-04-05T23:47:39.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.412+0000 7fddacc00700 1 -- 172.21.15.16:0/2551890949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fdda801d5f0 con 0x7fddb0073c60 2024-04-05T23:47:39.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.413+0000 7fddacc00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdd940a1310 0x7fdd940a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.413+0000 7fddacc00700 1 -- 172.21.15.16:0/2551890949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdda80cdf20 con 0x7fddb0073c60 2024-04-05T23:47:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.413+0000 7fddafe00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdd940a1310 0x7fdd940a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.414+0000 7fddafe00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdd940a1310 0x7fdd940a37d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009c60 tx=0x7fdda0009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:39.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.415+0000 7fddacc00700 1 -- 172.21.15.16:0/2551890949 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdda8097400 con 0x7fddb0073c60 2024-04-05T23:47:39.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.631+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fddb00746c0 con 0x7fddb0073c60 2024-04-05T23:47:39.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.631+0000 7fddacc00700 1 -- 172.21.15.16:0/2551890949 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdda809aa20 con 0x7fddb0073c60 2024-04-05T23:47:39.623 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-05T23:47:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdd940a1310 msgr2=0x7fdd940a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdd940a1310 0x7fdd940a37d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009c60 tx=0x7fdda0009260 comp rx=0 tx=0).stop 2024-04-05T23:47:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddb0073c60 msgr2=0x7fddb01a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddb0073c60 0x7fddb01a5660 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdda800b070 tx=0x7fdda800bcf0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 shutdown_connections 2024-04-05T23:47:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdd940a1310 0x7fdd940a37d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddb0073c60 0x7fddb01a5660 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddb01010a0 0x7fddb00731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 --2- 172.21.15.16:0/2551890949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb01061c0 0x7fddb0073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 >> 172.21.15.16:0/2551890949 conn(0x7fddb00fc900 msgr2=0x7fddb00fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 shutdown_connections 2024-04-05T23:47:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.634+0000 7fddb6c00700 1 -- 172.21.15.16:0/2551890949 wait complete. 2024-04-05T23:47:39.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-05T23:47:39.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.855+0000 7f34a6c00700 1 -- 172.21.15.16:0/3883706866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01010c0 msgr2=0x7f34a01014a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.855+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3883706866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01010c0 0x7f34a01014a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3490009990 tx=0x7f3490009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.855+0000 7f34a6c00700 1 -- 172.21.15.16:0/3883706866 shutdown_connections 2024-04-05T23:47:39.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.855+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3883706866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01010c0 0x7f34a01014a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.855+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3883706866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a01061e0 0x7f34a010dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.855+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3883706866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34a0101a70 0x7f34a0105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.855+0000 7f34a6c00700 1 -- 172.21.15.16:0/3883706866 >> 172.21.15.16:0/3883706866 conn(0x7f34a00fc920 msgr2=0x7f34a00fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:39.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.855+0000 7f34a6c00700 1 -- 172.21.15.16:0/3883706866 shutdown_connections 2024-04-05T23:47:39.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.855+0000 7f34a6c00700 1 -- 172.21.15.16:0/3883706866 wait complete. 2024-04-05T23:47:39.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.856+0000 7f34a6c00700 1 Processor -- start 2024-04-05T23:47:39.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.856+0000 7f34a6c00700 1 -- start start 2024-04-05T23:47:39.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.856+0000 7f34a6c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34a01010c0 0x7f34a0199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:39.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.856+0000 7f34a6c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01061e0 0x7f34a0199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:39.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.856+0000 7f34a6c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a019de20 0x7f34a019e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:39.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.856+0000 7f34a6c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a019e8a0 con 0x7f34a01061e0 2024-04-05T23:47:39.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.856+0000 7f34a6c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a019ea10 con 0x7f34a019de20 2024-04-05T23:47:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.856+0000 7f34a6c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a019ebd0 con 0x7f34a01010c0 2024-04-05T23:47:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01061e0 0x7f34a0199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f34a4c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a019de20 0x7f34a019e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01061e0 0x7f34a0199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59270/0 (socket says 172.21.15.16:59270) 2024-04-05T23:47:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349f400700 1 -- 172.21.15.16:0/3662120146 learned_addr learned my addr 172.21.15.16:0/3662120146 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349fe00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34a01010c0 0x7f34a0199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349f400700 1 -- 172.21.15.16:0/3662120146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34a01010c0 msgr2=0x7f34a0199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349f400700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34a01010c0 0x7f34a0199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349f400700 1 -- 172.21.15.16:0/3662120146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a019de20 msgr2=0x7f34a019e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:39.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349f400700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a019de20 0x7f34a019e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:39.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349f400700 1 -- 172.21.15.16:0/3662120146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34900095f0 con 0x7f34a01061e0 2024-04-05T23:47:39.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f34a4c00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a019de20 0x7f34a019e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:47:39.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.857+0000 7f349f400700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01061e0 0x7f34a0199ad0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f348c00d950 tx=0x7f348c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:39.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.858+0000 7f349cc00700 1 -- 172.21.15.16:0/3662120146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c004220 con 0x7f34a01061e0 2024-04-05T23:47:39.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.858+0000 7f349cc00700 1 -- 172.21.15.16:0/3662120146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f348c007620 con 0x7f34a01061e0 2024-04-05T23:47:39.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.858+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a01a6130 con 0x7f34a01061e0 2024-04-05T23:47:39.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.858+0000 7f349cc00700 1 -- 172.21.15.16:0/3662120146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c0056c0 con 0x7f34a01061e0 2024-04-05T23:47:39.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.858+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a01a6660 con 0x7f34a01061e0 2024-04-05T23:47:39.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.859+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a0102590 con 0x7f34a01061e0 2024-04-05T23:47:39.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.861+0000 7f349cc00700 1 -- 172.21.15.16:0/3662120146 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f348c004380 con 0x7f34a01061e0 2024-04-05T23:47:39.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.863+0000 7f349cc00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34840a1310 0x7f34840a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:39.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.863+0000 7f349cc00700 1 -- 172.21.15.16:0/3662120146 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f348c0ccec0 con 0x7f34a01061e0 2024-04-05T23:47:39.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.863+0000 7f349fe00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34840a1310 0x7f34840a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:39.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.864+0000 7f349fe00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34840a1310 0x7f34840a37d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f349000b2e0 tx=0x7f3490005fd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:39.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:39.866+0000 7f349cc00700 1 -- 172.21.15.16:0/3662120146 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f348c096290 con 0x7f34a01061e0 2024-04-05T23:47:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.085+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34a0067c30 con 0x7f34a01061e0 2024-04-05T23:47:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.086+0000 7f349cc00700 1 -- 172.21.15.16:0/3662120146 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+11 (secure 0 0 0) 0x7f348c016020 con 0x7f34a01061e0 2024-04-05T23:47:40.077 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-05T23:47:40.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34840a1310 msgr2=0x7f34840a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:40.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34840a1310 0x7f34840a37d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f349000b2e0 tx=0x7f3490005fd0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01061e0 msgr2=0x7f34a0199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01061e0 0x7f34a0199ad0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f348c00d950 tx=0x7f348c00dc60 comp rx=0 tx=0).stop 2024-04-05T23:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 shutdown_connections 2024-04-05T23:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f34840a1310 0x7f34840a37d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a019de20 0x7f34a019e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34a01010c0 0x7f34a0199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 --2- 172.21.15.16:0/3662120146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a01061e0 0x7f34a0199ad0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 >> 172.21.15.16:0/3662120146 conn(0x7f34a00fc920 msgr2=0x7f34a00fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 shutdown_connections 2024-04-05T23:47:40.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.088+0000 7f34a6c00700 1 -- 172.21.15.16:0/3662120146 wait complete. 2024-04-05T23:47:40.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-05T23:47:40.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:39 smithi016 ceph-mon[38248]: pgmap v444: 97 pgs: 97 active+clean; 506 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 89 KiB/s wr, 14 op/s 2024-04-05T23:47:40.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:39 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2093192238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:47:40.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:39 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/2551890949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:47:40.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:39 smithi027 ceph-mon[78597]: pgmap v444: 97 pgs: 97 active+clean; 506 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 89 KiB/s wr, 14 op/s 2024-04-05T23:47:40.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:39 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2093192238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:47:40.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:39 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/2551890949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:47:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.308+0000 7fdec0400700 1 -- 172.21.15.16:0/741993666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 msgr2=0x7fdeb8101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.308+0000 7fdec0400700 1 --2- 172.21.15.16:0/741993666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 0x7fdeb8101480 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009990 tx=0x7fdeac009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.308+0000 7fdec0400700 1 -- 172.21.15.16:0/741993666 shutdown_connections 2024-04-05T23:47:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.308+0000 7fdec0400700 1 --2- 172.21.15.16:0/741993666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 0x7fdeb8101480 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.308+0000 7fdec0400700 1 --2- 172.21.15.16:0/741993666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdeb81061c0 0x7fdeb810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.308+0000 7fdec0400700 1 --2- 172.21.15.16:0/741993666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeb8101a50 0x7fdeb8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.308+0000 7fdec0400700 1 -- 172.21.15.16:0/741993666 >> 172.21.15.16:0/741993666 conn(0x7fdeb80fc900 msgr2=0x7fdeb80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.308+0000 7fdec0400700 1 -- 172.21.15.16:0/741993666 shutdown_connections 2024-04-05T23:47:40.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.308+0000 7fdec0400700 1 -- 172.21.15.16:0/741993666 wait complete. 2024-04-05T23:47:40.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.309+0000 7fdec0400700 1 Processor -- start 2024-04-05T23:47:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.309+0000 7fdec0400700 1 -- start start 2024-04-05T23:47:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdec0400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 0x7fdeb81973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdec0400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdeb81061c0 0x7fdeb8197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdec0400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeb819bc60 0x7fdeb819c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdec0400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb819c6e0 con 0x7fdeb81010a0 2024-04-05T23:47:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdec0400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb819c850 con 0x7fdeb819bc60 2024-04-05T23:47:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdec0400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb819ca10 con 0x7fdeb81061c0 2024-04-05T23:47:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebda00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 0x7fdeb81973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:40.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebda00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 0x7fdeb81973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59288/0 (socket says 172.21.15.16:59288) 2024-04-05T23:47:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebda00700 1 -- 172.21.15.16:0/3239587858 learned_addr learned my addr 172.21.15.16:0/3239587858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebe400700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeb819bc60 0x7fdeb819c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebd000700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdeb81061c0 0x7fdeb8197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebda00700 1 -- 172.21.15.16:0/3239587858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdeb81061c0 msgr2=0x7fdeb8197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebda00700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdeb81061c0 0x7fdeb8197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebda00700 1 -- 172.21.15.16:0/3239587858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeb819bc60 msgr2=0x7fdeb819c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebda00700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeb819bc60 0x7fdeb819c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.310+0000 7fdebda00700 1 -- 172.21.15.16:0/3239587858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac0095f0 con 0x7fdeb81010a0 2024-04-05T23:47:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.311+0000 7fdebda00700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 0x7fdeb81973d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fdeac000c00 tx=0x7fdeac005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.311+0000 7fdeb6000700 1 -- 172.21.15.16:0/3239587858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac01d070 con 0x7fdeb81010a0 2024-04-05T23:47:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.311+0000 7fdeb6000700 1 -- 172.21.15.16:0/3239587858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdeac022470 con 0x7fdeb81010a0 2024-04-05T23:47:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.311+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb81a3f10 con 0x7fdeb81010a0 2024-04-05T23:47:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.311+0000 7fdeb6000700 1 -- 172.21.15.16:0/3239587858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac017650 con 0x7fdeb81010a0 2024-04-05T23:47:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.311+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb81a42d0 con 0x7fdeb81010a0 2024-04-05T23:47:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.312+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb8109430 con 0x7fdeb81010a0 2024-04-05T23:47:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.314+0000 7fdeb6000700 1 -- 172.21.15.16:0/3239587858 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fdeac0225e0 con 0x7fdeb81010a0 2024-04-05T23:47:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.315+0000 7fdeb6000700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdea00a1290 0x7fdea00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.315+0000 7fdeb6000700 1 -- 172.21.15.16:0/3239587858 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fdeac0cdfa0 con 0x7fdeb81010a0 2024-04-05T23:47:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.316+0000 7fdebd000700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdea00a1290 0x7fdea00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.316+0000 7fdebd000700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdea00a1290 0x7fdea00a3750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdeb81988d0 tx=0x7fdea4009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:40.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.319+0000 7fdeb6000700 1 -- 172.21.15.16:0/3239587858 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdeac097520 con 0x7fdeb81010a0 2024-04-05T23:47:40.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:39 smithi130 ceph-mon[79417]: pgmap v444: 97 pgs: 97 active+clean; 506 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 89 KiB/s wr, 14 op/s 2024-04-05T23:47:40.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:39 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2093192238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T23:47:40.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:39 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/2551890949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-05T23:47:40.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.609+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdeb8102570 con 0x7fdeb81010a0 2024-04-05T23:47:40.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.609+0000 7fdeb6000700 1 -- 172.21.15.16:0/3239587858 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v101) v1 ==== 81+0+1002 (secure 0 0 0) 0x7fdeac027080 con 0x7fdeb81010a0 2024-04-05T23:47:40.601 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/985211078 2024-04-06T23:33:34.593227+0000 2024-04-05T23:47:40.601 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/3856233973 2024-04-06T23:26:11.600470+0000 2024-04-05T23:47:40.601 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/3856233973 2024-04-06T23:26:11.600470+0000 2024-04-05T23:47:40.601 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1740984099 2024-04-06T23:33:34.593227+0000 2024-04-05T23:47:40.601 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1696925921 2024-04-06T23:26:11.600470+0000 2024-04-05T23:47:40.601 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4157043736 2024-04-06T23:33:34.593227+0000 2024-04-05T23:47:40.601 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2307977786 2024-04-06T23:26:11.600470+0000 2024-04-05T23:47:40.601 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1141331215 2024-04-06T23:26:37.227308+0000 2024-04-05T23:47:40.602 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1106968003 2024-04-06T23:26:11.600470+0000 2024-04-05T23:47:40.602 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1565223355 2024-04-06T23:33:34.593227+0000 2024-04-05T23:47:40.602 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3668421167 2024-04-06T23:26:37.227308+0000 2024-04-05T23:47:40.602 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/2429779560 2024-04-06T23:26:37.227308+0000 2024-04-05T23:47:40.602 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/2429779560 2024-04-06T23:26:37.227308+0000 2024-04-05T23:47:40.602 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/749035377 2024-04-06T23:26:37.227308+0000 2024-04-05T23:47:40.602 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2846240830 2024-04-06T23:33:34.593227+0000 2024-04-05T23:47:40.602 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4184690359 2024-04-06T23:33:34.593227+0000 2024-04-05T23:47:40.603 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1565223355 2024-04-06T23:33:34.593227+0000 2024-04-05T23:47:40.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdea00a1290 msgr2=0x7fdea00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:40.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdea00a1290 0x7fdea00a3750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdeb81988d0 tx=0x7fdea4009260 comp rx=0 tx=0).stop 2024-04-05T23:47:40.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 msgr2=0x7fdeb81973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:40.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 0x7fdeb81973d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fdeac000c00 tx=0x7fdeac005740 comp rx=0 tx=0).stop 2024-04-05T23:47:40.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 shutdown_connections 2024-04-05T23:47:40.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fdea00a1290 0x7fdea00a3750 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb81010a0 0x7fdeb81973d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdeb819bc60 0x7fdeb819c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 --2- 172.21.15.16:0/3239587858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdeb81061c0 0x7fdeb8197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:40.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 >> 172.21.15.16:0/3239587858 conn(0x7fdeb80fc900 msgr2=0x7fdeb80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:40.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 shutdown_connections 2024-04-05T23:47:40.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:40.612+0000 7fdec0400700 1 -- 172.21.15.16:0/3239587858 wait complete. 2024-04-05T23:47:40.605 INFO:teuthology.orchestra.run.smithi016.stderr:listed 17 entries 2024-04-05T23:47:40.621 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-05T23:47:40.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:47:40.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-05T23:47:40.908 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:47:40.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:47:40.909 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-05T23:47:41.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:47:41.010 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-05T23:47:41.098 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-05T23:47:41.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:47:41.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-05T23:47:41.182 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:40 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3662120146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:47:41.183 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:40 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3239587858' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T23:47:41.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:47:41.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-05T23:47:41.214 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:40 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3662120146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:47:41.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:40 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3239587858' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T23:47:41.312 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-05T23:47:41.313 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-05T23:47:41.313 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T23:47:41.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:40 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3662120146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-05T23:47:41.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:40 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3239587858' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T23:47:41.357 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348768k,nr_inodes=4087192,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/8bbe8b0007b5fb90a3f1c9d44fbb9ae8704104090ad1eed5885907de5b1e9184/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/8bbe8b0007b5fb90a3f1c9d44fbb9ae8704104090ad1eed5885907de5b1e9184/diff,workdir=/var/lib/containers/storage/overlay/8bbe8b0007b5fb90a3f1c9d44fbb9ae8704104090ad1eed5885907de5b1e9184/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b65ef9912f52ceec0406d3d0de0e3c7c1c731122c626e38702fc300c97da94b2/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/b65ef9912f52ceec0406d3d0de0e3c7c1c731122c626e38702fc300c97da94b2/diff,workdir=/var/lib/containers/storage/overlay/b65ef9912f52ceec0406d3d0de0e3c7c1c731122c626e38702fc300c97da94b2/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3276008k,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 overlay /var/lib/containers/storage/overlay/7924bb5970613ce4939d36b2f41bd4315896724b998727405e225a6d2f36db35/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/7924bb5970613ce4939d36b2f41bd4315896724b998727405e225a6d2f36db35/diff,workdir=/var/lib/containers/storage/overlay/7924bb5970613ce4939d36b2f41bd4315896724b998727405e225a6d2f36db35/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4a9e4cd01fc47b69bddf7ac0913aaf679a9e3ce592066779e03674654ae1ffd7/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/4a9e4cd01fc47b69bddf7ac0913aaf679a9e3ce592066779e03674654ae1ffd7/diff,workdir=/var/lib/containers/storage/overlay/4a9e4cd01fc47b69bddf7ac0913aaf679a9e3ce592066779e03674654ae1ffd7/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d8dd1efb24661f359e75119ef9134211420e14b152644e18b6089b2ca6faf66f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/d8dd1efb24661f359e75119ef9134211420e14b152644e18b6089b2ca6faf66f/diff,workdir=/var/lib/containers/storage/overlay/d8dd1efb24661f359e75119ef9134211420e14b152644e18b6089b2ca6faf66f/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ab2a298806bb9b15a12db27f557c1b07638f8067a532413037e801bf4bedc86c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/ab2a298806bb9b15a12db27f557c1b07638f8067a532413037e801bf4bedc86c/diff,workdir=/var/lib/containers/storage/overlay/ab2a298806bb9b15a12db27f557c1b07638f8067a532413037e801bf4bedc86c/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/99a213b01938e31b323a57cbca5ef24c85ea7eb5306d6e015273350f3729b555/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/99a213b01938e31b323a57cbca5ef24c85ea7eb5306d6e015273350f3729b555/diff,workdir=/var/lib/containers/storage/overlay/99a213b01938e31b323a57cbca5ef24c85ea7eb5306d6e015273350f3729b555/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9c7f2fc62c9f749cbb7582ffd62b27fbdf9f7ec613010008c253d398b3295c23/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/9c7f2fc62c9f749cbb7582ffd62b27fbdf9f7ec613010008c253d398b3295c23/diff,workdir=/var/lib/containers/storage/overlay/9c7f2fc62c9f749cbb7582ffd62b27fbdf9f7ec613010008c253d398b3295c23/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b98418835f27fa27ce2d639732454851406a6ff5d835e7e711e765cf3369e1c4/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/b98418835f27fa27ce2d639732454851406a6ff5d835e7e711e765cf3369e1c4/diff,workdir=/var/lib/containers/storage/overlay/b98418835f27fa27ce2d639732454851406a6ff5d835e7e711e765cf3369e1c4/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/bcff74ad983ad7d01060b34c2a23260b5a69840e6b2b377bb2710ccf959e6203/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7KRBOSJE2EZV3FRKSP6B22GWIS:/var/lib/containers/storage/overlay/l/A7WHDVHVT4GFIO7UNBENFCSPJH,upperdir=/var/lib/containers/storage/overlay/bcff74ad983ad7d01060b34c2a23260b5a69840e6b2b377bb2710ccf959e6203/diff,workdir=/var/lib/containers/storage/overlay/bcff74ad983ad7d01060b34c2a23260b5a69840e6b2b377bb2710ccf959e6203/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 2024-04-05T23:47:41.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:47:41.358 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-05T23:47:41.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T23:47:41.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-05T23:47:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:Cannot find device "ceph-brx" 2024-04-05T23:47:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:47:41.557 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-05T23:47:41.571 INFO:tasks.cephadm:Teardown begin 2024-04-05T23:47:41.572 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-05T23:47:41.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:47:41.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.791+0000 7f7343400700 1 -- 172.21.15.16:0/2809017312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733c100f90 msgr2=0x7f733c101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.791+0000 7f7343400700 1 --2- 172.21.15.16:0/2809017312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733c100f90 0x7f733c101370 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7330009990 tx=0x7f7330009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.791+0000 7f7343400700 1 -- 172.21.15.16:0/2809017312 shutdown_connections 2024-04-05T23:47:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.791+0000 7f7343400700 1 --2- 172.21.15.16:0/2809017312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733c100f90 0x7f733c101370 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.791+0000 7f7343400700 1 --2- 172.21.15.16:0/2809017312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f733c1060b0 0x7f733c10dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.791+0000 7f7343400700 1 --2- 172.21.15.16:0/2809017312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f733c101940 0x7f733c105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.791+0000 7f7343400700 1 -- 172.21.15.16:0/2809017312 >> 172.21.15.16:0/2809017312 conn(0x7f733c0fc830 msgr2=0x7f733c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.791+0000 7f7343400700 1 -- 172.21.15.16:0/2809017312 shutdown_connections 2024-04-05T23:47:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.791+0000 7f7343400700 1 -- 172.21.15.16:0/2809017312 wait complete. 2024-04-05T23:47:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.792+0000 7f7343400700 1 Processor -- start 2024-04-05T23:47:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.792+0000 7f7343400700 1 -- start start 2024-04-05T23:47:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7343400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f733c100f90 0x7f733c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7343400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733c1060b0 0x7f733c073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7343400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f733c073c60 0x7f733c1a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7343400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1a5a80 con 0x7f733c1060b0 2024-04-05T23:47:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7343400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1a5bc0 con 0x7f733c100f90 2024-04-05T23:47:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7343400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1a5d80 con 0x7f733c073c60 2024-04-05T23:47:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f733be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733c1060b0 0x7f733c073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7341400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f733c073c60 0x7f733c1a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7340a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f733c100f90 0x7f733c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7340a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f733c100f90 0x7f733c0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51448/0 (socket says 172.21.15.16:51448) 2024-04-05T23:47:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f733be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733c1060b0 0x7f733c073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59298/0 (socket says 172.21.15.16:59298) 2024-04-05T23:47:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7340a00700 1 -- 172.21.15.16:0/1612988602 learned_addr learned my addr 172.21.15.16:0/1612988602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7341400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f733c073c60 0x7f733c1a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.16:46184/0 (socket says 172.21.15.16:46184) 2024-04-05T23:47:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7340a00700 1 -- 172.21.15.16:0/1612988602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f733c073c60 msgr2=0x7f733c1a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7340a00700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f733c073c60 0x7f733c1a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7340a00700 1 -- 172.21.15.16:0/1612988602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733c1060b0 msgr2=0x7f733c073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7340a00700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733c1060b0 0x7f733c073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.793+0000 7f7340a00700 1 -- 172.21.15.16:0/1612988602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7334009c40 con 0x7f733c100f90 2024-04-05T23:47:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.794+0000 7f7340a00700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f733c100f90 0x7f733c0731e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7330005910 tx=0x7f7330004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.794+0000 7f7339600700 1 -- 172.21.15.16:0/1612988602 <== mon.2 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733001d070 con 0x7f733c100f90 2024-04-05T23:47:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.794+0000 7f7339600700 1 -- 172.21.15.16:0/1612988602 <== mon.2 v2:172.21.15.27:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7330022470 con 0x7f733c100f90 2024-04-05T23:47:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.794+0000 7f7339600700 1 -- 172.21.15.16:0/1612988602 <== mon.2 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330017650 con 0x7f733c100f90 2024-04-05T23:47:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.794+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73300095f0 con 0x7f733c100f90 2024-04-05T23:47:41.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.794+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c1a6240 con 0x7f733c100f90 2024-04-05T23:47:41.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.795+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c1092d0 con 0x7f733c100f90 2024-04-05T23:47:41.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.797+0000 7f7339600700 1 -- 172.21.15.16:0/1612988602 <== mon.2 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f73300225e0 con 0x7f733c100f90 2024-04-05T23:47:41.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.798+0000 7f7339600700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73240a1290 0x7f73240a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:41.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.798+0000 7f7339600700 1 -- 172.21.15.16:0/1612988602 <== mon.2 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f73300cde20 con 0x7f733c100f90 2024-04-05T23:47:41.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.799+0000 7f733be00700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73240a1290 0x7f73240a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:41.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.799+0000 7f733be00700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73240a1290 0x7f73240a3750 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f733c0748c0 tx=0x7f7328009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:41.802+0000 7f7339600700 1 -- 172.21.15.16:0/1612988602 <== mon.2 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f733005dab0 con 0x7f733c100f90 2024-04-05T23:47:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.069+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f733c102460 con 0x7f733c100f90 2024-04-05T23:47:42.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.070+0000 7f7339600700 1 -- 172.21.15.16:0/1612988602 <== mon.2 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+10928 (secure 0 0 0) 0x7f7330097280 con 0x7f733c100f90 2024-04-05T23:47:42.061 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:47:42.062 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":83,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:46:50.909992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":50,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26296":{"gid":26296,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/2605400145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":2605400145},{"type":"v1","addr":"172.21.15.130:6835","nonce":2605400145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:47:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.072+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73240a1290 msgr2=0x7f73240a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.072+0000 7f7343400700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73240a1290 0x7f73240a3750 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f733c0748c0 tx=0x7f7328009260 comp rx=0 tx=0).stop 2024-04-05T23:47:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.072+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f733c100f90 msgr2=0x7f733c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.072+0000 7f7343400700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f733c100f90 0x7f733c0731e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7330005910 tx=0x7f7330004730 comp rx=0 tx=0).stop 2024-04-05T23:47:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.073+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 shutdown_connections 2024-04-05T23:47:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.073+0000 7f7343400700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f73240a1290 0x7f73240a3750 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.073+0000 7f7343400700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f733c073c60 0x7f733c1a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.073+0000 7f7343400700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f733c100f90 0x7f733c0731e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.073+0000 7f7343400700 1 --2- 172.21.15.16:0/1612988602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733c1060b0 0x7f733c073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.073+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 >> 172.21.15.16:0/1612988602 conn(0x7f733c0fc830 msgr2=0x7f733c0fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.073+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 shutdown_connections 2024-04-05T23:47:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.073+0000 7f7343400700 1 -- 172.21.15.16:0/1612988602 wait complete. 2024-04-05T23:47:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2024-04-05T23:47:42.088 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T23:47:42.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:41 smithi027 ceph-mon[78597]: pgmap v445: 97 pgs: 97 active+clean; 514 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 892 KiB/s wr, 39 op/s 2024-04-05T23:47:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.299+0000 7fac7cc00700 1 -- 172.21.15.16:0/1735555900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac78100fc0 msgr2=0x7fac781013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.299+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1735555900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac78100fc0 0x7fac781013a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fac680098f0 tx=0x7fac68009c00 comp rx=0 tx=0).stop 2024-04-05T23:47:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.299+0000 7fac7cc00700 1 -- 172.21.15.16:0/1735555900 shutdown_connections 2024-04-05T23:47:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.299+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1735555900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac781060e0 0x7fac7810dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.299+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1735555900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac78100fc0 0x7fac781013a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.299+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1735555900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac78101970 0x7fac781059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.299+0000 7fac7cc00700 1 -- 172.21.15.16:0/1735555900 >> 172.21.15.16:0/1735555900 conn(0x7fac780fc860 msgr2=0x7fac780fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.299+0000 7fac7cc00700 1 -- 172.21.15.16:0/1735555900 shutdown_connections 2024-04-05T23:47:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.299+0000 7fac7cc00700 1 -- 172.21.15.16:0/1735555900 wait complete. 2024-04-05T23:47:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.300+0000 7fac7cc00700 1 Processor -- start 2024-04-05T23:47:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac7cc00700 1 -- start start 2024-04-05T23:47:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac7cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac78100fc0 0x7fac78199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac7cc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac781060e0 0x7fac78199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac7cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7819de00 0x7fac7819e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac7cc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7819e880 con 0x7fac7819de00 2024-04-05T23:47:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac7cc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7819e9f0 con 0x7fac78100fc0 2024-04-05T23:47:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac7cc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7819ebb0 con 0x7fac781060e0 2024-04-05T23:47:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac76800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7819de00 0x7fac7819e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac76800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7819de00 0x7fac7819e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51286/0 (socket says 172.21.15.16:51286) 2024-04-05T23:47:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac76800700 1 -- 172.21.15.16:0/1016530582 learned_addr learned my addr 172.21.15.16:0/1016530582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac75e00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac78100fc0 0x7fac78199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.301+0000 7fac75400700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac781060e0 0x7fac78199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac76800700 1 -- 172.21.15.16:0/1016530582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac781060e0 msgr2=0x7fac78199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac76800700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac781060e0 0x7fac78199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac76800700 1 -- 172.21.15.16:0/1016530582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac78100fc0 msgr2=0x7fac78199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac76800700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac78100fc0 0x7fac78199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac76800700 1 -- 172.21.15.16:0/1016530582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac70009d00 con 0x7fac7819de00 2024-04-05T23:47:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac75e00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac78100fc0 0x7fac78199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:47:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac76800700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7819de00 0x7fac7819e280 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fac7000e3c0 tx=0x7fac7000e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac6ea00700 1 -- 172.21.15.16:0/1016530582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac700090d0 con 0x7fac7819de00 2024-04-05T23:47:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac68009520 con 0x7fac7819de00 2024-04-05T23:47:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac6ea00700 1 -- 172.21.15.16:0/1016530582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac7000f040 con 0x7fac7819de00 2024-04-05T23:47:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac6ea00700 1 -- 172.21.15.16:0/1016530582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac70007c20 con 0x7fac7819de00 2024-04-05T23:47:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.302+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac781a64b0 con 0x7fac7819de00 2024-04-05T23:47:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.303+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac78102490 con 0x7fac7819de00 2024-04-05T23:47:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.305+0000 7fac6ea00700 1 -- 172.21.15.16:0/1016530582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fac70003a00 con 0x7fac7819de00 2024-04-05T23:47:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.306+0000 7fac6ea00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fac5c0a1290 0x7fac5c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.307+0000 7fac6ea00700 1 -- 172.21.15.16:0/1016530582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8556+0+0 (secure 0 0 0) 0x7fac700ccc60 con 0x7fac7819de00 2024-04-05T23:47:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.307+0000 7fac75e00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fac5c0a1290 0x7fac5c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.307+0000 7fac75e00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fac5c0a1290 0x7fac5c0a3750 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fac680098c0 tx=0x7fac68019040 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.310+0000 7fac6ea00700 1 -- 172.21.15.16:0/1016530582 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fac700960c0 con 0x7fac7819de00 2024-04-05T23:47:42.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:41 smithi016 ceph-mon[38248]: pgmap v445: 97 pgs: 97 active+clean; 514 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 892 KiB/s wr, 39 op/s 2024-04-05T23:47:42.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:41 smithi130 ceph-mon[79417]: pgmap v445: 97 pgs: 97 active+clean; 514 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 892 KiB/s wr, 39 op/s 2024-04-05T23:47:42.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.577+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac7819a8c0 con 0x7fac7819de00 2024-04-05T23:47:42.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.578+0000 7fac6ea00700 1 -- 172.21.15.16:0/1016530582 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+10928 (secure 0 0 0) 0x7fac700996e0 con 0x7fac7819de00 2024-04-05T23:47:42.570 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:47:42.571 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1930674650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1930674650},{"type":"v1","addr":"172.21.15.16:6835","nonce":1930674650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14517,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/226142538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":226142538},{"type":"v1","addr":"172.21.15.16:6839","nonce":226142538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":83,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T23:32:04.511660+0000","modified":"2024-04-05T23:46:50.909992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24566,"mds_1":24550,"mds_2":24526,"mds_3":24476,"mds_4":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14529":{"gid":14529,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/2579712996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2579712996},{"type":"v1","addr":"172.21.15.16:6841","nonce":2579712996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6833/1368666940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1368666940},{"type":"v1","addr":"172.21.15.27:6833","nonce":1368666940}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.130:6833/477825490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":477825490},{"type":"v1","addr":"172.21.15.130:6833","nonce":477825490}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"d","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.16:6837/2898043660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2898043660},{"type":"v1","addr":"172.21.15.16:6837","nonce":2898043660}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6835/686908192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":686908192},{"type":"v1","addr":"172.21.15.27:6835","nonce":686908192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.27:6837/541968377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":541968377},{"type":"v1","addr":"172.21.15.27:6837","nonce":541968377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":50,"addr":"172.21.15.130:6837/3873050281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3873050281},{"type":"v1","addr":"172.21.15.130:6837","nonce":3873050281}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6839/2970827239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2970827239},{"type":"v1","addr":"172.21.15.130:6839","nonce":2970827239}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6839/491598025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":491598025},{"type":"v1","addr":"172.21.15.27:6839","nonce":491598025}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26296":{"gid":26296,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/2605400145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":2605400145},{"type":"v1","addr":"172.21.15.130:6835","nonce":2605400145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-05T23:47:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fac5c0a1290 msgr2=0x7fac5c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fac5c0a1290 0x7fac5c0a3750 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fac680098c0 tx=0x7fac68019040 comp rx=0 tx=0).stop 2024-04-05T23:47:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7819de00 msgr2=0x7fac7819e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7819de00 0x7fac7819e280 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fac7000e3c0 tx=0x7fac7000e6d0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 shutdown_connections 2024-04-05T23:47:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fac5c0a1290 0x7fac5c0a3750 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac7819de00 0x7fac7819e280 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac78100fc0 0x7fac78199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 --2- 172.21.15.16:0/1016530582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac781060e0 0x7fac78199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 >> 172.21.15.16:0/1016530582 conn(0x7fac780fc860 msgr2=0x7fac780fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 shutdown_connections 2024-04-05T23:47:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.580+0000 7fac7cc00700 1 -- 172.21.15.16:0/1016530582 wait complete. 2024-04-05T23:47:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2024-04-05T23:47:42.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T23:47:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.799+0000 7f6a4cc00700 1 -- 172.21.15.16:0/1793323653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a4810b300 msgr2=0x7f6a4810d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.799+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/1793323653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a4810b300 0x7f6a4810d7c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6a4000b0a0 tx=0x7f6a4000b3b0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.799+0000 7f6a4cc00700 1 -- 172.21.15.16:0/1793323653 shutdown_connections 2024-04-05T23:47:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.799+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/1793323653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a4810b300 0x7f6a4810d7c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.799+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/1793323653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a48101ee0 0x7f6a481022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.799+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/1793323653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a48102800 0x7f6a4810ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.799+0000 7f6a4cc00700 1 -- 172.21.15.16:0/1793323653 >> 172.21.15.16:0/1793323653 conn(0x7f6a480fb720 msgr2=0x7f6a480fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.799+0000 7f6a4cc00700 1 -- 172.21.15.16:0/1793323653 shutdown_connections 2024-04-05T23:47:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.799+0000 7f6a4cc00700 1 -- 172.21.15.16:0/1793323653 wait complete. 2024-04-05T23:47:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.800+0000 7f6a4cc00700 1 Processor -- start 2024-04-05T23:47:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.800+0000 7f6a4cc00700 1 -- start start 2024-04-05T23:47:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a4cc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a48101ee0 0x7f6a48197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a4cc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48102800 0x7f6a48197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a4cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4810b300 0x7f6a4819bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a4cc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4819c230 con 0x7f6a48101ee0 2024-04-05T23:47:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a4cc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4819c3a0 con 0x7f6a4810b300 2024-04-05T23:47:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a4cc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4819c560 con 0x7f6a48102800 2024-04-05T23:47:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a45e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a48101ee0 0x7f6a48197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a45e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a48101ee0 0x7f6a48197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51310/0 (socket says 172.21.15.16:51310) 2024-04-05T23:47:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a45e00700 1 -- 172.21.15.16:0/721128687 learned_addr learned my addr 172.21.15.16:0/721128687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:42.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a46800700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4810b300 0x7f6a4819bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:42.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.801+0000 7f6a45400700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48102800 0x7f6a48197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:42.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a45e00700 1 -- 172.21.15.16:0/721128687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48102800 msgr2=0x7f6a48197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a45e00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48102800 0x7f6a48197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a45e00700 1 -- 172.21.15.16:0/721128687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4810b300 msgr2=0x7f6a4819bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:42.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a45e00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4810b300 0x7f6a4819bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:42.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a45e00700 1 -- 172.21.15.16:0/721128687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a34009520 con 0x7f6a48101ee0 2024-04-05T23:47:42.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a46800700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4810b300 0x7f6a4819bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:47:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a45e00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a48101ee0 0x7f6a48197310 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6a3800e9a0 tx=0x7f6a3800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a3ea00700 1 -- 172.21.15.16:0/721128687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a38009790 con 0x7f6a48101ee0 2024-04-05T23:47:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a3ea00700 1 -- 172.21.15.16:0/721128687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a38010430 con 0x7f6a48101ee0 2024-04-05T23:47:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a3ea00700 1 -- 172.21.15.16:0/721128687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a38018680 con 0x7f6a48101ee0 2024-04-05T23:47:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a40009d00 con 0x7f6a48101ee0 2024-04-05T23:47:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.802+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a481a3ea0 con 0x7f6a48101ee0 2024-04-05T23:47:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.803+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a48106390 con 0x7f6a48101ee0 2024-04-05T23:47:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.805+0000 7f6a3ea00700 1 -- 172.21.15.16:0/721128687 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f6a380105a0 con 0x7f6a48101ee0 2024-04-05T23:47:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.807+0000 7f6a3ea00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a2c0a13e0 0x7f6a2c0a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.807+0000 7f6a3ea00700 1 -- 172.21.15.16:0/721128687 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8556+0+0 (secure 0 0 0) 0x7f6a38014070 con 0x7f6a48101ee0 2024-04-05T23:47:42.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.807+0000 7f6a45400700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a2c0a13e0 0x7f6a2c0a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:42.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.807+0000 7f6a45400700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a2c0a13e0 0x7f6a2c0a38a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6a48198860 tx=0x7f6a34009260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:42.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:42.810+0000 7f6a3ea00700 1 -- 172.21.15.16:0/721128687 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6a38052ab0 con 0x7f6a48101ee0 2024-04-05T23:47:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.031+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a481986b0 con 0x7f6a48101ee0 2024-04-05T23:47:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.033+0000 7f6a3ea00700 1 -- 172.21.15.16:0/721128687 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+23052 (secure 0 0 0) 0x7f6a38096200 con 0x7f6a48101ee0 2024-04-05T23:47:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-05T23:47:43.025 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"fsid":"b3471b30-f3a3-11ee-b649-cb9ed24678a4","created":"2024-04-05T23:25:35.231049+0000","modified":"2024-04-05T23:47:14.602437+0000","last_up_change":"2024-04-05T23:31:25.213564+0000","last_in_change":"2024-04-05T23:31:08.887435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T23:28:33.862433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-05T23:32:02.641673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-05T23:32:03.972521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5fd073ed-664a-4b48-912a-3a2b1708c3a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6803","nonce":2345860669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6805","nonce":2345860669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6809","nonce":2345860669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2345860669},{"type":"v1","addr":"172.21.15.16:6807","nonce":2345860669}]},"public_addr":"172.21.15.16:6803/2345860669","cluster_addr":"172.21.15.16:6805/2345860669","heartbeat_back_addr":"172.21.15.16:6809/2345860669","heartbeat_front_addr":"172.21.15.16:6807/2345860669","state":["exists","up"]},{"osd":1,"uuid":"20981429-c950-4821-abbc-06a9d380c25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6801","nonce":1426362237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6803","nonce":1426362237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6807","nonce":1426362237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1426362237},{"type":"v1","addr":"172.21.15.27:6805","nonce":1426362237}]},"public_addr":"172.21.15.27:6801/1426362237","cluster_addr":"172.21.15.27:6803/1426362237","heartbeat_back_addr":"172.21.15.27:6807/1426362237","heartbeat_front_addr":"172.21.15.27:6805/1426362237","state":["exists","up"]},{"osd":2,"uuid":"794cd3d0-bb75-41a5-9ea9-e946ddea82b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6801","nonce":3222862878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6803","nonce":3222862878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6807","nonce":3222862878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3222862878},{"type":"v1","addr":"172.21.15.130:6805","nonce":3222862878}]},"public_addr":"172.21.15.130:6801/3222862878","cluster_addr":"172.21.15.130:6803/3222862878","heartbeat_back_addr":"172.21.15.130:6807/3222862878","heartbeat_front_addr":"172.21.15.130:6805/3222862878","state":["exists","up"]},{"osd":3,"uuid":"3ef270b8-eac6-4738-82f3-7aa26c536370","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6811","nonce":1156660397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6813","nonce":1156660397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6817","nonce":1156660397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1156660397},{"type":"v1","addr":"172.21.15.16:6815","nonce":1156660397}]},"public_addr":"172.21.15.16:6811/1156660397","cluster_addr":"172.21.15.16:6813/1156660397","heartbeat_back_addr":"172.21.15.16:6817/1156660397","heartbeat_front_addr":"172.21.15.16:6815/1156660397","state":["exists","up"]},{"osd":4,"uuid":"e348d76d-044d-4a3a-8420-8e283130eb95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6809","nonce":1591043365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6811","nonce":1591043365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6815","nonce":1591043365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1591043365},{"type":"v1","addr":"172.21.15.27:6813","nonce":1591043365}]},"public_addr":"172.21.15.27:6809/1591043365","cluster_addr":"172.21.15.27:6811/1591043365","heartbeat_back_addr":"172.21.15.27:6815/1591043365","heartbeat_front_addr":"172.21.15.27:6813/1591043365","state":["exists","up"]},{"osd":5,"uuid":"9d921f6c-abd3-47b6-a898-5f5bf0afd5d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6809","nonce":1595822481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6811","nonce":1595822481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6815","nonce":1595822481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1595822481},{"type":"v1","addr":"172.21.15.130:6813","nonce":1595822481}]},"public_addr":"172.21.15.130:6809/1595822481","cluster_addr":"172.21.15.130:6811/1595822481","heartbeat_back_addr":"172.21.15.130:6815/1595822481","heartbeat_front_addr":"172.21.15.130:6813/1595822481","state":["exists","up"]},{"osd":6,"uuid":"6d56ac7f-b02c-4aaf-9c0f-dd51ac4817d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6819","nonce":2729842886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6821","nonce":2729842886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6825","nonce":2729842886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2729842886},{"type":"v1","addr":"172.21.15.16:6823","nonce":2729842886}]},"public_addr":"172.21.15.16:6819/2729842886","cluster_addr":"172.21.15.16:6821/2729842886","heartbeat_back_addr":"172.21.15.16:6825/2729842886","heartbeat_front_addr":"172.21.15.16:6823/2729842886","state":["exists","up"]},{"osd":7,"uuid":"bc642843-7b4c-48ea-9c1c-74345e3da8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6817","nonce":3123372166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6819","nonce":3123372166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6823","nonce":3123372166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3123372166},{"type":"v1","addr":"172.21.15.27:6821","nonce":3123372166}]},"public_addr":"172.21.15.27:6817/3123372166","cluster_addr":"172.21.15.27:6819/3123372166","heartbeat_back_addr":"172.21.15.27:6823/3123372166","heartbeat_front_addr":"172.21.15.27:6821/3123372166","state":["exists","up"]},{"osd":8,"uuid":"584a112e-7c86-49ea-b1b6-30733d0f1e31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6817","nonce":3206875138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6819","nonce":3206875138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6823","nonce":3206875138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3206875138},{"type":"v1","addr":"172.21.15.130:6821","nonce":3206875138}]},"public_addr":"172.21.15.130:6817/3206875138","cluster_addr":"172.21.15.130:6819/3206875138","heartbeat_back_addr":"172.21.15.130:6823/3206875138","heartbeat_front_addr":"172.21.15.130:6821/3206875138","state":["exists","up"]},{"osd":9,"uuid":"14bd3f6d-201c-45c2-b82a-aa358d1a0f28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6827","nonce":2518129600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6829","nonce":2518129600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6833","nonce":2518129600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2518129600},{"type":"v1","addr":"172.21.15.16:6831","nonce":2518129600}]},"public_addr":"172.21.15.16:6827/2518129600","cluster_addr":"172.21.15.16:6829/2518129600","heartbeat_back_addr":"172.21.15.16:6833/2518129600","heartbeat_front_addr":"172.21.15.16:6831/2518129600","state":["exists","up"]},{"osd":10,"uuid":"366e5c3c-48b4-4718-9128-021c930445ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6825","nonce":15853906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6827","nonce":15853906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6831","nonce":15853906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":15853906},{"type":"v1","addr":"172.21.15.27:6829","nonce":15853906}]},"public_addr":"172.21.15.27:6825/15853906","cluster_addr":"172.21.15.27:6827/15853906","heartbeat_back_addr":"172.21.15.27:6831/15853906","heartbeat_front_addr":"172.21.15.27:6829/15853906","state":["exists","up"]},{"osd":11,"uuid":"2f8660f8-f57e-4f24-9b2e-bab7a440fc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6825","nonce":4254452589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6827","nonce":4254452589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6831","nonce":4254452589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":4254452589},{"type":"v1","addr":"172.21.15.130:6829","nonce":4254452589}]},"public_addr":"172.21.15.130:6825/4254452589","cluster_addr":"172.21.15.130:6827/4254452589","heartbeat_back_addr":"172.21.15.130:6831/4254452589","heartbeat_front_addr":"172.21.15.130:6829/4254452589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:27:54.789506+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:13.085443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:30.663014+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:28:51.933785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:10.434212+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:28.544825+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:29:49.966436+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:08.922453+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:26.593544+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:30:47.625315+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:06.054681+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T23:31:23.824306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/985211078":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/3856233973":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1740984099":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/1696925921":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/4157043736":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/2307977786":"2024-04-06T23:26:11.600470+0000","172.21.15.16:0/1141331215":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/1106968003":"2024-04-06T23:26:11.600470+0000","172.21.15.16:6800/1565223355":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/3668421167":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6800/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:6801/2429779560":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/749035377":"2024-04-06T23:26:37.227308+0000","172.21.15.16:0/2846240830":"2024-04-06T23:33:34.593227+0000","172.21.15.16:0/4184690359":"2024-04-06T23:33:34.593227+0000","172.21.15.16:6801/1565223355":"2024-04-06T23:33:34.593227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":23,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T23:47:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a2c0a13e0 msgr2=0x7f6a2c0a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a2c0a13e0 0x7f6a2c0a38a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6a48198860 tx=0x7f6a34009260 comp rx=0 tx=0).stop 2024-04-05T23:47:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a48101ee0 msgr2=0x7f6a48197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a48101ee0 0x7f6a48197310 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6a3800e9a0 tx=0x7f6a3800ecb0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 shutdown_connections 2024-04-05T23:47:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f6a2c0a13e0 0x7f6a2c0a38a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a48101ee0 0x7f6a48197310 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4810b300 0x7f6a4819bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 --2- 172.21.15.16:0/721128687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48102800 0x7f6a48197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 >> 172.21.15.16:0/721128687 conn(0x7f6a480fb720 msgr2=0x7f6a480fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 shutdown_connections 2024-04-05T23:47:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.036+0000 7f6a4cc00700 1 -- 172.21.15.16:0/721128687 wait complete. 2024-04-05T23:47:43.047 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-05T23:47:43.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:42 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1612988602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:47:43.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:42 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/1016530582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:47:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.255+0000 7ff3ea000700 1 -- 172.21.15.16:0/819126327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4103c60 msgr2=0x7ff3e4107ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.255+0000 7ff3ea000700 1 --2- 172.21.15.16:0/819126327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4103c60 0x7ff3e4107ca0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009990 tx=0x7ff3d0009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.255+0000 7ff3ea000700 1 -- 172.21.15.16:0/819126327 shutdown_connections 2024-04-05T23:47:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.255+0000 7ff3ea000700 1 --2- 172.21.15.16:0/819126327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e41032b0 0x7ff3e4103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.255+0000 7ff3ea000700 1 --2- 172.21.15.16:0/819126327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e41083d0 0x7ff3e410ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.255+0000 7ff3ea000700 1 --2- 172.21.15.16:0/819126327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4103c60 0x7ff3e4107ca0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.255+0000 7ff3ea000700 1 -- 172.21.15.16:0/819126327 >> 172.21.15.16:0/819126327 conn(0x7ff3e40feaf0 msgr2=0x7ff3e4100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.255+0000 7ff3ea000700 1 -- 172.21.15.16:0/819126327 shutdown_connections 2024-04-05T23:47:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.255+0000 7ff3ea000700 1 -- 172.21.15.16:0/819126327 wait complete. 2024-04-05T23:47:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.256+0000 7ff3ea000700 1 Processor -- start 2024-04-05T23:47:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.256+0000 7ff3ea000700 1 -- start start 2024-04-05T23:47:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3ea000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e41032b0 0x7ff3e41994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3ea000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e41083d0 0x7ff3e4199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3ea000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e419ddc0 0x7ff3e419e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3ea000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e419e840 con 0x7ff3e41032b0 2024-04-05T23:47:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3ea000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e419e9b0 con 0x7ff3e41083d0 2024-04-05T23:47:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3ea000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e419eb70 con 0x7ff3e419ddc0 2024-04-05T23:47:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3e3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e41032b0 0x7ff3e41994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3e3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e41032b0 0x7ff3e41994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51332/0 (socket says 172.21.15.16:51332) 2024-04-05T23:47:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3e2a00700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e41083d0 0x7ff3e4199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3e3400700 1 -- 172.21.15.16:0/4170481736 learned_addr learned my addr 172.21.15.16:0/4170481736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3e3e00700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e419ddc0 0x7ff3e419e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.257+0000 7ff3e3400700 1 -- 172.21.15.16:0/4170481736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e419ddc0 msgr2=0x7ff3e419e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3e3400700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e419ddc0 0x7ff3e419e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3e3400700 1 -- 172.21.15.16:0/4170481736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e41083d0 msgr2=0x7ff3e4199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3e3400700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e41083d0 0x7ff3e4199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3e3400700 1 -- 172.21.15.16:0/4170481736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d00095f0 con 0x7ff3e41032b0 2024-04-05T23:47:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3e2a00700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e41083d0 0x7ff3e4199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T23:47:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3e3400700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e41032b0 0x7ff3e41994e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400e9a0 tx=0x7ff3d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3dbe00700 1 -- 172.21.15.16:0/4170481736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4009790 con 0x7ff3e41032b0 2024-04-05T23:47:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e41a60d0 con 0x7ff3e41032b0 2024-04-05T23:47:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3dbe00700 1 -- 172.21.15.16:0/4170481736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3d4010430 con 0x7ff3e41032b0 2024-04-05T23:47:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3dbe00700 1 -- 172.21.15.16:0/4170481736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4018680 con 0x7ff3e41032b0 2024-04-05T23:47:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.258+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e41a6600 con 0x7ff3e41032b0 2024-04-05T23:47:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.259+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e410b6a0 con 0x7ff3e41032b0 2024-04-05T23:47:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.261+0000 7ff3dbe00700 1 -- 172.21.15.16:0/4170481736 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7ff3d40105a0 con 0x7ff3e41032b0 2024-04-05T23:47:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.262+0000 7ff3dbe00700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff3c80a1310 0x7ff3c80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.262+0000 7ff3dbe00700 1 -- 172.21.15.16:0/4170481736 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8556+0+0 (secure 0 0 0) 0x7ff3d4014070 con 0x7ff3e41032b0 2024-04-05T23:47:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.263+0000 7ff3e2a00700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff3c80a1310 0x7ff3c80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:43.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.263+0000 7ff3e2a00700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff3c80a1310 0x7ff3c80a37d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000b2e0 tx=0x7ff3d0005960 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:43.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.266+0000 7ff3dbe00700 1 -- 172.21.15.16:0/4170481736 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff3d4052ab0 con 0x7ff3e41032b0 2024-04-05T23:47:43.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:42 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1612988602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:47:43.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:42 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/1016530582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:47:43.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:42 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1612988602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:47:43.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:42 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/1016530582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T23:47:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.532+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff3e419a7e0 con 0x7ff3e41032b0 2024-04-05T23:47:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.542+0000 7ff3dbe00700 1 -- 172.21.15.16:0/4170481736 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff3d4096280 con 0x7ff3e41032b0 2024-04-05T23:47:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff3c80a1310 msgr2=0x7ff3c80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff3c80a1310 0x7ff3c80a37d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000b2e0 tx=0x7ff3d0005960 comp rx=0 tx=0).stop 2024-04-05T23:47:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e41032b0 msgr2=0x7ff3e41994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e41032b0 0x7ff3e41994e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400e9a0 tx=0x7ff3d400ecb0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 shutdown_connections 2024-04-05T23:47:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7ff3c80a1310 0x7ff3c80a37d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3e41032b0 0x7ff3e41994e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3e419ddc0 0x7ff3e419e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 --2- 172.21.15.16:0/4170481736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e41083d0 0x7ff3e4199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 >> 172.21.15.16:0/4170481736 conn(0x7ff3e40feaf0 msgr2=0x7ff3e40ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 shutdown_connections 2024-04-05T23:47:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.545+0000 7ff3ea000700 1 -- 172.21.15.16:0/4170481736 wait complete. 2024-04-05T23:47:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-05T23:47:43.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-05T23:47:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.780+0000 7fd821e00700 1 -- 172.21.15.16:0/2247715319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c1010a0 msgr2=0x7fd81c101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.780+0000 7fd821e00700 1 --2- 172.21.15.16:0/2247715319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c1010a0 0x7fd81c101480 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009990 tx=0x7fd80c009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.780+0000 7fd821e00700 1 -- 172.21.15.16:0/2247715319 shutdown_connections 2024-04-05T23:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.780+0000 7fd821e00700 1 --2- 172.21.15.16:0/2247715319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c1061c0 0x7fd81c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.780+0000 7fd821e00700 1 --2- 172.21.15.16:0/2247715319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c1010a0 0x7fd81c101480 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.780+0000 7fd821e00700 1 --2- 172.21.15.16:0/2247715319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c101a50 0x7fd81c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.780+0000 7fd821e00700 1 -- 172.21.15.16:0/2247715319 >> 172.21.15.16:0/2247715319 conn(0x7fd81c0fc900 msgr2=0x7fd81c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:43.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.780+0000 7fd821e00700 1 -- 172.21.15.16:0/2247715319 shutdown_connections 2024-04-05T23:47:43.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.780+0000 7fd821e00700 1 -- 172.21.15.16:0/2247715319 wait complete. 2024-04-05T23:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.781+0000 7fd821e00700 1 Processor -- start 2024-04-05T23:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.781+0000 7fd821e00700 1 -- start start 2024-04-05T23:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.781+0000 7fd821e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1010a0 0x7fd81c199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.781+0000 7fd821e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c1061c0 0x7fd81c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd821e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c19de40 0x7fd81c19e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:43.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd821e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c19e8c0 con 0x7fd81c1061c0 2024-04-05T23:47:43.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd821e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c19ea30 con 0x7fd81c19de40 2024-04-05T23:47:43.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd821e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c19ebf0 con 0x7fd81c1010a0 2024-04-05T23:47:43.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd813e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c1061c0 0x7fd81c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:43.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd813e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c1061c0 0x7fd81c199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51378/0 (socket says 172.21.15.16:51378) 2024-04-05T23:47:43.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd813e00700 1 -- 172.21.15.16:0/653900711 learned_addr learned my addr 172.21.15.16:0/653900711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:43.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd81b400700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c19de40 0x7fd81c19e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:43.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd81aa00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1010a0 0x7fd81c199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:43.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd813e00700 1 -- 172.21.15.16:0/653900711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1010a0 msgr2=0x7fd81c199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd813e00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1010a0 0x7fd81c199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd813e00700 1 -- 172.21.15.16:0/653900711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c19de40 msgr2=0x7fd81c19e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:43.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd813e00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c19de40 0x7fd81c19e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:43.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.782+0000 7fd813e00700 1 -- 172.21.15.16:0/653900711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c0095f0 con 0x7fd81c1061c0 2024-04-05T23:47:43.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.783+0000 7fd813e00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c1061c0 0x7fd81c199aa0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd80400e970 tx=0x7fd80400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.783+0000 7fd820a00700 1 -- 172.21.15.16:0/653900711 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804009790 con 0x7fd81c1061c0 2024-04-05T23:47:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.783+0000 7fd820a00700 1 -- 172.21.15.16:0/653900711 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd80400c7e0 con 0x7fd81c1061c0 2024-04-05T23:47:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.783+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c1a6150 con 0x7fd81c1061c0 2024-04-05T23:47:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.783+0000 7fd820a00700 1 -- 172.21.15.16:0/653900711 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8040054e0 con 0x7fd81c1061c0 2024-04-05T23:47:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.783+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c1a6630 con 0x7fd81c1061c0 2024-04-05T23:47:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.784+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c102570 con 0x7fd81c1061c0 2024-04-05T23:47:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.786+0000 7fd820a00700 1 -- 172.21.15.16:0/653900711 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7fd80400c950 con 0x7fd81c1061c0 2024-04-05T23:47:43.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.787+0000 7fd820a00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8000a1290 0x7fd8000a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:43.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.788+0000 7fd820a00700 1 -- 172.21.15.16:0/653900711 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8942+0+0 (secure 0 0 0) 0x7fd804014070 con 0x7fd81c1061c0 2024-04-05T23:47:43.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.788+0000 7fd81aa00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8000a1290 0x7fd8000a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:43.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.788+0000 7fd81aa00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8000a1290 0x7fd8000a3750 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd80c000c00 tx=0x7fd80c00b260 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:43.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:43.791+0000 7fd820a00700 1 -- 172.21.15.16:0/653900711 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd804096370 con 0x7fd81c1061c0 2024-04-05T23:47:43.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a[38225]: 2024-04-05T23:47:43.533+0000 7f0cad000700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T23:47:43.831 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-d[89447]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.831 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-d[89447]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.831 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-d[89447]: starting mds.d at 2024-04-05T23:47:43.832 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-a[90400]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.832 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-a[90400]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.832 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mds[90424]: -- 172.21.15.16:0/1188280096 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b12f5621a0 con 0x55b12f53b000 2024-04-05T23:47:43.832 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-a[90400]: starting mds.a at 2024-04-05T23:47:43.833 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.833 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.833 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: starting mds.c at 2024-04-05T23:47:43.833 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-l[116913]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.833 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-l[116913]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.834 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mds[116936]: -- 172.21.15.130:0/2408464302 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55571c3241a0 con 0x55571c2fac00 2024-04-05T23:47:43.834 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-l[116913]: starting mds.l at 2024-04-05T23:47:43.834 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.834 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.835 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: starting mds.f at 2024-04-05T23:47:43.835 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-i[117948]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.835 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-i[117948]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.835 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mds[117971]: -- 172.21.15.130:0/1660764256 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ce6ace61a0 con 0x55ce6acbcc00 2024-04-05T23:47:43.835 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-i[117948]: starting mds.i at 2024-04-05T23:47:43.942 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-b[116618]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.942 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-b[116618]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.942 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-b[116618]: starting mds.b at 2024-04-05T23:47:43.943 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-e[115929]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.943 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-e[115929]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.943 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-e[115929]: starting mds.e at 2024-04-05T23:47:43.943 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-h[116269]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.943 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-h[116269]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.944 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-h[116269]: starting mds.h at 2024-04-05T23:47:43.944 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-k[115586]: ignoring --setuser ceph since I am not root 2024-04-05T23:47:43.944 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-k[115586]: ignoring --setgroup ceph since I am not root 2024-04-05T23:47:43.944 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-k[115586]: starting mds.k at 2024-04-05T23:47:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.056+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd81c19a860 con 0x7fd81c1061c0 2024-04-05T23:47:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-mon[78597]: pgmap v446: 97 pgs: 97 active+clean; 514 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 878 KiB/s wr, 31 op/s 2024-04-05T23:47:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/721128687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:47:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4170481736' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-05T23:47:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-mon[78597]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T23:47:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-mon[78597]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T23:47:44.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-mon[78597]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-04-05T23:47:44.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-mon[78597]: osdmap e102: 12 total, 12 up, 12 in 2024-04-05T23:47:44.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/4170481736' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-05T23:47:44.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:43 smithi027 ceph-mon[78597]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-05T23:47:44.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mon[38248]: pgmap v446: 97 pgs: 97 active+clean; 514 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 878 KiB/s wr, 31 op/s 2024-04-05T23:47:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/721128687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:47:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4170481736' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-05T23:47:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mon[38248]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T23:47:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mon[38248]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T23:47:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mon[38248]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-04-05T23:47:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mon[38248]: osdmap e102: 12 total, 12 up, 12 in 2024-04-05T23:47:44.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/4170481736' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-05T23:47:44.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:43 smithi016 ceph-mon[38248]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-05T23:47:44.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mon[79417]: pgmap v446: 97 pgs: 97 active+clean; 514 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 878 KiB/s wr, 31 op/s 2024-04-05T23:47:44.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/721128687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T23:47:44.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4170481736' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-05T23:47:44.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mon[79417]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T23:47:44.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mon[79417]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T23:47:44.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mon[79417]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-04-05T23:47:44.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mon[79417]: osdmap e102: 12 total, 12 up, 12 in 2024-04-05T23:47:44.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/4170481736' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-05T23:47:44.333 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:43 smithi130 ceph-mon[79417]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-05T23:47:44.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.545+0000 7fd820a00700 1 -- 172.21.15.16:0/653900711 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd804099990 con 0x7fd81c1061c0 2024-04-05T23:47:44.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.550+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8000a1290 msgr2=0x7fd8000a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:44.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.550+0000 7fd821e00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8000a1290 0x7fd8000a3750 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd80c000c00 tx=0x7fd80c00b260 comp rx=0 tx=0).stop 2024-04-05T23:47:44.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.550+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c1061c0 msgr2=0x7fd81c199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:44.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.550+0000 7fd821e00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c1061c0 0x7fd81c199aa0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd80400e970 tx=0x7fd80400ec80 comp rx=0 tx=0).stop 2024-04-05T23:47:44.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.551+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 shutdown_connections 2024-04-05T23:47:44.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.551+0000 7fd821e00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7fd8000a1290 0x7fd8000a3750 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.551+0000 7fd821e00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c19de40 0x7fd81c19e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.551+0000 7fd821e00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1010a0 0x7fd81c199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.551+0000 7fd821e00700 1 --2- 172.21.15.16:0/653900711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd81c1061c0 0x7fd81c199aa0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.551+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 >> 172.21.15.16:0/653900711 conn(0x7fd81c0fc900 msgr2=0x7fd81c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:44.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.551+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 shutdown_connections 2024-04-05T23:47:44.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.552+0000 7fd821e00700 1 -- 172.21.15.16:0/653900711 wait complete. 2024-04-05T23:47:44.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-05T23:47:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.782+0000 7f8c96600700 1 -- 172.21.15.16:0/2525052393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c901010a0 msgr2=0x7f8c90101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.782+0000 7f8c96600700 1 --2- 172.21.15.16:0/2525052393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c901010a0 0x7f8c90101480 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009990 tx=0x7f8c80009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.782+0000 7f8c96600700 1 -- 172.21.15.16:0/2525052393 shutdown_connections 2024-04-05T23:47:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.782+0000 7f8c96600700 1 --2- 172.21.15.16:0/2525052393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c901010a0 0x7f8c90101480 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.782+0000 7f8c96600700 1 --2- 172.21.15.16:0/2525052393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c901061c0 0x7f8c9010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.782+0000 7f8c96600700 1 --2- 172.21.15.16:0/2525052393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c90101a50 0x7f8c90105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.782+0000 7f8c96600700 1 -- 172.21.15.16:0/2525052393 >> 172.21.15.16:0/2525052393 conn(0x7f8c900fc900 msgr2=0x7f8c900fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.782+0000 7f8c96600700 1 -- 172.21.15.16:0/2525052393 shutdown_connections 2024-04-05T23:47:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.783+0000 7f8c96600700 1 -- 172.21.15.16:0/2525052393 wait complete. 2024-04-05T23:47:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.784+0000 7f8c96600700 1 Processor -- start 2024-04-05T23:47:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.784+0000 7f8c96600700 1 -- start start 2024-04-05T23:47:44.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c96600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c901010a0 0x7f8c901995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:44.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c96600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c901061c0 0x7f8c90199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:44.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c96600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c9019dea0 0x7f8c9019e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:44.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c96600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9019e920 con 0x7f8c9019dea0 2024-04-05T23:47:44.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c96600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9019ea90 con 0x7f8c901061c0 2024-04-05T23:47:44.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c96600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9019ec50 con 0x7f8c901010a0 2024-04-05T23:47:44.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c8fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c9019dea0 0x7f8c9019e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:44.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c8fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c9019dea0 0x7f8c9019e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51398/0 (socket says 172.21.15.16:51398) 2024-04-05T23:47:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c8fe00700 1 -- 172.21.15.16:0/3089096671 learned_addr learned my addr 172.21.15.16:0/3089096671 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c8ea00700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c901061c0 0x7f8c90199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c8f400700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c901010a0 0x7f8c901995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c8fe00700 1 -- 172.21.15.16:0/3089096671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c901010a0 msgr2=0x7f8c901995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.785+0000 7f8c8fe00700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c901010a0 0x7f8c901995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.786+0000 7f8c8fe00700 1 -- 172.21.15.16:0/3089096671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c901061c0 msgr2=0x7f8c90199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.786+0000 7f8c8fe00700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c901061c0 0x7f8c90199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.786+0000 7f8c8fe00700 1 -- 172.21.15.16:0/3089096671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c800095f0 con 0x7f8c9019dea0 2024-04-05T23:47:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.786+0000 7f8c8fe00700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c9019dea0 0x7f8c9019e320 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800ef30 tx=0x7f8c8800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:44.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.786+0000 7f8c87e00700 1 -- 172.21.15.16:0/3089096671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c88018070 con 0x7f8c9019dea0 2024-04-05T23:47:44.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.786+0000 7f8c87e00700 1 -- 172.21.15.16:0/3089096671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c8800f040 con 0x7f8c9019dea0 2024-04-05T23:47:44.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.786+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c901a6150 con 0x7f8c9019dea0 2024-04-05T23:47:44.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.786+0000 7f8c87e00700 1 -- 172.21.15.16:0/3089096671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8800cd00 con 0x7f8c9019dea0 2024-04-05T23:47:44.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.786+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c901a6680 con 0x7f8c9019dea0 2024-04-05T23:47:44.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.787+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c90102570 con 0x7f8c9019dea0 2024-04-05T23:47:44.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.789+0000 7f8c87e00700 1 -- 172.21.15.16:0/3089096671 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f8c88003a00 con 0x7f8c9019dea0 2024-04-05T23:47:44.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.790+0000 7f8c87e00700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8c740a1310 0x7f8c740a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:44.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.791+0000 7f8c87e00700 1 -- 172.21.15.16:0/3089096671 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8942+0+0 (secure 0 0 0) 0x7f8c880cc920 con 0x7f8c9019dea0 2024-04-05T23:47:44.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.791+0000 7f8c8f400700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8c740a1310 0x7f8c740a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:44.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.791+0000 7f8c8f400700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8c740a1310 0x7f8c740a37d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000b2e0 tx=0x7f8c80005fd0 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:44.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:44.795+0000 7f8c87e00700 1 -- 172.21.15.16:0/3089096671 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8c88095c00 con 0x7f8c9019dea0 2024-04-05T23:47:45.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.014+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8c90101a50 con 0x7f8c9019dea0 2024-04-05T23:47:45.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/653900711' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T23:47:45.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T23:47:45.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T23:47:45.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: Cluster is now healthy 2024-04-05T23:47:45.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.27:6833/182597772,v1:172.21.15.27:6836/182597772] up:boot 2024-04-05T23:47:45.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.27:6838/3673910150,v1:172.21.15.27:6839/3673910150] up:boot 2024-04-05T23:47:45.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6834/755683590,v1:172.21.15.130:6835/755683590] up:boot 2024-04-05T23:47:45.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6832/1580873428,v1:172.21.15.130:6833/1580873428] up:boot 2024-04-05T23:47:45.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6836/2019932061,v1:172.21.15.130:6837/2019932061] up:boot 2024-04-05T23:47:45.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.27:6834/3489949202,v1:172.21.15.27:6837/3489949202] up:boot 2024-04-05T23:47:45.216 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.27:6832/1917810605,v1:172.21.15.27:6835/1917810605] up:boot 2024-04-05T23:47:45.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.130:6838/2552974570,v1:172.21.15.130:6839/2552974570] up:boot 2024-04-05T23:47:45.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.16:6836/3584763211,v1:172.21.15.16:6840/3584763211] up:boot 2024-04-05T23:47:45.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: mds.? [v2:172.21.15.16:6837/4237319875,v1:172.21.15.16:6841/4237319875] up:boot 2024-04-05T23:47:45.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/653900711' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-05T23:47:45.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: fsmap 12 up:standby 2024-04-05T23:47:45.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-05T23:47:45.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-05T23:47:45.217 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-05T23:47:45.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-05T23:47:45.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-05T23:47:45.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-05T23:47:45.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-05T23:47:45.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:47:45.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-05T23:47:45.218 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:44 smithi027 ceph-mon[78597]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-05T23:47:45.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/653900711' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T23:47:45.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T23:47:45.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T23:47:45.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: Cluster is now healthy 2024-04-05T23:47:45.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.27:6833/182597772,v1:172.21.15.27:6836/182597772] up:boot 2024-04-05T23:47:45.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.27:6838/3673910150,v1:172.21.15.27:6839/3673910150] up:boot 2024-04-05T23:47:45.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6834/755683590,v1:172.21.15.130:6835/755683590] up:boot 2024-04-05T23:47:45.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6832/1580873428,v1:172.21.15.130:6833/1580873428] up:boot 2024-04-05T23:47:45.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6836/2019932061,v1:172.21.15.130:6837/2019932061] up:boot 2024-04-05T23:47:45.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.27:6834/3489949202,v1:172.21.15.27:6837/3489949202] up:boot 2024-04-05T23:47:45.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.27:6832/1917810605,v1:172.21.15.27:6835/1917810605] up:boot 2024-04-05T23:47:45.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.130:6838/2552974570,v1:172.21.15.130:6839/2552974570] up:boot 2024-04-05T23:47:45.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.16:6836/3584763211,v1:172.21.15.16:6840/3584763211] up:boot 2024-04-05T23:47:45.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: mds.? [v2:172.21.15.16:6837/4237319875,v1:172.21.15.16:6841/4237319875] up:boot 2024-04-05T23:47:45.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/653900711' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-05T23:47:45.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: fsmap 12 up:standby 2024-04-05T23:47:45.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-05T23:47:45.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-05T23:47:45.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-05T23:47:45.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-05T23:47:45.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-05T23:47:45.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-05T23:47:45.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-05T23:47:45.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:47:45.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-05T23:47:45.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:44 smithi016 ceph-mon[38248]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-05T23:47:45.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/653900711' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T23:47:45.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T23:47:45.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T23:47:45.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: Cluster is now healthy 2024-04-05T23:47:45.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.27:6833/182597772,v1:172.21.15.27:6836/182597772] up:boot 2024-04-05T23:47:45.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.27:6838/3673910150,v1:172.21.15.27:6839/3673910150] up:boot 2024-04-05T23:47:45.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6834/755683590,v1:172.21.15.130:6835/755683590] up:boot 2024-04-05T23:47:45.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6832/1580873428,v1:172.21.15.130:6833/1580873428] up:boot 2024-04-05T23:47:45.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6836/2019932061,v1:172.21.15.130:6837/2019932061] up:boot 2024-04-05T23:47:45.335 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.27:6834/3489949202,v1:172.21.15.27:6837/3489949202] up:boot 2024-04-05T23:47:45.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.27:6832/1917810605,v1:172.21.15.27:6835/1917810605] up:boot 2024-04-05T23:47:45.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.130:6838/2552974570,v1:172.21.15.130:6839/2552974570] up:boot 2024-04-05T23:47:45.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.16:6836/3584763211,v1:172.21.15.16:6840/3584763211] up:boot 2024-04-05T23:47:45.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: mds.? [v2:172.21.15.16:6837/4237319875,v1:172.21.15.16:6841/4237319875] up:boot 2024-04-05T23:47:45.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/653900711' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-05T23:47:45.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: fsmap 12 up:standby 2024-04-05T23:47:45.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-05T23:47:45.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-05T23:47:45.336 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-05T23:47:45.337 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-05T23:47:45.337 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-05T23:47:45.337 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-05T23:47:45.337 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-05T23:47:45.337 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-05T23:47:45.337 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-05T23:47:45.337 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:44 smithi130 ceph-mon[79417]: from='mgr.15240 172.21.15.16:0/624385738' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-05T23:47:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.964+0000 7f8c87e00700 1 -- 172.21.15.16:0/3089096671 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8c88099220 con 0x7f8c9019dea0 2024-04-05T23:47:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8c740a1310 msgr2=0x7f8c740a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8c740a1310 0x7f8c740a37d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000b2e0 tx=0x7f8c80005fd0 comp rx=0 tx=0).stop 2024-04-05T23:47:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c9019dea0 msgr2=0x7f8c9019e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c9019dea0 0x7f8c9019e320 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800ef30 tx=0x7f8c8800c3a0 comp rx=0 tx=0).stop 2024-04-05T23:47:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 shutdown_connections 2024-04-05T23:47:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f8c740a1310 0x7f8c740a37d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c9019dea0 0x7f8c9019e320 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c901010a0 0x7f8c901995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 --2- 172.21.15.16:0/3089096671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c901061c0 0x7f8c90199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 >> 172.21.15.16:0/3089096671 conn(0x7f8c900fc900 msgr2=0x7f8c900fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 shutdown_connections 2024-04-05T23:47:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:45.969+0000 7f8c96600700 1 -- 172.21.15.16:0/3089096671 wait complete. 2024-04-05T23:47:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-04-05T23:47:45.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-05T23:47:46.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.211+0000 7f7cbbe00700 1 -- 172.21.15.16:0/467840052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cbc100ed0 msgr2=0x7f7cbc1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:46.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.211+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/467840052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cbc100ed0 0x7f7cbc1012b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009990 tx=0x7f7cac009ca0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.212+0000 7f7cbbe00700 1 -- 172.21.15.16:0/467840052 shutdown_connections 2024-04-05T23:47:46.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.212+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/467840052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7cbc105ff0 0x7f7cbc10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.212+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/467840052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cbc100ed0 0x7f7cbc1012b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.212+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/467840052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc101880 0x7f7cbc1058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.212+0000 7f7cbbe00700 1 -- 172.21.15.16:0/467840052 >> 172.21.15.16:0/467840052 conn(0x7f7cbc0fc750 msgr2=0x7f7cbc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:46.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.212+0000 7f7cbbe00700 1 -- 172.21.15.16:0/467840052 shutdown_connections 2024-04-05T23:47:46.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.212+0000 7f7cbbe00700 1 -- 172.21.15.16:0/467840052 wait complete. 2024-04-05T23:47:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.213+0000 7f7cbbe00700 1 Processor -- start 2024-04-05T23:47:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.213+0000 7f7cbbe00700 1 -- start start 2024-04-05T23:47:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.213+0000 7f7cbbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cbc100ed0 0x7f7cbc1971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.213+0000 7f7cbbe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc105ff0 0x7f7cbc1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.213+0000 7f7cbbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7cbc19ba40 0x7f7cbc19bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.213+0000 7f7cbbe00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc19c4c0 con 0x7f7cbc105ff0 2024-04-05T23:47:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.213+0000 7f7cbbe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc19c630 con 0x7f7cbc100ed0 2024-04-05T23:47:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.213+0000 7f7cbbe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc19c7f0 con 0x7f7cbc19ba40 2024-04-05T23:47:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cbc100ed0 0x7f7cbc1971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbb400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7cbc19ba40 0x7f7cbc19bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cbc100ed0 0x7f7cbc1971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.16:51072/0 (socket says 172.21.15.16:51072) 2024-04-05T23:47:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbaa00700 1 -- 172.21.15.16:0/3015779820 learned_addr learned my addr 172.21.15.16:0/3015779820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-05T23:47:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cba000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc105ff0 0x7f7cbc1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbb400700 1 -- 172.21.15.16:0/3015779820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cbc100ed0 msgr2=0x7f7cbc1971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbb400700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cbc100ed0 0x7f7cbc1971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbb400700 1 -- 172.21.15.16:0/3015779820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc105ff0 msgr2=0x7f7cbc1976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbb400700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc105ff0 0x7f7cbc1976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbb400700 1 -- 172.21.15.16:0/3015779820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0095f0 con 0x7f7cbc19ba40 2024-04-05T23:47:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cbb400700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7cbc19ba40 0x7f7cbc19bec0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400efc0 tx=0x7f7cb400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.214+0000 7f7cb3400700 1 -- 172.21.15.16:0/3015779820 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4018070 con 0x7f7cbc19ba40 2024-04-05T23:47:46.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.215+0000 7f7cb3400700 1 -- 172.21.15.16:0/3015779820 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cb400f040 con 0x7f7cbc19ba40 2024-04-05T23:47:46.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.215+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc1a3cf0 con 0x7f7cbc19ba40 2024-04-05T23:47:46.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.215+0000 7f7cb3400700 1 -- 172.21.15.16:0/3015779820 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb400cd40 con 0x7f7cbc19ba40 2024-04-05T23:47:46.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.215+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc1a41d0 con 0x7f7cbc19ba40 2024-04-05T23:47:46.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.216+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc109210 con 0x7f7cbc19ba40 2024-04-05T23:47:46.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.218+0000 7f7cb3400700 1 -- 172.21.15.16:0/3015779820 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136004+0+0 (secure 0 0 0) 0x7f7cb4003a00 con 0x7f7cbc19ba40 2024-04-05T23:47:46.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.219+0000 7f7cb3400700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ca00a1290 0x7f7ca00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T23:47:46.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.219+0000 7f7cb3400700 1 -- 172.21.15.16:0/3015779820 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8417+0+0 (secure 0 0 0) 0x7f7cb40cc930 con 0x7f7cbc19ba40 2024-04-05T23:47:46.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.219+0000 7f7cbaa00700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ca00a1290 0x7f7ca00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T23:47:46.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.220+0000 7f7cbaa00700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ca00a1290 0x7f7ca00a3750 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7cac006010 tx=0x7f7cac00b280 comp rx=0 tx=0).ready entity=mgr.15240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T23:47:46.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.221+0000 7f7cb3400700 1 -- 172.21.15.16:0/3015779820 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7cb4095e10 con 0x7f7cbc19ba40 2024-04-05T23:47:46.214 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:45 smithi027 ceph-mon[78597]: pgmap v448: 97 pgs: 97 active+clean; 514 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.0 MiB/s wr, 37 op/s 2024-04-05T23:47:46.214 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:45 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3089096671' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-05T23:47:46.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:45 smithi016 ceph-mon[38248]: pgmap v448: 97 pgs: 97 active+clean; 514 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.0 MiB/s wr, 37 op/s 2024-04-05T23:47:46.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:45 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3089096671' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-05T23:47:46.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:45 smithi130 ceph-mon[79417]: pgmap v448: 97 pgs: 97 active+clean; 514 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.0 MiB/s wr, 37 op/s 2024-04-05T23:47:46.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:45 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3089096671' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-05T23:47:46.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.437+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7cbc1023a0 con 0x7f7cbc19ba40 2024-04-05T23:47:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.972+0000 7f7cb3400700 1 -- 172.21.15.16:0/3015779820 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7cb4099430 con 0x7f7cbc19ba40 2024-04-05T23:47:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.977+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ca00a1290 msgr2=0x7f7ca00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.977+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ca00a1290 0x7f7ca00a3750 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7cac006010 tx=0x7f7cac00b280 comp rx=0 tx=0).stop 2024-04-05T23:47:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.977+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7cbc19ba40 msgr2=0x7f7cbc19bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T23:47:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.977+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7cbc19ba40 0x7f7cbc19bec0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400efc0 tx=0x7f7cb400c3a0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.977+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 shutdown_connections 2024-04-05T23:47:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.977+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.16:6800/3895420073,v1:172.21.15.16:6801/3895420073] conn(0x7f7ca00a1290 0x7f7ca00a3750 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.977+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7cbc19ba40 0x7f7cbc19bec0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.978+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cbc100ed0 0x7f7cbc1971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.978+0000 7f7cbbe00700 1 --2- 172.21.15.16:0/3015779820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc105ff0 0x7f7cbc1976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T23:47:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.978+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 >> 172.21.15.16:0/3015779820 conn(0x7f7cbc0fc750 msgr2=0x7f7cbc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T23:47:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.979+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 shutdown_connections 2024-04-05T23:47:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-05T23:47:46.979+0000 7f7cbbe00700 1 -- 172.21.15.16:0/3015779820 wait complete. 2024-04-05T23:47:46.973 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-04-05T23:47:47.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T23:47:47.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T23:47:47.111 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T23:47:47.142 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-05T23:47:47.142 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-05T23:47:47.185 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-05T23:47:47.202 DEBUG:teuthology.orchestra.run.smithi130:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-05T23:47:47.214 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:46 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3089096671' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-05T23:47:47.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:46 smithi027 ceph-mon[78597]: osdmap e103: 12 total, 12 up, 12 in 2024-04-05T23:47:47.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:46 smithi027 ceph-mon[78597]: from='client.? 172.21.15.16:0/3015779820' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-05T23:47:47.215 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:46 smithi027 ceph-mon[78597]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-05T23:47:47.219 INFO:tasks.cephadm:Stopping all daemons... 2024-04-05T23:47:47.219 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-05T23:47:47.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.a 2024-04-05T23:47:47.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:46 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3089096671' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-05T23:47:47.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:46 smithi016 ceph-mon[38248]: osdmap e103: 12 total, 12 up, 12 in 2024-04-05T23:47:47.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:46 smithi016 ceph-mon[38248]: from='client.? 172.21.15.16:0/3015779820' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-05T23:47:47.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:46 smithi016 ceph-mon[38248]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-05T23:47:47.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:46 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3089096671' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-05T23:47:47.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:46 smithi130 ceph-mon[79417]: osdmap e103: 12 total, 12 up, 12 in 2024-04-05T23:47:47.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:46 smithi130 ceph-mon[79417]: from='client.? 172.21.15.16:0/3015779820' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-05T23:47:47.332 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:46 smithi130 ceph-mon[79417]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-05T23:47:47.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:47 smithi016 systemd[1]: Stopping Ceph mon.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:47:48.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:47 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a[38225]: 2024-04-05T23:47:47.689+0000 7f0cb3e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:47:48.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:47 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a[38225]: 2024-04-05T23:47:47.689+0000 7f0cb3e00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-05T23:47:48.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 05 23:47:47 smithi016 bash[116124]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-a 2024-04-05T23:47:48.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.a.service' 2024-04-05T23:47:48.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:47:48.394 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-05T23:47:48.394 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-05T23:47:48.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.b 2024-04-05T23:47:48.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:48 smithi027 systemd[1]: Stopping Ceph mon.b for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:47:48.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-b[78574]: 2024-04-05T23:47:48.628+0000 7fc9a620d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:47:48.715 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-b[78574]: 2024-04-05T23:47:48.628+0000 7fc9a620d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-05T23:47:49.116 INFO:journalctl@ceph.mon.b.smithi027.stdout:Apr 05 23:47:48 smithi027 bash[124363]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-b 2024-04-05T23:47:49.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.b.service' 2024-04-05T23:47:49.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:47:49.168 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-05T23:47:49.168 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-05T23:47:49.168 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.c 2024-04-05T23:47:49.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:49 smithi130 systemd[1]: Stopping Ceph mon.c for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:47:49.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-c[79394]: 2024-04-05T23:47:49.428+0000 7ffbc3ce7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:47:49.582 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-c[79394]: 2024-04-05T23:47:49.428+0000 7ffbc3ce7700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-05T23:47:50.040 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 05 23:47:49 smithi130 bash[124958]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mon-c 2024-04-05T23:47:50.046 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mon.c.service' 2024-04-05T23:47:50.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:47:50.097 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-05T23:47:50.097 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-05T23:47:50.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.x 2024-04-05T23:47:50.526 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:47:50 smithi016 systemd[1]: Stopping Ceph mgr.x for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:47:50.796 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:47:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:47:50.534+0000 7fa211600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:47:50.796 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:47:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x[38500]: 2024-04-05T23:47:50.534+0000 7fa211600700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-05T23:47:51.080 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 05 23:47:50 smithi016 bash[116264]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-x 2024-04-05T23:47:51.171 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.x.service' 2024-04-05T23:47:51.255 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:47:51.255 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-05T23:47:51.255 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-05T23:47:51.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.y 2024-04-05T23:47:51.613 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:47:51 smithi027 systemd[1]: Stopping Ceph mgr.y for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:47:51.961 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:47:51 smithi027 bash[124501]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-y 2024-04-05T23:47:51.962 INFO:journalctl@ceph.mgr.y.smithi027.stdout:Apr 05 23:47:51 smithi027 systemd[1]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-05T23:47:51.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.y.service' 2024-04-05T23:47:52.007 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:47:52.007 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-05T23:47:52.008 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-05T23:47:52.008 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.z 2024-04-05T23:47:52.332 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:47:52 smithi130 systemd[1]: Stopping Ceph mgr.z for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:47:52.804 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:47:52 smithi130 bash[125098]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mgr-z 2024-04-05T23:47:52.804 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 05 23:47:52 smithi130 systemd[1]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-05T23:47:52.810 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mgr.z.service' 2024-04-05T23:47:52.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:47:52.847 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-05T23:47:52.847 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-05T23:47:52.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.0 2024-04-05T23:47:53.231 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:47:52 smithi016 systemd[1]: Stopping Ceph osd.0 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:47:53.580 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:47:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-0[57079]: 2024-04-05T23:47:53.238+0000 7fdf9f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:47:53.581 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:47:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-0[57079]: 2024-04-05T23:47:53.238+0000 7fdf9f200700 -1 osd.0 104 *** Got signal Terminated *** 2024-04-05T23:47:53.581 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:47:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-0[57079]: 2024-04-05T23:47:53.238+0000 7fdf9f200700 -1 osd.0 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:47:59.330 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 05 23:47:58 smithi016 bash[116402]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-0 2024-04-05T23:48:00.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.0.service' 2024-04-05T23:48:00.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:48:00.593 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-05T23:48:00.593 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-05T23:48:00.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.1 2024-04-05T23:48:00.965 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:48:00 smithi027 systemd[1]: Stopping Ceph osd.1 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:48:00.965 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:48:00 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-1[83584]: 2024-04-05T23:48:00.842+0000 7f47ecb40700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:48:00.966 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:48:00 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-1[83584]: 2024-04-05T23:48:00.842+0000 7f47ecb40700 -1 osd.1 104 *** Got signal Terminated *** 2024-04-05T23:48:00.966 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:48:00 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-1[83584]: 2024-04-05T23:48:00.842+0000 7f47ecb40700 -1 osd.1 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:48:06.714 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 05 23:48:06 smithi027 bash[124654]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-1 2024-04-05T23:48:07.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.1.service' 2024-04-05T23:48:07.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:48:07.915 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-05T23:48:07.916 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-05T23:48:07.916 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.2 2024-04-05T23:48:08.332 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:48:07 smithi130 systemd[1]: Stopping Ceph osd.2 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:48:08.333 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:48:08 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-2[85197]: 2024-04-05T23:48:08.167+0000 7fc4dd597700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:48:08.333 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:48:08 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-2[85197]: 2024-04-05T23:48:08.167+0000 7fc4dd597700 -1 osd.2 104 *** Got signal Terminated *** 2024-04-05T23:48:08.333 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:48:08 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-2[85197]: 2024-04-05T23:48:08.167+0000 7fc4dd597700 -1 osd.2 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:48:14.082 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 05 23:48:13 smithi130 bash[125244]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-2 2024-04-05T23:48:15.178 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.2.service' 2024-04-05T23:48:15.218 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:48:15.218 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-05T23:48:15.218 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-05T23:48:15.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.3 2024-04-05T23:48:15.580 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 05 23:48:15 smithi016 systemd[1]: Stopping Ceph osd.3 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:48:16.080 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 05 23:48:15 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-3[65112]: 2024-04-05T23:48:15.620+0000 7f5b73600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:48:16.081 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 05 23:48:15 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-3[65112]: 2024-04-05T23:48:15.620+0000 7f5b73600700 -1 osd.3 104 *** Got signal Terminated *** 2024-04-05T23:48:16.081 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 05 23:48:15 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-3[65112]: 2024-04-05T23:48:15.620+0000 7f5b73600700 -1 osd.3 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:48:20.218 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:19 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:19.823+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:21.214 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:20 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:20.868+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:21.724 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 05 23:48:21 smithi016 bash[116690]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-3 2024-04-05T23:48:22.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:21 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:21.733+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:22.214 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:21 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:21.861+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:23.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:22 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:22.774+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:23.083 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:22 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:22.814+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:23.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.3.service' 2024-04-05T23:48:23.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:22 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:22.880+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:23.223 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:48:23.223 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-05T23:48:23.223 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-05T23:48:23.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.4 2024-04-05T23:48:23.715 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 05 23:48:23 smithi027 systemd[1]: Stopping Ceph osd.4 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:48:23.715 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 05 23:48:23 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-4[92155]: 2024-04-05T23:48:23.472+0000 7f5993956700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:48:23.715 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 05 23:48:23 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-4[92155]: 2024-04-05T23:48:23.472+0000 7f5993956700 -1 osd.4 104 *** Got signal Terminated *** 2024-04-05T23:48:23.715 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 05 23:48:23 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-4[92155]: 2024-04-05T23:48:23.472+0000 7f5993956700 -1 osd.4 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:48:23.784 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:23 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:23.452+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:23.784 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:23 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:23.593+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:24.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:23 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:23.815+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:24.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:23 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:23.984+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:24.082 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:23 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:23.793+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:24.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:23 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:23.894+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:24.818 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:24 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:24.643+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:24.819 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:24 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:24.437+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:25.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:24 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:24.808+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:25.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:24 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:24.945+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:25.082 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:24 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:24.827+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:25.214 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:24 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:24.872+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:25.804 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:25 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:25.440+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:25.805 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:25 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:25.647+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:26.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:25 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:25.761+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:26.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:25 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:25.952+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:26.082 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:25 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:25.814+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:26.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:25 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:25.835+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:26.818 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:26 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:26.400+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:26.819 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:26 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:26.643+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:26.819 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:26 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:26.643+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.580650+0000 front 2024-04-05T23:48:03.580510+0000 (oldest deadline 2024-04-05T23:48:26.480689+0000) 2024-04-05T23:48:27.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:26 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:26.785+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:27.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:26 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:26.990+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:27.082 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:26 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:26.828+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:27.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:26 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:26.915+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:27.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:26 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:26.866+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:27.655 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:27.369+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:28.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:27 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:27.833+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:28.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:28.010+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:28.082 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:27.875+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:28.082 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:27.665+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:28.082 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:27 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:27.665+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.580650+0000 front 2024-04-05T23:48:03.580510+0000 (oldest deadline 2024-04-05T23:48:26.480689+0000) 2024-04-05T23:48:28.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:27 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:27.942+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:28.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:27 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:27.841+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:28.636 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:28 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:28.395+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:29.082 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:28 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:28.646+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:29.082 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:28 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:28.646+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.580650+0000 front 2024-04-05T23:48:03.580510+0000 (oldest deadline 2024-04-05T23:48:26.480689+0000) 2024-04-05T23:48:29.083 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:28 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:28.837+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:29.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:28 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:28.901+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:29.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:28 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:28.844+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:29.215 INFO:journalctl@ceph.osd.4.smithi027.stdout:Apr 05 23:48:29 smithi027 bash[124908]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-4 2024-04-05T23:48:29.330 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:28.861+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:29.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:28.992+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:29.669 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:29.358+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:29.669 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:29.358+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:30.059 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:29 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:29.888+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:30.059 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:29 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:29.815+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:30.059 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:29 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:29.815+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:30.082 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:29.868+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:30.082 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:29.678+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:30.083 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:29 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:29.678+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.580650+0000 front 2024-04-05T23:48:03.580510+0000 (oldest deadline 2024-04-05T23:48:26.480689+0000) 2024-04-05T23:48:30.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.4.service' 2024-04-05T23:48:30.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:48:30.300 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-05T23:48:30.301 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-05T23:48:30.301 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.5 2024-04-05T23:48:30.330 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:29.877+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:30.331 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:29.877+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:30.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:29.983+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:30.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:29 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:29.983+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:30.582 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:30 smithi130 systemd[1]: Stopping Ceph osd.5 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:48:30.583 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:30.572+0000 7f6ea1885700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:48:30.583 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:30.572+0000 7f6ea1885700 -1 osd.5 104 *** Got signal Terminated *** 2024-04-05T23:48:30.583 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:30.572+0000 7f6ea1885700 -1 osd.5 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:48:30.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:30.362+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:30.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:30.362+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:31.082 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:30.699+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:31.083 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:30.699+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.580650+0000 front 2024-04-05T23:48:03.580510+0000 (oldest deadline 2024-04-05T23:48:26.480689+0000) 2024-04-05T23:48:31.083 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:30 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:30.896+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:31.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:30 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:30.863+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:31.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:30 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:30.794+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:31.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:30 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:30.794+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:31.330 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:30.894+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:31.331 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:30.894+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:31.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:30.948+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:31.332 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:30 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:30.948+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:31.700 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:31.363+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:31.700 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:31.363+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:32.082 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:31.914+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:32.083 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:31.710+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:32.083 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:31 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:31.710+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.580650+0000 front 2024-04-05T23:48:03.580510+0000 (oldest deadline 2024-04-05T23:48:26.480689+0000) 2024-04-05T23:48:32.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:31 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:31.910+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:32.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:31 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:31.800+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:32.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:31 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:31.800+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:32.330 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:31 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:31.879+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:32.331 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:31 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:31.879+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:32.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:31 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:31.956+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:32.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:31 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:31.956+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:32.653 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:32.394+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:32.653 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:32.394+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:32.936 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:32.663+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:32.936 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:32.663+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.580650+0000 front 2024-04-05T23:48:03.580510+0000 (oldest deadline 2024-04-05T23:48:26.480689+0000) 2024-04-05T23:48:33.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:32 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:32.899+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:33.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:32 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:32.809+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:33.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:32 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:32.809+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:33.330 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:32.865+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:33.331 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:32.865+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:33.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:32.933+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:33.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:32 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:32.933+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:33.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:32 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:32.945+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:33.832 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:33 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:33.646+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:33.832 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:33 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:33.646+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.580650+0000 front 2024-04-05T23:48:03.580510+0000 (oldest deadline 2024-04-05T23:48:26.480689+0000) 2024-04-05T23:48:33.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:33 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:33.403+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:33.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:33 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:33.403+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:34.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:33.836+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:34.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:33.836+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:34.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:33.908+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:34.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:33.908+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:34.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:33 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:33.869+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:34.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:33 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:33.825+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:34.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:33 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:33.825+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:34.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:33 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:33.935+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:34.832 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:34 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:34.605+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:54.179569+0000 front 2024-04-05T23:47:54.179498+0000 (oldest deadline 2024-04-05T23:48:23.580024+0000) 2024-04-05T23:48:34.832 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:34 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5[93836]: 2024-04-05T23:48:34.605+0000 7f6e9e67c700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.580650+0000 front 2024-04-05T23:48:03.580510+0000 (oldest deadline 2024-04-05T23:48:26.480689+0000) 2024-04-05T23:48:34.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:34 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:34.365+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:34.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:34 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:34.365+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:34.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:34 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:34.365+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:35.080 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:34.868+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:35.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:34.868+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:35.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:34.797+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:35.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:34 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:34.797+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:35.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:34 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:34.898+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:35.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:34 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:34.850+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:35.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:34 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:34.850+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:35.317 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:34 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:34.950+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:35.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:35 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:35.327+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:35.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:35 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:35.327+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:35.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:35 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:35.327+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:36.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:35 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:35.761+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:36.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:35 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:35.761+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:36.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:35 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:35.894+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:36.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:35 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:35.894+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:36.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:35 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:35.923+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:36.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:35 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:35.923+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:36.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:35 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:35.811+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:36.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:35 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:35.811+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:36.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:35 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:35.811+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:36.291 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 05 23:48:36 smithi130 bash[125502]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-5 2024-04-05T23:48:36.291 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:36 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:36.299+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:36.291 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:36 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:36.299+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:36.292 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:35 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:35.966+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:36.292 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:35 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:35.966+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:36.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:36 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:36.299+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:37.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:36 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:36.788+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:37.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:36 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:36.788+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:37.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:36 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:36.871+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:37.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:36 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:36.871+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:37.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:36 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:36.871+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:37.082 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:36 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:36.926+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:37.082 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:36 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:36.926+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:37.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:36.918+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:37.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:36.918+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:37.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:36.855+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:37.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:36.855+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:37.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:36.855+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:37.442 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:37 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:37.309+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:37.442 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:37 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:37.309+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:37.442 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:37 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:37.309+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:37.450 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.5.service' 2024-04-05T23:48:37.490 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:48:37.490 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-05T23:48:37.491 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-05T23:48:37.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.6 2024-04-05T23:48:37.831 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:37 smithi016 systemd[1]: Stopping Ceph osd.6 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:48:37.831 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:37.775+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:37.831 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:37.775+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:37.831 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:37.775+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.710809+0000 front 2024-04-05T23:48:12.710764+0000 (oldest deadline 2024-04-05T23:48:36.810842+0000) 2024-04-05T23:48:38.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:37 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:37.894+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:38.510 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:37 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:37.894+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:38.511 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:37 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:37.901+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:38.511 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:37 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:37.901+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:38.511 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:37 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:37.901+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:38.512 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:37 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:37.952+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:38.512 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:37 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:37.952+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:38.512 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:37.883+0000 7f94c9600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:48:38.512 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:37.883+0000 7f94c9600700 -1 osd.6 104 *** Got signal Terminated *** 2024-04-05T23:48:38.513 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:37.883+0000 7f94c9600700 -1 osd.6 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:48:38.513 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:37.907+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:38.513 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:37.907+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:38.513 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:37 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:37.907+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:38.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:38 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:38.356+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:38.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:38 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:38.356+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:38.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:38 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:38.356+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:39.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:38 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:38.755+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:39.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:38 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:38.755+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:39.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:38 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:38.755+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.710809+0000 front 2024-04-05T23:48:12.710764+0000 (oldest deadline 2024-04-05T23:48:36.810842+0000) 2024-04-05T23:48:39.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:38 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:38.896+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:39.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:38 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:38.896+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:39.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:38 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:38.896+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:39.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:38 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:38.856+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:39.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:38 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:38.856+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:39.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:38 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:38.897+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:39.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:38 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:38.897+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:39.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:38 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:38.897+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:39.300 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:38 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:38.909+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:39.301 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:38 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:38.909+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:39.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:39 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:39.310+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:39.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:39 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:39.310+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:39.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:39 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:39.310+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:40.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:39 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:39.738+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:40.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:39 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:39.738+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:40.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:39 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:39.738+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.710809+0000 front 2024-04-05T23:48:12.710764+0000 (oldest deadline 2024-04-05T23:48:36.810842+0000) 2024-04-05T23:48:40.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:39 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:39.921+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:40.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:39 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:39.921+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:40.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:39 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:39.921+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:40.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:39 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:39.822+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:40.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:39 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:39.822+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:40.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:39 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:39.918+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:40.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:39 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:39.918+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:40.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:39 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:39.918+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:40.320 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:39 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:39.875+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:40.320 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:39 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:39.875+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:40.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:40 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:40.330+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:40.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:40 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:40.330+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:40.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:40 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:40.330+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:41.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:40 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:40.713+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:41.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:40 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:40.713+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:41.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:40 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:40.713+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.710809+0000 front 2024-04-05T23:48:12.710764+0000 (oldest deadline 2024-04-05T23:48:36.810842+0000) 2024-04-05T23:48:41.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:40 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:40.953+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:41.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:40 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:40.953+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:41.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:40 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:40.953+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:41.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:40 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:40.824+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:41.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:40 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:40.824+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:41.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:40 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:40.876+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:41.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:40 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:40.876+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:41.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:40 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:40.876+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:41.331 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:40 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:40.908+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:41.331 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:40 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:40.908+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:41.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:41 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:41.340+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:41.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:41 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:41.340+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:41.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:41 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:41.340+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:41.958 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:41 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:41.712+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:41.958 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:41 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:41.712+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:41.958 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:41 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:41.712+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.710809+0000 front 2024-04-05T23:48:12.710764+0000 (oldest deadline 2024-04-05T23:48:36.810842+0000) 2024-04-05T23:48:42.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:41 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:41.849+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:42.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:41 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:41.849+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:42.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:41 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:41.863+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:42.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:41 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:41.863+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:42.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:41 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:41.863+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:42.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:41 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:41.967+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:42.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:41 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:41.967+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:42.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:41 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:41.967+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:42.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:41 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:41.967+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:42.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:41 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:41.952+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:42.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:41 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:41.952+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:42.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:42 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:42.349+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:42.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:42 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:42.349+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:42.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:42 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:42.349+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:42.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:42 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:42.349+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:43.080 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:42 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:42.749+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:56.308191+0000 front 2024-04-05T23:47:56.307943+0000 (oldest deadline 2024-04-05T23:48:21.608113+0000) 2024-04-05T23:48:43.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:42 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:42.749+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.909574+0000 front 2024-04-05T23:48:03.909600+0000 (oldest deadline 2024-04-05T23:48:29.209781+0000) 2024-04-05T23:48:43.081 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:42 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6[73809]: 2024-04-05T23:48:42.749+0000 7f94c5800700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.710809+0000 front 2024-04-05T23:48:12.710764+0000 (oldest deadline 2024-04-05T23:48:36.810842+0000) 2024-04-05T23:48:43.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:42 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:42.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:43.082 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:42 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:42.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:43.082 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:42 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:42.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:43.082 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:42 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:42.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:43.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:42 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:42.828+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:43.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:42 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:42.828+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:43.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:42 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:42.828+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:19.576523+0000 front 2024-04-05T23:48:19.576540+0000 (oldest deadline 2024-04-05T23:48:41.876569+0000) 2024-04-05T23:48:43.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:42 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:42.912+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:43.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:42 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:42.912+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:43.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:42 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:42.912+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:43.324 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:42 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:42.969+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:43.324 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:42 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:42.969+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:43.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:43.332+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:43.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:43.332+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:43.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:43.332+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:43.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:43.332+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:43.830 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 05 23:48:43 smithi016 bash[116943]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-6 2024-04-05T23:48:44.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:43.800+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:44.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:43.800+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:44.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:43.800+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:19.576523+0000 front 2024-04-05T23:48:19.576540+0000 (oldest deadline 2024-04-05T23:48:41.876569+0000) 2024-04-05T23:48:44.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:43.908+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:44.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:43.908+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:44.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:43 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:43.908+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:44.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:44 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:44.030+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:44.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:44 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:44.030+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:44.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:44 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:44.030+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:44.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:44 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:44.030+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:44.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:43.968+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:44.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:43 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:43.968+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:44.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:44 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:44.374+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:44.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:44 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:44.374+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:44.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:44 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:44.374+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:44.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:44 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:44.374+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:45.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.6.service' 2024-04-05T23:48:45.179 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:48:45.179 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-05T23:48:45.179 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-05T23:48:45.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.7 2024-04-05T23:48:45.210 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:44 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:44.828+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:45.211 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:44 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:44.828+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:45.211 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:44 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:44.828+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:19.576523+0000 front 2024-04-05T23:48:19.576540+0000 (oldest deadline 2024-04-05T23:48:41.876569+0000) 2024-04-05T23:48:45.211 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:44 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:44.948+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:45.211 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:44 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:44.948+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:45.211 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:44 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:44.948+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:45.211 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:44 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:44.948+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:45.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:45 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:45.057+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:45.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:45 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:45.057+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:45.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:45 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:45.057+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:45.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:45 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:45.057+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:45.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:45 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:45.014+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:45.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:45 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:45.014+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:45.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:45 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:45.014+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:45.465 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:45 smithi027 systemd[1]: Stopping Ceph osd.7 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:48:45.465 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:45.428+0000 7f1f51864700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:48:45.465 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:45.428+0000 7f1f51864700 -1 osd.7 104 *** Got signal Terminated *** 2024-04-05T23:48:45.465 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:45.428+0000 7f1f51864700 -1 osd.7 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:48:45.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:45 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:45.362+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:45.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:45 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:45.362+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:45.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:45 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:45.362+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:45.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:45 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:45.362+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:46.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:45.799+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:46.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:45.799+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:46.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:45.799+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:19.576523+0000 front 2024-04-05T23:48:19.576540+0000 (oldest deadline 2024-04-05T23:48:41.876569+0000) 2024-04-05T23:48:46.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:45.937+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:46.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:45.937+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:46.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:45.937+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:46.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:45 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:45.937+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:46.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:46 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:46.032+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:46.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:46 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:46.032+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:46.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:46 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:46.032+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:46.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:46 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:46.032+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:46.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:46.025+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:46.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:46.025+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:46.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:46.025+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:46.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:46.345+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:46.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:46.345+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:46.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:46.345+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:46.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:46.345+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:47.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:46 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:46.771+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:47.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:46 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:46.771+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:47.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:46 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:46.771+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:19.576523+0000 front 2024-04-05T23:48:19.576540+0000 (oldest deadline 2024-04-05T23:48:41.876569+0000) 2024-04-05T23:48:47.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:46 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:46.961+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:47.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:46 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:46.961+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:47.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:46 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:46.961+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:47.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:46 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:46.961+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:47.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:47 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:47.028+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:47.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:47 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:47.028+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:47.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:47 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:47.028+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:47.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:47 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:47.028+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:47.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:46.975+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:47.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:46.975+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:47.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:46 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:46.975+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:47.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:47 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:47.366+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:47.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:47 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:47.366+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:47.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:47 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:47.366+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:47.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:47 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:47.366+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:48.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:47 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:47.775+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:48.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:47 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:47.775+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:48.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:47 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:47.775+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:19.576523+0000 front 2024-04-05T23:48:19.576540+0000 (oldest deadline 2024-04-05T23:48:41.876569+0000) 2024-04-05T23:48:48.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:47 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:47.942+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:48.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:47 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:47.942+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:48.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:47 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:47.942+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:48.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:47 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:47.942+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:48.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:48 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:48.032+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:48.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:48 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:48.032+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:48.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:48 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:48.032+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:48.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:48 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:48.032+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:48.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:47 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:47.935+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:48.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:47 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:47.935+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:48.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:47 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:47.935+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:48.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:48 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:48.363+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:48.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:48 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:48.363+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:48.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:48 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:48.363+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:48.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:48 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:48.363+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:49.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:48.742+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:49.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:48.742+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:49.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:48.742+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:19.576523+0000 front 2024-04-05T23:48:19.576540+0000 (oldest deadline 2024-04-05T23:48:41.876569+0000) 2024-04-05T23:48:49.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:48.939+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:49.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:48.939+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:49.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:48.939+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:49.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:48.939+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:49.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:49 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:49.027+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:49.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:49 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:49.027+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:49.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:49 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:49.027+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:49.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:49 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:49.027+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:49.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:48 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:48.885+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:49.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:48 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:48.885+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:49.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:48 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:48.885+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:49.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:49.341+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:49.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:49.341+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:49.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:49.341+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:49.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:49.341+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:49.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:49.341+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:50.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:49 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:49.739+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:00.470986+0000 front 2024-04-05T23:48:00.471009+0000 (oldest deadline 2024-04-05T23:48:26.371171+0000) 2024-04-05T23:48:50.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:49 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:49.739+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.074126+0000 front 2024-04-05T23:48:12.074094+0000 (oldest deadline 2024-04-05T23:48:34.974184+0000) 2024-04-05T23:48:50.215 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:49 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7[100893]: 2024-04-05T23:48:49.739+0000 7f1f4a653700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:19.576523+0000 front 2024-04-05T23:48:19.576540+0000 (oldest deadline 2024-04-05T23:48:41.876569+0000) 2024-04-05T23:48:50.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:49 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:49.942+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:50.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:49 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:49.942+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:50.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:49 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:49.942+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:50.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:49 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:49.942+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:50.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:50.019+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:50.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:50.019+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:50.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:50.019+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:50.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:50.019+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:50.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:49.931+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:50.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:49.931+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:50.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:49.931+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:50.332 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:49 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:49.931+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.210806+0000 front 2024-04-05T23:48:25.210745+0000 (oldest deadline 2024-04-05T23:48:49.910856+0000) 2024-04-05T23:48:50.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:50.357+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:50.832 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:50.357+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:50.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:50.357+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:50.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:50.358+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:50.833 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:50.358+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:51.214 INFO:journalctl@ceph.osd.7.smithi027.stdout:Apr 05 23:48:50 smithi027 bash[125163]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-7 2024-04-05T23:48:51.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:50 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:50.959+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:51.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:50 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:50.959+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:51.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:50 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:50.959+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:51.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:50 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:50.959+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:51.310 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:50.981+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:51.310 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:50.981+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:51.310 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:50.981+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:51.310 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:50 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:50.981+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.210806+0000 front 2024-04-05T23:48:25.210745+0000 (oldest deadline 2024-04-05T23:48:49.910856+0000) 2024-04-05T23:48:51.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:50.976+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:51.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:50.976+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:51.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:50.976+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:51.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:50 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:50.976+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:51.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:51 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:51.319+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:51.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:51 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:51.319+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:51.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:51 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:51.319+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:51.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:51 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:51.319+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:51.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:51 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:51.319+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:52.057 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:51 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:51.935+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:52.058 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:51 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:51.935+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:52.058 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:51 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:51.935+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:52.058 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:51 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:51.935+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:52.058 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:51 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:51.935+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:48:52.065 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.7.service' 2024-04-05T23:48:52.147 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:48:52.147 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-05T23:48:52.147 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-05T23:48:52.148 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.8 2024-04-05T23:48:52.327 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:51 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:51.972+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:52.327 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:51 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:51.972+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:52.327 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:51 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:51.972+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:52.327 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:51 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:51.972+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.210806+0000 front 2024-04-05T23:48:25.210745+0000 (oldest deadline 2024-04-05T23:48:49.910856+0000) 2024-04-05T23:48:52.328 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:52 smithi130 systemd[1]: Stopping Ceph osd.8 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:48:52.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:51 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:51.963+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:52.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:51 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:51.963+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:52.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:51 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:51.963+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:52.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:51 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:51.963+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:52.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:51 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:51.963+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:48:52.582 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:52.410+0000 7f26278f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:48:52.582 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:52.410+0000 7f26278f6700 -1 osd.8 104 *** Got signal Terminated *** 2024-04-05T23:48:52.582 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:52.410+0000 7f26278f6700 -1 osd.8 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:48:52.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:52.336+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:52.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:52.336+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:52.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:52.336+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:52.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:52.336+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:52.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:52.336+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:53.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:52 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:52.962+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:53.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:52 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:52.962+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:53.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:52 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:52.962+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:53.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:52 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:52.962+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:53.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:52 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:52.962+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:48:53.288 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:52.982+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:53.288 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:52.982+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:53.289 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:52.982+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:53.289 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:52.982+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.210806+0000 front 2024-04-05T23:48:25.210745+0000 (oldest deadline 2024-04-05T23:48:49.910856+0000) 2024-04-05T23:48:53.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:52 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:52.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:53.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:52 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:52.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:53.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:52 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:52.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:53.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:52 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:52.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:53.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:52 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:52.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:48:53.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:53.298+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:53.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:53.298+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:53.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:53.298+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:53.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:53.298+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:53.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:53 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:53.298+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:54.315 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:54.013+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:54.316 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:54.013+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:54.316 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:54.013+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:54.316 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:54.013+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.210806+0000 front 2024-04-05T23:48:25.210745+0000 (oldest deadline 2024-04-05T23:48:49.910856+0000) 2024-04-05T23:48:54.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:53.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:54.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:53.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:54.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:53.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:54.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:53.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:54.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:53 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:53.936+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:48:54.464 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:53 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:53.987+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:54.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:53 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:53.987+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:54.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:53 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:53.987+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:54.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:53 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:53.987+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:54.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:53 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:53.987+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:48:54.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:54.324+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:54.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:54.325+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:54.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:54.325+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:54.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:54.325+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:54.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:54.325+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:55.283 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:54.977+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:55.283 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:54.977+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:55.283 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:54.977+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:55.283 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:54 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:54.977+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.210806+0000 front 2024-04-05T23:48:25.210745+0000 (oldest deadline 2024-04-05T23:48:49.910856+0000) 2024-04-05T23:48:55.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:54 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:54.956+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:55.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:54 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:54.956+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:55.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:54 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:54.956+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:55.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:54 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:54.956+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:55.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:54 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:54.956+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:48:55.464 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:54 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:54.978+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:55.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:54 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:54.978+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:55.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:54 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:54.978+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:55.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:54 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:54.978+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:55.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:54 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:54.978+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:48:55.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:55 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:55.292+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:55.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:55 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:55.292+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:55.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:55 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:55.292+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:55.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:55 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:55.292+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:55.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:55 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:55.292+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:56.249 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:55 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:55.950+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:56.249 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:55 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:55.950+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:56.250 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:55 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:55.950+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:56.250 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:55 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:55.950+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.210806+0000 front 2024-04-05T23:48:25.210745+0000 (oldest deadline 2024-04-05T23:48:49.910856+0000) 2024-04-05T23:48:56.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:55 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:55.965+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:56.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:55 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:55.965+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:56.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:55 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:55.965+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:56.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:55 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:55.965+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:56.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:55 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:55.965+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:48:56.464 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:55 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:55.992+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:56.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:55 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:55.992+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:56.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:55 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:55.992+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:56.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:55 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:55.992+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:56.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:55 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:55.992+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:48:56.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:56.258+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:56.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:56.258+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:56.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:56.258+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:56.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:56.258+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:56.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:56.258+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:57.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:56 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:56.947+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:57.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:56 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:56.947+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:57.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:56 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:56.947+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:57.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:56 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:56.947+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:57.332 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:56 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:56.947+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:48:57.332 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:56 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:56.947+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.484213+0000 front 2024-04-05T23:48:34.484389+0000 (oldest deadline 2024-04-05T23:48:56.784195+0000) 2024-04-05T23:48:57.333 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:56.998+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.805679+0000 front 2024-04-05T23:47:57.805825+0000 (oldest deadline 2024-04-05T23:48:21.905591+0000) 2024-04-05T23:48:57.333 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:56.998+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:13.008249+0000 front 2024-04-05T23:48:13.008268+0000 (oldest deadline 2024-04-05T23:48:35.308453+0000) 2024-04-05T23:48:57.333 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:56.998+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:15.309090+0000 front 2024-04-05T23:48:15.309098+0000 (oldest deadline 2024-04-05T23:48:44.109627+0000) 2024-04-05T23:48:57.334 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8[102580]: 2024-04-05T23:48:56.998+0000 7f26246ed700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.210806+0000 front 2024-04-05T23:48:25.210745+0000 (oldest deadline 2024-04-05T23:48:49.910856+0000) 2024-04-05T23:48:57.334 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:57 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:57.221+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:57.334 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:57 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:57.221+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:57.335 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:57 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:57.221+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:57.335 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:57 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:57.221+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:57.335 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:57 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:57.221+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:57.464 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.004+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:57.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.004+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:57.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.004+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:57.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.004+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:57.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.004+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:48:57.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.004+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:48:58.214 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:58.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:58.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:58.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:58.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:48:58.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:57 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:57.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:48:58.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:57 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:57.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:58.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:57 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:57.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:58.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:57 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:57.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:58.332 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:57 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:57.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:58.332 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:57 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:57.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:48:58.332 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:57 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:57.981+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.484213+0000 front 2024-04-05T23:48:34.484389+0000 (oldest deadline 2024-04-05T23:48:56.784195+0000) 2024-04-05T23:48:58.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:58 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:58.184+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:58.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:58 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:58.184+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:58.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:58 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:58.184+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:58.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:58 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:58.184+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:58.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:58 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:58.184+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:58.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:58 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:58.184+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:48:58.334 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 05 23:48:58 smithi130 bash[125772]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-8 2024-04-05T23:48:59.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:58 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:58.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:48:59.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:58 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:58.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:48:59.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:58 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:58.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:48:59.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:58 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:58.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:48:59.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:58 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:58.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:48:59.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:58 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:58.973+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:48:59.322 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:59 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:59.151+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:48:59.323 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:59 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:59.151+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:48:59.323 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:59 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:59.151+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:48:59.323 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:59 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:59.151+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:48:59.323 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:59 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:59.151+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:48:59.323 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:48:59 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:48:59.151+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:48:59.330 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.8.service' 2024-04-05T23:48:59.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:58 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:58.992+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:48:59.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:58 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:58.992+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:48:59.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:58 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:58.992+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:48:59.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:58 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:58.992+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:48:59.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:58 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:58.992+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:48:59.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:58 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:58.992+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.484213+0000 front 2024-04-05T23:48:34.484389+0000 (oldest deadline 2024-04-05T23:48:56.784195+0000) 2024-04-05T23:48:59.371 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:48:59.371 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-05T23:48:59.371 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-05T23:48:59.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.9 2024-04-05T23:48:59.762 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 systemd[1]: Stopping Ceph osd.9 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:00.080 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:59.771+0000 7ff415800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:00.080 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:59.771+0000 7ff415800700 -1 osd.9 104 *** Got signal Terminated *** 2024-04-05T23:49:00.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:59.771+0000 7ff415800700 -1 osd.9 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:49:00.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:59.958+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:49:00.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:59.958+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:49:00.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:59.958+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:49:00.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:59.958+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:49:00.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:59.958+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:49:00.081 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:48:59 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:48:59.958+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.484213+0000 front 2024-04-05T23:48:34.484389+0000 (oldest deadline 2024-04-05T23:48:56.784195+0000) 2024-04-05T23:49:00.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:59 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:59.956+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:00.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:59 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:59.956+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:00.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:59 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:59.956+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:00.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:59 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:59.956+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:00.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:59 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:59.956+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:00.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:48:59 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:48:59.956+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:00.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:00 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:00.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:00.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:00 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:00.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:00.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:00 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:00.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:00.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:00 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:00.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:00.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:00 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:00.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:00.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:00 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:00.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:01.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:00 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:00.932+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:01.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:00 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:00.932+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:01.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:00 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:00.932+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:01.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:00 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:00.932+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:01.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:00 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:00.932+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:01.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:00 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:00.932+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:01.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:00.913+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:49:01.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:00.913+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:49:01.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:00.913+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:49:01.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:00.913+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:49:01.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:00.913+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:49:01.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:00 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:00.913+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.484213+0000 front 2024-04-05T23:48:34.484389+0000 (oldest deadline 2024-04-05T23:48:56.784195+0000) 2024-04-05T23:49:01.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:01 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:01.170+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:01.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:01 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:01.170+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:01.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:01 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:01.170+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:01.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:01 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:01.170+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:01.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:01 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:01.170+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:01.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:01 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:01.170+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:02.214 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:01 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:01.885+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:02.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:01 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:01.885+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:02.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:01 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:01.885+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:02.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:01 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:01.885+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:02.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:01 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:01.885+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:02.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:01 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:01.885+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:02.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:01 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:01.919+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:49:02.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:01 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:01.919+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:49:02.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:01 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:01.919+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:49:02.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:01 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:01.919+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:49:02.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:01 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:01.919+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:49:02.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:01 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:01.919+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.484213+0000 front 2024-04-05T23:48:34.484389+0000 (oldest deadline 2024-04-05T23:48:56.784195+0000) 2024-04-05T23:49:02.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:02 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:02.192+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:02.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:02 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:02.192+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:02.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:02 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:02.192+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:02.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:02 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:02.192+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:02.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:02 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:02.192+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:02.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:02 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:02.192+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:03.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:02 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:02.851+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:03.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:02 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:02.851+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:03.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:02 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:02.851+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:03.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:02 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:02.851+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:03.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:02 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:02.851+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:03.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:02 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:02.851+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:03.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:02 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:02.877+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:49:03.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:02 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:02.877+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:49:03.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:02 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:02.877+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:49:03.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:02 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:02.877+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:49:03.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:02 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:02.877+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:49:03.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:02 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:02.877+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.484213+0000 front 2024-04-05T23:48:34.484389+0000 (oldest deadline 2024-04-05T23:48:56.784195+0000) 2024-04-05T23:49:03.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:03.162+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:03.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:03.162+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:03.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:03.162+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:03.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:03.162+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:03.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:03.162+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:03.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:03 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:03.162+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:04.214 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:03 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:03.858+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:04.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:03 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:03.858+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:04.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:03 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:03.858+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:04.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:03 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:03.858+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:04.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:03 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:03.858+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:04.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:03 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:03.858+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:04.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:03.889+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:58.078830+0000 front 2024-04-05T23:47:58.078864+0000 (oldest deadline 2024-04-05T23:48:23.979011+0000) 2024-04-05T23:49:04.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:03.889+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:03.979550+0000 front 2024-04-05T23:48:03.979653+0000 (oldest deadline 2024-04-05T23:48:29.279539+0000) 2024-04-05T23:49:04.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:03.889+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.180897+0000 front 2024-04-05T23:48:12.180938+0000 (oldest deadline 2024-04-05T23:48:36.280984+0000) 2024-04-05T23:49:04.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:03.889+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:16.281430+0000 front 2024-04-05T23:48:16.281570+0000 (oldest deadline 2024-04-05T23:48:41.581625+0000) 2024-04-05T23:49:04.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:03.889+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:25.682925+0000 front 2024-04-05T23:48:25.682963+0000 (oldest deadline 2024-04-05T23:48:51.583103+0000) 2024-04-05T23:49:04.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:03 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9[81938]: 2024-04-05T23:49:03.889+0000 7ff411a00700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.484213+0000 front 2024-04-05T23:48:34.484389+0000 (oldest deadline 2024-04-05T23:48:56.784195+0000) 2024-04-05T23:49:04.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:04 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:04.180+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:04.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:04 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:04.180+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:04.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:04 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:04.180+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:04.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:04 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:04.180+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:04.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:04 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:04.180+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:04.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:04 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:04.180+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:05.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:04 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:04.843+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:05.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:04 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:04.843+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:05.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:04 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:04.843+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:05.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:04 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:04.843+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:05.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:04 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:04.843+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:05.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:04 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:04.843+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:05.581 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 05 23:49:05 smithi016 bash[117201]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-9 2024-04-05T23:49:05.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:05 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:05.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:05.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:05 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:05.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:05.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:05 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:05.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:05.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:05 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:05.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:05.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:05 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:05.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:05.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:05 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:05.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:06.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:05 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:05.880+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:06.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:05 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:05.880+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:06.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:05 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:05.880+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:06.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:05 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:05.880+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:06.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:05 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:05.880+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:06.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:05 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:05.880+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:06.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:06 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:06.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:06.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:06 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:06.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:06.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:06 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:06.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:06.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:06 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:06.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:06.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:06 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:06.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:06.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:06 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:06.182+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:06.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.9.service' 2024-04-05T23:49:07.037 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:07.038 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-05T23:49:07.038 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-05T23:49:07.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.10 2024-04-05T23:49:07.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:06 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:06.915+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:07.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:06 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:06.915+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:07.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:06 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:06.915+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:07.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:06 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:06.915+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:07.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:06 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:06.915+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:07.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:06 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:06.915+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:07.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:06 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:06.915+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-05T23:48:40.961207+0000 front 2024-04-05T23:48:40.961138+0000 (oldest deadline 2024-04-05T23:49:06.161687+0000) 2024-04-05T23:49:07.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 systemd[1]: Stopping Ceph osd.10 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:07.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:07 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:07.176+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:07.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:07 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:07.176+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:07.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:07 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:07.176+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:07.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:07 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:07.176+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:07.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:07 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:07.176+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:07.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:07 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:07.176+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:07.714 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.292+0000 7f5778205700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:07.715 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.292+0000 7f5778205700 -1 osd.10 104 *** Got signal Terminated *** 2024-04-05T23:49:07.715 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.292+0000 7f5778205700 -1 osd.10 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:49:08.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.878+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:08.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.878+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:08.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.878+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:08.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.878+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:08.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.878+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:08.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.878+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:08.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:07 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:07.879+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-05T23:48:40.961207+0000 front 2024-04-05T23:48:40.961138+0000 (oldest deadline 2024-04-05T23:49:06.161687+0000) 2024-04-05T23:49:08.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:08 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:08.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:08.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:08 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:08.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:08.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:08 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:08.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:08.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:08 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:08.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:08.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:08 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:08.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:08.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:08 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:08.132+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:09.214 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:08 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:08.830+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:09.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:08 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:08.830+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:09.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:08 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:08.830+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:09.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:08 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:08.830+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:09.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:08 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:08.830+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:09.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:08 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:08.830+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:09.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:08 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:08.830+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-05T23:48:40.961207+0000 front 2024-04-05T23:48:40.961138+0000 (oldest deadline 2024-04-05T23:49:06.161687+0000) 2024-04-05T23:49:09.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:09 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:09.102+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:09.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:09 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:09.102+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:09.582 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:09 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:09.102+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:09.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:09 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:09.102+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:09.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:09 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:09.102+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:09.583 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:09 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:09.102+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:10.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:09 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:09.877+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:10.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:09 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:09.877+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:10.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:09 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:09.877+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:10.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:09 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:09.877+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:10.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:09 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:09.877+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:10.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:09 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:09.877+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:10.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:09 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:09.877+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-05T23:48:40.961207+0000 front 2024-04-05T23:48:40.961138+0000 (oldest deadline 2024-04-05T23:49:06.161687+0000) 2024-04-05T23:49:10.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:10 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:10.074+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:10.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:10 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:10.074+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:10.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:10 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:10.074+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:10.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:10 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:10.074+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:10.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:10 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:10.074+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:10.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:10 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:10.074+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:11.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:10.905+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:11.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:10.905+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:11.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:10.905+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:11.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:10.905+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:11.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:10.905+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:11.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:10.905+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:11.216 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:10 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:10.905+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-05T23:48:40.961207+0000 front 2024-04-05T23:48:40.961138+0000 (oldest deadline 2024-04-05T23:49:06.161687+0000) 2024-04-05T23:49:11.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:11 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:11.042+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:11.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:11 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:11.042+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:11.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:11 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:11.042+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:11.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:11 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:11.042+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:11.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:11 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:11.042+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:11.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:11 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:11.042+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:12.214 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:11 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:11.894+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:53.951253+0000 front 2024-04-05T23:47:53.951205+0000 (oldest deadline 2024-04-05T23:48:19.251346+0000) 2024-04-05T23:49:12.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:11 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:11.894+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.653278+0000 front 2024-04-05T23:48:05.653246+0000 (oldest deadline 2024-04-05T23:48:29.153456+0000) 2024-04-05T23:49:12.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:11 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:11.894+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:12.054793+0000 front 2024-04-05T23:48:12.054687+0000 (oldest deadline 2024-04-05T23:48:35.554842+0000) 2024-04-05T23:49:12.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:11 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:11.894+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.156810+0000 front 2024-04-05T23:48:20.156878+0000 (oldest deadline 2024-04-05T23:48:44.856833+0000) 2024-04-05T23:49:12.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:11 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:11.894+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.157637+0000 front 2024-04-05T23:48:27.157723+0000 (oldest deadline 2024-04-05T23:48:51.857828+0000) 2024-04-05T23:49:12.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:11 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:11.894+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:32.959173+0000 front 2024-04-05T23:48:32.959050+0000 (oldest deadline 2024-04-05T23:48:56.959472+0000) 2024-04-05T23:49:12.215 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:11 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10[109699]: 2024-04-05T23:49:11.894+0000 7f5774ffc700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-05T23:48:40.961207+0000 front 2024-04-05T23:48:40.961138+0000 (oldest deadline 2024-04-05T23:49:06.161687+0000) 2024-04-05T23:49:12.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:12 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:12.055+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:12.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:12 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:12.055+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:12.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:12 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:12.055+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:12.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:12 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:12.055+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:12.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:12 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:12.055+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:12.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:12 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:12.055+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:13.334 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:13 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:13.070+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:13.334 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:13 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:13.070+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:13.334 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:13 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:13.070+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:13.335 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:13 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:13.070+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:13.335 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:13 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:13.070+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:13.335 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:13 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:13.070+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:13.465 INFO:journalctl@ceph.osd.10.smithi027.stdout:Apr 05 23:49:13 smithi027 bash[125431]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-10 2024-04-05T23:49:14.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.10.service' 2024-04-05T23:49:14.253 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:14.253 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-05T23:49:14.254 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-05T23:49:14.254 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.11 2024-04-05T23:49:14.514 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:14.101+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:14.515 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:14.101+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:14.515 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:14.101+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:14.515 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:14.101+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:14.515 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:14.101+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:14.515 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:14.101+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:14.515 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 systemd[1]: Stopping Ceph osd.11 for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:14.515 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:14.523+0000 7f3662b02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:14.516 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:14.523+0000 7f3662b02700 -1 osd.11 104 *** Got signal Terminated *** 2024-04-05T23:49:14.516 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:14 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:14.523+0000 7f3662b02700 -1 osd.11 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T23:49:15.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:15 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:15.056+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:15.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:15 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:15.056+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:15.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:15 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:15.056+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:15.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:15 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:15.056+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:15.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:15 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:15.056+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:15.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:15 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:15.056+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:15.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:15 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:15.056+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6820 osd.7 since back 2024-04-05T23:48:48.569434+0000 front 2024-04-05T23:48:48.569328+0000 (oldest deadline 2024-04-05T23:49:14.469186+0000) 2024-04-05T23:49:16.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:16 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:16.089+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:16.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:16 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:16.089+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:16.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:16 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:16.089+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:16.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:16 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:16.089+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:16.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:16 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:16.089+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:16.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:16 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:16.089+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:16.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:16 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:16.089+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6820 osd.7 since back 2024-04-05T23:48:48.569434+0000 front 2024-04-05T23:48:48.569328+0000 (oldest deadline 2024-04-05T23:49:14.469186+0000) 2024-04-05T23:49:17.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:17 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:17.040+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:17.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:17 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:17.040+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:17.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:17 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:17.040+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:17.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:17 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:17.040+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:17.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:17 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:17.040+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:17.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:17 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:17.040+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:17.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:17 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:17.040+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6820 osd.7 since back 2024-04-05T23:48:48.569434+0000 front 2024-04-05T23:48:48.569328+0000 (oldest deadline 2024-04-05T23:49:14.469186+0000) 2024-04-05T23:49:18.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:18 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:18.022+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:18.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:18 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:18.022+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:18.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:18 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:18.022+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:18.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:18 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:18.022+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:18.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:18 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:18.022+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:18.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:18 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:18.022+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:18.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:18 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:18.022+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6820 osd.7 since back 2024-04-05T23:48:48.569434+0000 front 2024-04-05T23:48:48.569328+0000 (oldest deadline 2024-04-05T23:49:14.469186+0000) 2024-04-05T23:49:19.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:19 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:19.060+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-05T23:47:57.459275+0000 front 2024-04-05T23:47:57.459289+0000 (oldest deadline 2024-04-05T23:48:22.759413+0000) 2024-04-05T23:49:19.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:19 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:19.060+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6804 osd.1 since back 2024-04-05T23:48:05.060910+0000 front 2024-04-05T23:48:05.060957+0000 (oldest deadline 2024-04-05T23:48:29.160984+0000) 2024-04-05T23:49:19.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:19 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:19.060+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-05T23:48:09.161473+0000 front 2024-04-05T23:48:09.161496+0000 (oldest deadline 2024-04-05T23:48:33.861643+0000) 2024-04-05T23:49:19.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:19 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:19.060+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-05T23:48:20.263454+0000 front 2024-04-05T23:48:20.263434+0000 (oldest deadline 2024-04-05T23:48:41.363306+0000) 2024-04-05T23:49:19.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:19 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:19.060+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6812 osd.4 since back 2024-04-05T23:48:27.566069+0000 front 2024-04-05T23:48:27.565995+0000 (oldest deadline 2024-04-05T23:48:49.266065+0000) 2024-04-05T23:49:19.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:19 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:19.060+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-05T23:48:34.567081+0000 front 2024-04-05T23:48:34.567169+0000 (oldest deadline 2024-04-05T23:48:58.067357+0000) 2024-04-05T23:49:19.333 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:19 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11[110765]: 2024-04-05T23:49:19.060+0000 7f365f8f9700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.27:6820 osd.7 since back 2024-04-05T23:48:48.569434+0000 front 2024-04-05T23:48:48.569328+0000 (oldest deadline 2024-04-05T23:49:14.469186+0000) 2024-04-05T23:49:20.332 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 05 23:49:20 smithi130 bash[126027]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-osd-11 2024-04-05T23:49:21.268 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@osd.11.service' 2024-04-05T23:49:21.308 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:21.308 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-05T23:49:21.308 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-05T23:49:21.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.a 2024-04-05T23:49:21.694 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:49:21 smithi016 systemd[1]: Stopping Ceph mds.a for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:22.080 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:49:21 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-a[90400]: 2024-04-05T23:49:21.703+0000 7f70eba00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:22.080 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:49:21 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-a[90400]: 2024-04-05T23:49:21.703+0000 7f70eba00700 -1 mds.a *** got signal Terminated *** 2024-04-05T23:49:24.080 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 05 23:49:23 smithi016 bash[117471]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-a 2024-04-05T23:49:24.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.a.service' 2024-04-05T23:49:24.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:24.342 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-05T23:49:24.342 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-05T23:49:24.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.d 2024-04-05T23:49:24.580 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:49:24 smithi016 systemd[1]: Stopping Ceph mds.d for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:25.080 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:49:24 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-d[89447]: 2024-04-05T23:49:24.710+0000 7fe2d0800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:25.080 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:49:24 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-d[89447]: 2024-04-05T23:49:24.710+0000 7fe2d0800700 -1 mds.d *** got signal Terminated *** 2024-04-05T23:49:28.081 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 05 23:49:27 smithi016 bash[117610]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-d 2024-04-05T23:49:28.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.d.service' 2024-04-05T23:49:28.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:28.303 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-05T23:49:28.303 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-05T23:49:28.303 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.g 2024-04-05T23:49:28.580 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 05 23:49:28 smithi016 systemd[1]: Stopping Ceph mds.g for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:29.080 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 05 23:49:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-g[90014]: 2024-04-05T23:49:28.710+0000 7efec9c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:29.080 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 05 23:49:28 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-g[90014]: 2024-04-05T23:49:28.710+0000 7efec9c00700 -1 mds.g *** got signal Terminated *** 2024-04-05T23:49:33.330 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 05 23:49:32 smithi016 bash[117748]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-g 2024-04-05T23:49:33.361 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.g.service' 2024-04-05T23:49:33.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:33.441 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-05T23:49:33.441 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-05T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.j 2024-04-05T23:49:33.791 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 05 23:49:33 smithi016 systemd[1]: Stopping Ceph mds.j for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:34.080 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 05 23:49:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-j[89059]: 2024-04-05T23:49:33.801+0000 7f8d19e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:34.080 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 05 23:49:33 smithi016 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-j[89059]: 2024-04-05T23:49:33.801+0000 7f8d19e00700 -1 mds.j *** got signal Terminated *** 2024-04-05T23:49:35.764 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 05 23:49:35 smithi016 bash[117888]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-j 2024-04-05T23:49:35.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.j.service' 2024-04-05T23:49:35.871 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:35.871 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-05T23:49:35.871 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-05T23:49:35.872 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.b 2024-04-05T23:49:36.214 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:49:35 smithi027 systemd[1]: Stopping Ceph mds.b for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:36.215 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:49:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-b[116618]: 2024-04-05T23:49:36.138+0000 7f5ee4279700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:36.215 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:49:36 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-b[116618]: 2024-04-05T23:49:36.138+0000 7f5ee4279700 -1 mds.b *** got signal Terminated *** 2024-04-05T23:49:40.020 INFO:journalctl@ceph.mds.b.smithi027.stdout:Apr 05 23:49:39 smithi027 bash[125689]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-b 2024-04-05T23:49:40.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.b.service' 2024-04-05T23:49:40.063 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:40.063 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-05T23:49:40.063 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-05T23:49:40.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.e 2024-04-05T23:49:40.320 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:49:40 smithi027 systemd[1]: Stopping Ceph mds.e for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:40.714 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:49:40 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-e[115929]: 2024-04-05T23:49:40.329+0000 7f58ecf24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:40.715 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:49:40 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-e[115929]: 2024-04-05T23:49:40.329+0000 7f58ecf24700 -1 mds.e *** got signal Terminated *** 2024-04-05T23:49:43.964 INFO:journalctl@ceph.mds.e.smithi027.stdout:Apr 05 23:49:43 smithi027 bash[125828]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-e 2024-04-05T23:49:44.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.e.service' 2024-04-05T23:49:44.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:44.047 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-05T23:49:44.047 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-05T23:49:44.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.h 2024-04-05T23:49:44.302 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:49:44 smithi027 systemd[1]: Stopping Ceph mds.h for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:44.714 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:49:44 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-h[116269]: 2024-04-05T23:49:44.310+0000 7f88998d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:44.715 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:49:44 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-h[116269]: 2024-04-05T23:49:44.310+0000 7f88998d8700 -1 mds.h *** got signal Terminated *** 2024-04-05T23:49:48.072 INFO:journalctl@ceph.mds.h.smithi027.stdout:Apr 05 23:49:47 smithi027 bash[125967]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-h 2024-04-05T23:49:48.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.h.service' 2024-04-05T23:49:48.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:48.114 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-05T23:49:48.114 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-05T23:49:48.114 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.k 2024-04-05T23:49:48.368 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:49:48 smithi027 systemd[1]: Stopping Ceph mds.k for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:48.714 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:49:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-k[115586]: 2024-04-05T23:49:48.377+0000 7f06ee6b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:48.715 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:49:48 smithi027 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-k[115586]: 2024-04-05T23:49:48.377+0000 7f06ee6b8700 -1 mds.k *** got signal Terminated *** 2024-04-05T23:49:51.964 INFO:journalctl@ceph.mds.k.smithi027.stdout:Apr 05 23:49:51 smithi027 bash[126105]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-k 2024-04-05T23:49:52.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.k.service' 2024-04-05T23:49:52.071 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:52.071 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-05T23:49:52.072 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-05T23:49:52.072 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.c 2024-04-05T23:49:52.582 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:49:52 smithi130 systemd[1]: Stopping Ceph mds.c for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:52.582 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:49:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: 2024-04-05T23:49:52.326+0000 7f8bcf9bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:52.582 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:49:52 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c[117608]: 2024-04-05T23:49:52.326+0000 7f8bcf9bb700 -1 mds.c *** got signal Terminated *** 2024-04-05T23:49:56.082 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 05 23:49:55 smithi130 bash[126291]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-c 2024-04-05T23:49:56.089 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.c.service' 2024-04-05T23:49:56.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:49:56.127 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-05T23:49:56.127 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-05T23:49:56.128 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.f 2024-04-05T23:49:56.408 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:49:56 smithi130 systemd[1]: Stopping Ceph mds.f for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:49:56.832 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:49:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: 2024-04-05T23:49:56.417+0000 7fd43a79d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:49:56.832 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:49:56 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f[120998]: 2024-04-05T23:49:56.417+0000 7fd43a79d700 -1 mds.f *** got signal Terminated *** 2024-04-05T23:50:00.082 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 05 23:49:59 smithi130 bash[126429]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-f 2024-04-05T23:50:00.092 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.f.service' 2024-04-05T23:50:00.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:50:00.127 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-05T23:50:00.128 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-05T23:50:00.128 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.i 2024-04-05T23:50:00.399 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:50:00 smithi130 systemd[1]: Stopping Ceph mds.i for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:50:00.832 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:50:00 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-i[117948]: 2024-04-05T23:50:00.408+0000 7f8f81490700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:50:00.832 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:50:00 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-i[117948]: 2024-04-05T23:50:00.408+0000 7f8f81490700 -1 mds.i *** got signal Terminated *** 2024-04-05T23:50:04.082 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 05 23:50:03 smithi130 bash[126570]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-i 2024-04-05T23:50:04.089 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.i.service' 2024-04-05T23:50:04.124 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:50:04.125 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-05T23:50:04.125 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-05T23:50:04.125 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.l 2024-04-05T23:50:04.398 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:50:04 smithi130 systemd[1]: Stopping Ceph mds.l for b3471b30-f3a3-11ee-b649-cb9ed24678a4... 2024-04-05T23:50:04.832 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:50:04 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-l[116913]: 2024-04-05T23:50:04.407+0000 7f2a3df16700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T23:50:04.832 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:50:04 smithi130 ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-l[116913]: 2024-04-05T23:50:04.407+0000 7f2a3df16700 -1 mds.l *** got signal Terminated *** 2024-04-05T23:50:08.071 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 05 23:50:07 smithi130 bash[126709]: ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4-mds-l 2024-04-05T23:50:08.076 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b3471b30-f3a3-11ee-b649-cb9ed24678a4@mds.l.service' 2024-04-05T23:50:08.110 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T23:50:08.110 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-05T23:50:08.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-05T23:50:08.380 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:50:11.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-05T23:50:11.541 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:50:13.023 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-05T23:50:13.208 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:50:14.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T23:50:14.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T23:50:14.732 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T23:50:14.763 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-05T23:50:14.765 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/remote/smithi016/crash 2024-04-05T23:50:14.766 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/crash -- . 2024-04-05T23:50:14.851 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-05T23:50:14.851 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-04-05T23:50:14.854 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/remote/smithi027/crash 2024-04-05T23:50:14.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/crash -- . 2024-04-05T23:50:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-05T23:50:14.896 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-04-05T23:50:14.898 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/remote/smithi130/crash 2024-04-05T23:50:14.899 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/crash -- . 2024-04-05T23:50:14.929 INFO:teuthology.orchestra.run.smithi130.stderr:tar: /var/lib/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-05T23:50:14.930 INFO:teuthology.orchestra.run.smithi130.stderr:tar: Error is not recoverable: exiting now 2024-04-05T23:50:14.931 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-05T23:50:14.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-05T23:50:15.012 INFO:teuthology.orchestra.run.smithi016.stderr:grep: /var/log/ceph/b3471b30-f3a3-11ee-b649-cb9ed24678a4/ceph.log: No such file or directory 2024-04-05T23:50:15.018 INFO:tasks.cephadm:Compressing logs... 2024-04-05T23:50:15.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T23:50:15.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T23:50:15.025 DEBUG:teuthology.orchestra.run.smithi130:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T23:50:15.053 INFO:teuthology.orchestra.run.smithi130.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-05T23:50:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-05T23:50:15.083 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-05T23:52:00.348 INFO:tasks.cephadm:Archiving logs... 2024-04-05T23:52:00.349 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/remote/smithi016/log 2024-04-05T23:52:00.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-05T23:52:06.274 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/remote/smithi027/log 2024-04-05T23:52:06.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-05T23:52:17.050 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/remote/smithi130/log 2024-04-05T23:52:17.051 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-05T23:52:29.398 INFO:tasks.cephadm:Removing cluster... 2024-04-05T23:52:29.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 --force 2024-04-05T23:52:29.682 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:52:30.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 --force 2024-04-05T23:52:30.456 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:52:30.791 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b3471b30-f3a3-11ee-b649-cb9ed24678a4 --force 2024-04-05T23:52:30.982 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: b3471b30-f3a3-11ee-b649-cb9ed24678a4 2024-04-05T23:52:31.318 INFO:tasks.cephadm:Removing cephadm ... 2024-04-05T23:52:31.318 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-05T23:52:31.367 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-05T23:52:31.387 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-05T23:52:31.405 INFO:tasks.cephadm:Teardown complete 2024-04-05T23:52:31.405 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-05T23:52:31.418 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-05T23:52:31.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-05T23:52:31.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-05T23:52:31.430 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-05T23:52:31.565 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-05T23:52:31.576 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-05T23:52:31.576 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T23:52:31.581 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T23:52:31.583 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T23:52:31.598 INFO:teuthology.orchestra.run.smithi130.stderr:bash: ntpq: command not found 2024-04-05T23:52:31.599 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2024-04-05T23:52:31.603 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T23:52:31.603 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-04-05T23:52:31.603 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 -1091us[-1091us] +/- 85ms 2024-04-05T23:52:31.603 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv02.front.sepia.ceph.com 3 6 377 23 -696us[ -684us] +/- 51ms 2024-04-05T23:52:31.604 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +2208us[+2208us] +/- 87ms 2024-04-05T23:52:31.604 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:52:31.605 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T23:52:31.605 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-05T23:52:31.605 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 -1077us[-1080us] +/- 85ms 2024-04-05T23:52:31.605 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -669us[ -672us] +/- 51ms 2024-04-05T23:52:31.605 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +2308us[+2305us] +/- 87ms 2024-04-05T23:52:31.606 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:52:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:bash: ntpq: command not found 2024-04-05T23:52:31.625 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T23:52:31.625 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-05T23:52:31.625 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -1163us[-1163us] +/- 85ms 2024-04-05T23:52:31.625 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv02.front.sepia.ceph.com 3 7 377 89 -732us[ -766us] +/- 50ms 2024-04-05T23:52:31.625 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +2216us[+2216us] +/- 87ms 2024-04-05T23:52:31.625 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T23:52:31.626 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-05T23:52:31.637 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-05T23:52:31.638 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-05T23:52:31.675 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-05T23:52:31.814 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-05T23:52:31.896 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-05T23:52:31.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T23:52:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:52:32.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T23:52:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:52:32.070 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T23:52:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:52:32.112 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-05T23:52:32.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-05T23:52:32.154 INFO:teuthology.task.internal:Duration was 2568.658358 seconds 2024-04-05T23:52:32.154 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-05T23:52:32.204 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-05T23:52:32.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-05T23:52:32.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-05T23:52:32.212 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-05T23:52:32.259 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T23:52:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T23:52:32.318 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T23:52:32.728 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-05T23:52:32.729 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-04-05T23:52:32.729 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-05T23:52:32.785 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-04-05T23:52:32.786 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-05T23:52:32.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-04-05T23:52:32.813 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-05T23:52:32.836 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-05T23:52:32.836 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T23:52:32.840 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T23:52:32.856 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T23:52:32.936 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-05T23:52:32.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-05T23:52:32.941 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-05T23:52:32.943 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-05T23:52:33.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-05T23:52:33.113 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-05T23:52:33.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-05T23:52:33.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-05T23:52:33.209 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-05T23:52:33.236 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-05T23:52:33.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-05T23:52:33.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-05T23:52:33.256 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-05T23:52:33.281 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-04-05T23:52:33.302 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-04-05T23:52:33.314 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-04-05T23:52:33.349 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-05T23:52:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:52:33.388 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-05T23:52:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:52:33.405 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-05T23:52:33.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T23:52:33.421 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-05T23:52:33.433 INFO:teuthology.task.internal:Transferring archived files... 2024-04-05T23:52:33.435 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/remote/smithi016 2024-04-05T23:52:33.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-05T23:52:33.519 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/remote/smithi027 2024-04-05T23:52:33.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-05T23:52:33.561 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731/remote/smithi130 2024-04-05T23:52:33.561 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-05T23:52:33.602 INFO:teuthology.task.internal:Removing archive directory... 2024-04-05T23:52:33.603 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-05T23:52:33.607 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-05T23:52:33.609 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-05T23:52:33.652 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-05T23:52:33.665 INFO:teuthology.task.internal:Not uploading archives. 2024-04-05T23:52:33.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-05T23:52:33.696 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-05T23:52:33.696 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-05T23:52:33.701 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-05T23:52:33.703 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-05T23:52:33.717 INFO:teuthology.orchestra.run.smithi027.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 5 23:52 /home/ubuntu/cephtest 2024-04-05T23:52:33.719 INFO:teuthology.orchestra.run.smithi130.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 5 23:52 /home/ubuntu/cephtest 2024-04-05T23:52:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 5 23:52 /home/ubuntu/cephtest 2024-04-05T23:52:33.741 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-05T23:52:33.782 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-05T23:52:33.865 INFO:teuthology.nuke:Checking targets against current locks 2024-04-05T23:52:33.893 DEBUG:teuthology.nuke:shortname: smithi016 2024-04-05T23:52:33.893 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T23:52:33.916 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 22:56:58.800689', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhKN200afSOEQt8jS2PAtA+auKHXkuwd4t84/Xc01o8FAlxnbFSwngxY3y4AKZ54ltc0IxzGtPK4IeurQbL0+8='} 2024-04-05T23:52:33.921 DEBUG:teuthology.nuke:shortname: smithi027 2024-04-05T23:52:33.921 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T23:52:33.944 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 22:56:58.801532', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGE/ewOEzbuRlNTBnHCBUfCrNXyio0sdGQ70x+KuVD0SfW2MGXB1sB9LT4bN/Mkifm6DTomOKFyXZBdCSPc09JA='} 2024-04-05T23:52:33.950 DEBUG:teuthology.nuke:shortname: smithi130 2024-04-05T23:52:33.950 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T23:52:33.979 INFO:teuthology.orchestra.console.smithi016:Power off 2024-04-05T23:52:33.979 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-05T23:52:33.992 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641731', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 22:56:58.799143', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqL/Yp1fzQGCtcEKcn6RtQwjTyV7jysI6HKbhXnc2d0NScxPd6BhzW/NLwIN4Jt96vofWPfzMezXV74dE0ZUzA='} 2024-04-05T23:52:34.005 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2024-04-05T23:52:34.006 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T23:52:34.026 INFO:teuthology.orchestra.console.smithi027:Power off 2024-04-05T23:52:34.027 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-05T23:52:34.040 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-05T23:52:34.050 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-04-05T23:52:34.050 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T23:52:34.070 INFO:teuthology.orchestra.console.smithi130:Power off 2024-04-05T23:52:34.070 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-05T23:52:34.081 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-04-05T23:52:34.100 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2024-04-05T23:52:34.228 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T23:52:34.283 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-04-05T23:52:38.041 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T23:52:38.082 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T23:52:38.153 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-05T23:52:38.193 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-04-05T23:52:38.284 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T23:52:38.396 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-04-05T23:52:42.156 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T23:52:42.194 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T23:52:42.266 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2024-04-05T23:52:42.305 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-04-05T23:52:42.367 INFO:teuthology.orchestra.console.smithi016:Power off completed 2024-04-05T23:52:42.397 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T23:52:42.407 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-04-05T23:52:42.508 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-05T23:52:42.609 INFO:teuthology.orchestra.console.smithi130:Power off completed 2024-04-05T23:52:42.717 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 2568.658357858658 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8919605b0a5dc1f1650c8aec3051c77c0e3e240e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2b9517c1385d4e6e8c8160ff5799db69 status: fail success: false 2024-04-05T23:52:42.717 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-05T23:52:42.807 INFO:teuthology.run:FAIL